Background

The main and primary goal of clinical education is to prepare students for the implementation of clinical skills in different healthcare situations (1) whereas a significant part of learning and teaching occurs in the clinical environment (2). It is necessary to recognize factors that affect it (3). Factors such as having a structured educational program, effective teaching method, and appropriate assessment tool (4). One of the important ways to achieve this goal is evaluation for these factors in the clinical environment (5).

Measuring the quantity and the quality of the educational environment and atmosphere plays a key role in creating and improving the optimal educational environment in medical schools and clinical settings (6) On the contrary, inappropriate evaluation may prevent achieving the goals of the educational system (7). One of these unique clinical environments is the operating room where teaching and learning happen in complex situations and interaction with others (8).

Studies show that the operating room provides an opportunity for students to develop and enhance their clinical skills associated with peri-operative care, and integrate their theoretical and practical knowledge (9). Fortunately, in recent years, many studies have focused on measuring the educational environment (10). This improvement in the appropriate and accurate evaluation of the educational environment is the result of developing efficient measuring instruments in this area (11).

Applying the appropriate instrument for assessing the clinical educational environment improves the environment, as well as makes it possible to apply changes to it. These instruments were developed in different environments of clinical education. For example, Anesthesia Theatre Learning Environment Measure (ATEEM) Surgical Theatre Educational Environment Measure (STEEM), or Postgraduate Hospital Educational Environment Measure (PHEEM) (12).

As mentioned above, one of these instruments is the Surgical Theatre Educational Environment Measure (STEEM) which was first developed by Kevin Cassar in 2004 for surgical residents (13). In the following years, STEEM was implemented and localized in some countries such as Canada that entitled OREEM or AMOTEC in Australia and OTEEM in Nigeria (14-16). Given that Cassar performed psychometric tests on this tool by evaluating postgraduate students, Nagraj introduced another tool called Mini STEEM in 2007, which was designed for undergraduate students, and validated it. Mini STEEM contains 14 items and 3 subscales (17). In the literature review, various factors have been stated in the learning experience of operating room students. In this case, Vesalim et al. 2020, after a literature review and thematic analysis, point to 26 structures in 5 areas (18). Also, Jahangir et al. (2021) stated that the experience of medical students is unique and differs from the resident's perspective on the educational atmosphere of the operating room, but they did not confirm the content validity of Mini-STEEM for all undergraduate students and believed that not all aspects of students' educational experience were considered in the Mini-STEEM (19). So, he and colleagues developed another new tool for medical students named SOREEM (Surgical Operating Room Educational Experience measure for Medical Students), which consists of 50 items (20).

According to studies conducted in Iran, various clinical educational environments such as ambulatory care and inpatient learning educational environments, have been assessed using appropriate tools. However, the educational environment of operating rooms has not been evaluated through proper tools.

Objectives

Following other studies about the development of new or localized instruments for undergraduate students and considering the difference between surgical technologists and other undergraduate or postgraduate students in comparison of the internship period, teaching methods, and their role in the surgical team, the researchers decided to develop a psychometric and localized version of the (STEEM) tool for surgical technologists in Iran.

Methods

This cross-sectional descriptive-analytic study was conducted in 2021 to develop the psychometric scale Surgical Theater Educational Environment Measure (STEEM) tool.

The main tool used in this study is the STEEM standard questionnaire in English which consists of two parts. The first part was related to recording demographic information (age, sex, and semester). The second part, the STEEM questionnaire (Surgical Theater Educational Environment Measurement) consisted of 40 items in 4 subscales under the following titles: trainees’ perceptions of their trainer and training (13 items), trainees’ perceptions of learning opportunities (11 items), trainees' perceptions of the atmosphere in the operating theatre (8 items), and trainees' perceptions of supervision, workload, and support (8 items). A 5-point Likert scale with 1-5 coding was used as follows: Strongly agree (5), Agree (4), Unsure (3), Disagree (2), and strongly disagree (1).

Nineteen Items (8, 11, 14, 16, 19, 22, 23, 26, 27, 28, 30, 31, 33, 34, 35, 36, 37, 38, and 40) consist of negative statements and are coded as negative. The maximum score would be 200 (13).

Research samples consisted of undergraduate students in surgical technology from the Iran University of Medical Sciences and Alborz University of Medical Sciences, who were selected through the census sampling method based on inclusion criteria. The research sample consisted of 201 students, which was considered to be 5 times the number of questionnaire items.

The stages of questionnaire development according to Figure 1 were as follows:

Translation of the questionnaire into Persian: At first, in accordance with the standards of psychometric evaluation and localizing tools, the necessary permit was obtained from the original developer of the English version of the Surgical Theatre Educational Environment Measure (STEEM) questionnaire, Kevin Cassar, University of Aberdeen, Scotland, in order to implement and translate the questionnaire. After obtaining the permit, the questionnaire was translated from English into Persian by 3 translators. Then the Persian version was back-translated by 2 English speakers who were also fluent in Persian and the result was compared with the original questionnaire. During a session with translators and researchers, the differences between the two English versions were revised and the final version of the back-translation was agreed upon.

In order to determine the content and face validity of the questionnaire, ten professors familiar with training in surgical technology were provided with the questionnaires. The content validity of the questionnaire was evaluated from two aspects: content validity ratio (CVR) and content validity index (CVI).

Determine the content validity ratio: In calculating the content validity ratio, the necessity and usefulness of each item were assessed. The following formula was used to calculate the content validity ratio:

In this formula, Ne is the number of experts rating an item as Essential and N is the total number of experts rating the items. If the calculated CVR value for each item is greater than the critical value of 0.62, the content validity of that item will be approved (21).

Determine the content validity index: CVI was calculated according to Waltz and Bausell's (22) content validity index which determines the amount of relevance, simplicity, and clarity of each item in the questionnaire on a 4-point Likert scale for each item. For example, for “relevance” the following points are used: “completely relevant", “relevant”, “somehow irrelevant”, and "incompletely irrelevant” (23). In this study, the content validity index was calculated for each item by dividing the number of panelists scoring 3 and 4 by the total number of panelists (24). According to Hyrkas et al, a score of 0.79 and higher was considered for the confirmation of items according to CVI (25). Then the mean score of the content validity index was calculated for each item and also for the whole items.

Determine reliability: After calculating content validity and developing a questionnaire with acceptable CVI and CVR, reliability was evaluated by measuring
internal consistency and also the test-retest method. In order to measure internal consistency, Cronbach's alpha coefficient was calculated for 30 samples. For evaluating the reliability of the questionnaire, the
test-retest method was implemented by distributing the questionnaire at a two-week interval among qualified candidates. The intraclass correlation (ICC) was calculated. Values higher than 0.8 are acceptable for approving the reliability of the tool (26). Then the final questionnaire was distributed among the research population and the results were analyzed using SPSS
19 software.

Determine construct validity: For evaluating construct validity, exploratory factor analysis was performed at the following stages:

1. The assessment of sample adequacy was performed by the Kaiser-Meyer-Olkin test. KMO higher than 0.7 was defined to confirm sampling adequacy.
2. Examining the correlation matrix of variables by using Bartlett's test of sphericity for factor analysis.

If Bartlett’s test value was significant at a level of less than 0.05, then there was a significant relationship between the variables and it was possible to discover a new structure in the data.

3. Factor extraction by implementing the principal components analysis method
4. Rotating factors by applying Varimax rotation (27)

Results

201 undergraduate students of surgical technology participated in this study, 67% of whom were female and 33% male. The mean age of participants was 22 (20-25) years and they were in semesters 3 to 8. All the samples had the experience of presence in the surgical theater for at least two semesters.

After examining the face validity by the expert's panel, in the first validation phase, questions 31, 33, and 34 were omitted.

According to the results of the content validity examination in the 37-item questionnaire, and also considering the average content validity ratio of 0.57 and the content validity index of 0.92, a new 35-item questionnaire was developed after the following changes:

• Items No.1, 2, 5, 6, 8, 17, 18, 19, 20, 22, 25, 27, 30, 36, and 39 were approved after being modified.
• Items No.1, 10, 28, 37, 38, and 40 were omitted and replaced.
• Items No.21 and 24 were omitted and the rest were approved.

Figure 1. questionnaire development steps

In the stage of assessing the completion of the questionnaire by the students of surgical technology, the overall mean score of the questionnaire was calculated 89 out of 175. The highest mean score belonged to the first subscale, i.e. trainees' perceptions of their trainer and training (26.8). The lowest mean score belonged to the subscale trainees' perceptions of supervision, workload, and support (17.3). More detailed information can be found in Table 1.

The results of examinations regarding the reliability of the original 40-item questionnaire and the secondary 35-item questionnaire approved the reliability of the tool in both stages (The Intra Class Correlation (ICC) > 0.9).

The examination of construct validity by the
Kaiser-Meyer-Olkin (KMO) test approved the adequacy of the sample size (KMO = 0.88). Bartlett’s test also showed a level of significance (P ≤ 0.0001).

During calculating construct validity and factor analysis, in the secondary 35-item questionnaire,
5 items (The equivalent of items No.15, 16, 22, 26, and 36 in the 40-item questionnaire) with a factor loading of less than 0.4 were omitted. Factors were extracted by performing principal component analysis on the main components with an eigenvalue of more than 1, which led to the identification of seven factors that explained 62.6% of the total variance. However, researchers chose the first five factors that accounted for 55.6% of the total variance. Thus, the final questionnaire, called IMOTEC (Iran Measure of Operating Theater Educational Climate), was obtained with 30 items and 5 subscales (Appendix).

The names of some factors and the axis of some items changed. Due to the nature of the items, new domains were identified and named. Table 2 shows the factor rotation matrix. This matrix indicates the items associated with each factor. As seen in this table, item No.22 was loaded on two factors (3 and 5). Researchers chose domain 3 for this item. Finally, no other factor rotation was more appropriate in separating components than varimax rotation. Therefore, the results were interpreted based on this rotation.

Changes made to the domains were as follows:

• Item 11 from domains 1 to 5
• Items 34 and 35 from domains 4 to 1
• Item 14 from domains 2 to 5
• Items 17, 18, and 20 from domains 2 to 4
• Items 19 and 22 from domains 2 to 3
• Items 23, 24, 26, 27, and 30 from domains 3 to 2
• Items 29 and 33 from domains 4 to 3
• Item 31 from domains 4 to 5

After performing factor analysis and obtaining domains, Cronbach’s alpha was again calculated to examine the reliability of the questionnaire. Results are shown in Table 3. As seen in the table, the highest reliability is associated with domain training. The total reliability of the questionnaire (Cronbach's alpha) was calculated at 0.845.

Discussion

The psychometric evaluation of the Iranian version of STEEM was performed by modifying items 1, 2, 5, 6, 8, 11, 16, 17, 18, 19, 20, 22, 25, 27, 30, 36, and 39, removing and replacing items 10, 28, 37, 38 and 40, and removing items 21, 24, 31, 33 and 34. The results show that this tool was a valid, reliable, and practical tool for evaluating the educational environment of operating rooms for undergraduate trainees in Iran.

Similar to other studies conducted by Nagraj, Mahoney, Ibrahim, Binsaleh, Soomro, Majbar, and Ahmad Cronbach's alpha coefficient for the modified questionnaire was calculated to be higher than 0.7
(= 0.845) in this study, which approves the internal consistency and reliability of the tool (13-17, 19, 28-30).

Also, by the comparison of different subscales of the questionnaire regarding the students’ gender, no significant difference was found between female and male students in any of the subscales, which was in line with the study of Cassar (Scotland, 2004). However, by taking into account the academic term, in the subscales of learning opportunities and workloads, a significant difference was found in the scores of students of different semesters, which was consistent with the results of Cassar’s study of the subscale of learning opportunities (13).

Table 1. Central indices and dispersion of subscale scores in 4 domains of initial questionnaire

Table 2. Rotated components matrix

In this study, similar to Mahoney and Nagraj’s studies, the STEEM tool underwent modifications for the validity to enhance and match the educational environment of Iran. For example, regarding the validity of their data, Mahoney et al stated that Cassar’s original subscales were not supported by the data of their research. Instead of four subscales with equal importance, one specific subscale covers up to 32% of the total variance which focuses on the trainer’s skills and behavior and has an internal consistency of higher than 0.9. Besides, it identifies up to 5 subscales, which were named according to their items as ‘Supervisor Facilitation of Learning Opportunities’, ‘Characteristics of the Surgical List’, ‘Interaction with Non-Surgical Staff’, and ‘Distractions from Operating Theatre Learning’. All of these subscales had an internal consistency of more than 0.7 and the total internal consistency of 0.91 (15).

In the current study, performing exploratory factor analysis (KMO = 0.88) and extracting factors with an eigenvalue greater than one through principal component analysis resulted in the identification of seven factors, which accounted for 62.6% of the total variance. However, in the final evaluations, the first five factors were chosen, which explained 55.6% of the total variance. Based on covering relevant items, these five subscales are named as follows: teaching and training, interaction with theatre staff, learning opportunity, workload, and support and supervision. In Ahmad’s study, 5 subscales including structured learning process, psychological management and training, organizational support, quality of faculty & training, and students’ self-regulation and participation were identified in factor loading, which is aligned with the present study (20).

Additionally, the factor analysis output in Nagraj’s study consisted of three factors explaining 56% of the total variance and indicating that there are 3 subscales, which were named due to their relevant items as follows: good surgical operating experience, friendly atmosphere in theatre, and discrimination against me. In this study, after performing exploratory factor analysis (KMO = 0.770), 13 factors were found to count for 73.2% of the total variance, and according to this fact, the 40-item STEEM questionnaire was condensed to a 14-item questionnaire (17).

It should be noted that in order to enhance the suitability of the STEEM questionnaire for Nigerian educational environments, Ibrahim reduced the number of items to 33 by removing items associated with gender and racial discrimination and time limitations before distributing them.

Similarly, in this study 10 items were excluded according to face and content validity (6 items) and the results of exploratory factor analysis (5 items with factor loadings less than 0.4). Therefore, a shorter questionnaire with 30 items was presented to the respondents (16).

Similar to the Mahoney study in which 1 item in relation to private patients was added to Cassar’s
40-item questionnaire (15), in the present study,
5 questions were replaced as a substitute for a number of excluded questions, added to the questionnaire.

Table 3. Data distribution and Cronbach’s alpha coefficient of each domain after factor analysis

One of the important points criticized in the evaluation of the STEEM questionnaire is its Likert scale. A study entitled Surgical Theatre (Operating Room) Measure STEEM (OREEM) Scoring Overestimates Educational Environment: the 1-to-L Bias, which was conducted by Dimoliatis (2013) in Greece, states that the questionnaires of DREEM, ATEEM, PHEEM use a 5-point Likert scale scoring with a 0-4 coding, and criticized the 1-5 scoring of
5-point Likert scale in STEEM, OREEM, and Mini STEEM. Dimoliatis believes that this scoring method reduces the value of this precious tool because whenever we express a number as a percentage, we expect it to fall within a 0-100 range so that the scores will be interpreted as follows:

• Very poor - if the score lies between 0 and 24.9
• Poor - if it lies between 25 and 49.9
• Good if it lies between 50 and 74.9
• Very good if it lies between 75 and 100

However, if the items are coded from 1 to 5 in a
5-point Likert score, it might result in errors, as happens in STEEM, OREEM, and Mini STEEM, and is likely to distort respondents’ perceptions. For instance, when the total mean score is 148 out of 200, we expect it to be equal to 74% in the 0-100 range. However, this is not the case, and the given score falls within the
20-100 range. As a result, this method of scoring the questionnaire items from 1 to 5 will lead to a 20% overestimation of the actual percentage of the results, which reduces the usefulness of this tool. The obtained percentage would be equivalent to the total mean score only if the lower limit is zero. In other words, the quotient is a pseudo-percentage, not a percentage
(0-based) (31). Therefore, the present study was based on the STEEM questionnaire, and the calculations and assessments were done according to the 5-point Likert scale with a 1-5 coding, however, in order to increase the accuracy and efficiency of the tool and avoid computational errors while estimating the obtained scores, the final modified questionnaire of this study will use the 5-point Likert scale with a 0-4 coding, and according to the final 30-item questionnaire, the scores would be interpreted as follows:

• Very unfavorable: If the score is in the first zone, that is 0- 29.9.
• Unfavorable: If the score is in the second zone, which is 30-59.9.
• Favorable: If the score is in the third zone, that is 60-89.9.
• Very favorable: if the score is in the fourth zone, that is 90-120.

In addition, the scoring should be reversed for negative statements: 11, 14, 17, 19, 24, 25, 27, and 28.

Conclusion

According to the findings, the localized Iranian version of the STEEM questionnaire, called IMOTEC (Iran Measure of Operating Theater Educational Climate), is a reliable and valid tool that can be implemented for the assessment of the educational environment from the viewpoint of undergraduate students and can be used for educational, research and practical purposes to evaluate the educational environment of surgical theatres in Iran.