Background

Today, the education of medical and paramedical students is mainly based on competency-based education (1). In competency-based education, there is an emphasis on educational consequences, professional practice, and evaluation methods. Therefore, diverse evaluation tools such as portfolios are proposed to appraise learners’ competencies and practices (2). However, the term “Portfolio” became popular in education in 1990 and was first defined as an organized complex of learned subjects (3). Over the last two decades, medical education has broadly used it to develop learners’ competency (4). Portfolios are available in two formats: paper and electronic. However, for a variety of reasons, including saving and reconsideration data related to students’ practice over a certain period of time in a virtual environment, increasing motivation of students based on tutors’ feedback, sharing content with tutors and other students, bolding the important data, easily updating contents, easily accessing portfolios for administrators and tutors, helping to recognize the strengths and weaknesses of students and finally reducing paper consumption, electronic or web-based portfolios are more popular (5-7). Today, the portfolio is known as a tool for gathering and assessing information relevant to learners’ development in appointed assignments and defined competencies, and it is introduced as a competent method in motivating and guiding the learners towards reflection in Evaluation (2, 8). Tutors assist students in developing new knowledge and skills, correcting their malpractice, acquiring competency through effective feedback, and providing opportunities for reflection based on documents of portfolio (9, 10). Such reflection enables students to demonstrate growth and development as professionals (11). At the same time, developing the students’ reflective capacity on work and learning, portfolios help to improve autonomy and collaboration with others. Also, according to the results of studies, not only can the e-portfolio become a tool to improve learning skills and competencies, but it can also contribute to improving employability and guarantee a future job (12).

According to the results of several studies such as Ekayanti et al. (2017), Belcher et al. (2014), Chae & Lee (2021), Vaghee et al. (2016), Mapundu & Musara (2019), Paulo et al. (2021), the use of electronic portfolio has had positive effects on the learning and competencies of learners in different disciplines
(11-16). However, based on the researcher's literature review, the use of an e-portfolio with a certain framework has not been investigated in surgical technologists' students, especially because these students are forced to learn in critical conditions and acquire skills in the stressful climate of the operating room. On the other hand, the curriculum of this discipline in our country is different from the rest of the world. In the few examples that have been worked in Iran, it is not mentioned to reflection as a special characteristic of the portfolio. Also, as an educational outcome, it only referred to the students’ satisfaction in
the evaluation stage (17).

Objectives

The present study was conducted to develop a specific e-portfolio for surgical technologist students and evaluate their improvement via assessment of their satisfaction and learning.

Methods

This interventional study was conducted using a sequential mixed method in 2 phases and four stages from October 2018 to May 2020 after the approval of the initial research project in the Center for Medical Education Research and Development of the Ministry of Health and Medical Education, with license code No. 97006.

In the mixed method study, quantitative and qualitative research approaches are adapted to expand, understand, and interpret the data (18). Sequential designs are a combination of two-step research methodology, with qualitative or quantitative data collected in the first phase, and then the other data types are collected (19).

Since there was no standard framework for e-portfolios as formative assessment tools in accordance with the operating room technology curriculum in our country, the researchers decided to develop them using a qualitative method in the first phase and evaluate them using a quantitative method in the second phase, following a mixed-method study, via assessment of students’ satisfaction and learning.

Phase 1: This phase was implemented in three stages. Firstly, a literature review was done in PubMed, Science Direct, Google Scholar, SID, Scopus, and Magiran databases between 1990 and 2018 with the keywords of “clinical assessment tools,” “portfolio,” “e-portfolio,” “logbook,” “surgical technologist,” “operating room,” and “competency.” After identifying the key components and structure of a standard portfolio, including “discipline regulations,” “study resources,” “daily activities report,” “course educational objectives,” “work-based assessment tools,” and “feedback and reflection reports, in the second stage, a descriptive qualitative study was conducted to determine specific component for mentioned categories according to surgical technologists’ curriculum in Iran. For this purpose, based on the Elo & Kyngas method, qualitative content analysis was used with a deductive content analysis approach. In this approach, analysis processes are represented as three main phases: preparation, organizing, and reporting. Deductive content analysis is used when the analysis structure is operationalized based on previous knowledge (20).

Participants: Fourteen Iranian expert panel members participated in this phase, including eight instructors, two board members of the operating room technology committee, three surgical technologists, and one medical educationist. The inclusion criteria for selecting participants were the experience of at least five years of educational activity, five years of activity in policy making, and clinical work experience in this field for at least ten years. Expert selection considered educational qualifications, experience in operating room fields, and willingness to participate in this process.

Data analysis: The data analysis was performed by the guided content analysis method and according to designing the basic analytical matrix in the stage of organizing in Elo & Kyngas method, including categories including “study resources,” “daily activities report,” “course educational objectives,” “work-based assessment tools,” “feedback, and reflection. Data analysis started with the first interview and continued until the fourteen interviews when the data reached saturation (20). Four criteria of credibility, dependability, conformability, and transferability of Lincoln and Guba were used to confirm the accuracy of the data (21).

For this purpose, the method of prolonged engagement with data and spending time to collect and analyze data was used to improve the credibility of the data. Also, the member check method was used in the initial coding stage to increase the dependability of the data in the interview stage. In this way, the codes and interpretations obtained from the analysis of some interviews were checked with the participants. In addition, a review by external supervisors was used to confirm the correctness of the coding. Also, to achieve the conformability criterion, the entire data collection and analysis process was described accurately and in detail, and finally, by describing the full characteristics of the participants, the transferability of the data was provided for readers to judge.

In the third stage, the e-portfolio, after consulting with the informatics engineer, converted to an electronic version, was uploaded to the website of the paramedical school at the following address: https://tabib.abzums.ac.ir.

Phase 2: At this phase, a quantitative study was done to establish the effectiveness of the intervention. The extracted contents from the first phase study, uploaded to a virtual platform as the main elements of the operating room e-portfolio, were considered the main interventional plan. For this purpose, after training selective tutors and internship students, the e-portfolio was used as a formative assessment for them in the seventh academic semester.

Sampling in this phase was done in the form of a census and according to the inclusion criteria. Typical inclusion criteria for tutors included at least two years of work experience in coaching surgical technologist students and a willingness to participate in this research. Inclusion criteria for students were passing theories and practical courses and entering an internship course in the final year of education. A consent form was obtained from those who voluntarily accepted to participate in the study. Fifty-six students at the internship level and seven instructors participated in this phase.

Before using the e-portfolio, students' knowledge and practical skills were assessed as basic knowledge and practical skills. An MCQ was developed for knowledge assessment based on a blueprint of course content and consideration of the Milman checklist. Also, a checklist was developed based on instructional goals in this course to assess students' academic achievement in practice and skills. The validity of these tools was confirmed by content validity and expert panel opinions. The reliability of MCQ was confirmed by the calculation of Cronbach's alpha (0.81). For measurement of Reliability, the checklist was calculated inter-rater reliability (0.88).

From the beginning of the semester, students were required to record their training daily based on the discipline regulations, educational goals, and study resources, and they were encouraged to reflect on their daily experiences and record how they would act if they faced this situation the next time. The tutors also assessed and monitored the strengths and weaknesses of the students' performance weekly based on the assessment checklists and gave them feedback for improvement or correction. The director of the operating room group also evaluated the performance of the tutors and students at a higher supervisory level to ensure the strict implementation of the program and the feedback and opportunities for reflection given to the students. The implementation of this program continued for 16 weeks during the academic semester at the time of the study. Satisfaction, knowledge, and practical skills were evaluated after the use of the e-portfolio again. A standard survey evaluated the satisfaction of students, and its validity and reliability were confirmed in the Ahmadi et al. (2016) study (22). In this study, Cronbach's alpha was calculated as 0.78, which is acceptable for reliability. The collected data was entered into the software SPSS 22 and analyzed using t-test and chi-square tests.

Results

Phase 1: Participants in the study's first phase were eight instructors, two board members of the operating room technology committee, three surgical technologists, and one medical educationist. The average age of participants was 41 (± 2.4), respectively. Furthermore, their experiences in the field of operating room education or working were 18 (± 3.8) years. Directed content analysis of 14 structured interviews was led to extract six main categories, including; “discipline regulations,” “study resources,” “daily activities report,” “course educational objectives,” “work-based assessment tools,” and “feedback and reflection reports.”

Phase 2: Based on the quantitative study results,
56 students at the internship level and seven instructors participated in this phase; sixteen students (28.5%) were male, while 40 (71.5%) were female. The mean age of the students was 20.01(±1.74). The instructors were all female, with a mean age of 33.96 (± 1.37).
The results of the quantitative phase of the study, students' and instructors’ satisfaction levels, and improvement of students’ knowledge and skill scores in Tables 1 and 2 are shown.

Based on Fisher's exact and numerical analysis results, there was no significant difference between the satisfaction levels of students and instructors and their demographic traits, with a p-value of 0.518.

Results demonstrated that more than two-thirds of the instructors and the students were completely satisfied with the e-portfolio assessment. In an analysis of the survey’s open-ended question (would you please explain the advantages and disadvantages of this method of assessment in comparison to other commonly used methods?), instances like resource management, the need for internet infrastructure, access to information, improvement of interactions, organized assessment, learning by assessment, transparency of assessment, informative assessment, and finally opportunities for reflections and independent learning were identified.

The paired sample t-test results showed that the students' knowledge and practical skills significantly improved after using the e-portfolio.

Discussion

The findings of the qualitative phase of this study led to the extraction of six main categories for developing an authentic e-portfolio framework, including discipline regulations, study resources, daily activities, educational objectives, assessment tools, feedback, and reflection reports. Numerous studies, such as Ekayanti et al. (2017), Belcher et al. (2014), Chae & Lee (2021), and Tan & Ting (2022), confirmed these requirements and documents in making the framework of e-portfolios (13-15, 23).

In the quantitative phase of this study, the results show that most of the students and instructors were satisfied with the e-portfolio for clinical competency assessment. Asadi et al. (2014) preferred this method of assessment over other commonly used methods (24). In other similar studies, students were more satisfied with the portfolio method than the other commonly used methods because this method focuses on students’ progress and feedback from preceptors (25). According to Ahamdi et al. (2016), the total score of satisfaction with the e-portfolio assessment was higher than the other common assessments (22). Contrary to these results, Tailor et al. (2014) believed that 60% of medical interns did not benefit from e-portfolios (7). In another study, students reported low motivation and moderate self-confidence when it came to using e-portfolios (26). Different results can be due to initial anxiety in applying this method, incompletion of the prepared system content, unfamiliarity with the system, and lack of sufficient supervision and unstructured assessment. Van der et al. (2020) studied 90 e-portfolios.

Table 1. Frequency of students and instructors’ satisfactions with e-portfolio

df: Degree of freedom

Table 2. Comparison of the students’ knowledge and practical skill scores before and after using an e-portfolio

SD: Standard deviation

They indicated that the feedback received was mostly limited to general opinions and lacked specific comments and recommendations for further action. Also, no specific goals were defined for learners (27). Thus, the items considered, the preparedness of preceptors and learners, and the motivation for this tool's constant and correct use all impact its efficiency and students’ satisfaction. Yoo et al. (2020) believed that the results are more valid when evaluators with an appropriate education do the assessments based on certain standards and systematically (8). In many cases, the students emphasized structured, informative assessment in their learning path. Tan & Ting (2022) confirmed that one of the important uses of the portfolio is informative assessment (23), as the last subject that students mentioned in the open-ended question was the importance of feedback and reflection. They believed that this assessment tool gave them an opportunity to think and improve their learning. Developing reflective skills is one of the unique features of portfolios that lead to improving clinical skills (28). Sánchez et al.'s (2013) study showed that e-portfolios guided students' learning process by indicating knowledge gaps to themselves and teachers (29). Annemarieke et al. (2015) found that the e-portfolio approach is helpful for professional development, setting educational goals, assessing competency, and receiving support from supervisors (30). Other studies recommend that using portfolios and e-portfolios with feedback could help students with academic achievement (29, 31-33).

Also, the results of a study in the assessment aspect indicate that e-portfolios have a positive impact on the improvement of students' learning in cognitive and psychomotor domains. As a result, Lai & Wu (2016) claimed that portfolios aided clinical learning through features such as self-awareness, self-assessment, and self-judgment (34). Furthermore, Vaghee et al. (2016) reported that e-portfolios improved students' academic achievement by involving students in teaching-learning (16). Mapundu & Musara (2019) demonstrated that e-portfolios increase learning due to features such as flexibility and students' participation in learning (12). In other studies, portfolios enhanced learning during training in a women's ward (35) and enhanced both the quality and the depth of learning in medical students (25). Sánchez et al. (2013) investigated e-portfolios to assess surgical skills quantitatively. Seventy-nine percent of the students felt they had more access to their professors, and at the end of each course, on average, students reached 65% of the defined general goals and 87% of the skill goals (29). In another similar study, this tool increased the clinical competency of nurses in all aspects (36). All of the preceding studies are consistent with this study’s results. Therefore, this method seems to improve clinical skills by increasing interaction between the student and the preceptor, increasing feedback, and encouraging students to reconsider.

Paying attention to all the components of a standard structured portfolio was one of the strengths of this study, which can be used as a model in other disciplines. Also, the evaluation of the effectiveness of this assessment tool in the variables of satisfaction and learning is another positive highlight of the present study, but since this study was done in a quantitative stage in a semi-experimental design before and after in one group, it can be conducted in future studies between two groups of cases and control groups or to examine the effectiveness of it in related to other variables such as critical thinking, academic enthusiasm, etc.

Conclusion

Regarding the positive impacts of this method of assessment on students’ satisfaction, knowledge, and skills, it seems that applying an e-portfolio as one of the novel assessment tools can be a proper replacement for the present assessment methods by providing opportunities for feedback and reflection. Therefore, this method is recommended in the clinical assessment of surgical technologist students.