Background

The field of medicine is built upon a foundation of scientific inquiry and rigorous evidence-based practice. To uphold the highest standards of patient care, it is imperative for medical students to cultivate a deep understanding of research methodology and evidence evaluation. While educational interventions targeting primary research types have received extensive attention (1), the development of effective training courses focused on secondary research, such as review studies, has been relatively neglected thus far.

Systematic reviews and meta-analyses play a pivotal role in evidence-based medicine (EBM), serving as the highest level of evidence in the EBM pyramid (2). Medical education guidelines suggest undertaking a practical approach toward systematic reviews, emphasizing the need for implementing this research approach in medical education (3).

Research-related interventions are fundamental components of modern medical curricula (4). The majority of students are eager to learn research skills, and the reason why some students are not inclined toward research is their perception of a lack of connection between research and clinical activities (5). However, university medical curricula often fail to adequately prioritize the training and enhancement of research skills. The absence of basic research training creates a sense of disinterest among students and prevents them from participating in research fields. Consequently, many medical students encounter difficulties in research projects when they reach the final stages of their education. These challenges may include inadequate familiarity with research methodology, difficulty in data analysis, lack of scientific writing experience, and weaknesses in evaluating research effectiveness (6). Familiarity with systematic reviews and meta-analyses enables medical students to accurately analyze medical evidence and comprehensively evaluate different study findings. These skills empower them to make better decisions based on scientific information and benefit from the best existing evidence (7). Therefore, conducting comprehensive training workshops on systematic review methodology appears essential to address these gaps.

Objectives

This study aimed to design, implement, and evaluate a training course specifically focused on systematic reviews for medical students. The objective was to establish a solid foundation that underscores the importance of educational interventions in this area, to further promote such instructional designs at the national and international scale.

Methods

Study design: This study was conducted at Shahid Beheshti University of Medical Sciences (SBMU) in 2023 using a pretest-posttest design. The instructional design followed the five steps of the ADDIE instructional design model (analysis, design, development, implementation, evaluation) (8), and the first three steps of the Kirkpatrick’s evaluation model (reaction, learning, behavior). Kirkpatrick’s model evaluates the effectiveness of an educational program based on four different levels: reaction, learning, behavior, and results. This framework helps teachers and instructional designers to comprehensively evaluate the value of training and its impact (9).

Study population and participant sampling: Initially, a poster of the training course along with a questionnaire was shared with medical students of all academic levels at SBMU through online messengers. The student recruitment questionnaire included initial demographic information (name, age, gender, academic level, and email), familiarity with fundamentals of basic research, research experience, preferred research area, and a needs assessment for a systematic review training course which was assessed based on a 10-point Likert scale. The questionnaire was designed to assess the participants’ characteristics and their learning needs according to the first step of the ADDIE model (analysis). The minimum required sample size for the normal distribution of data was calculated to be 30. A total of 78 students from various academic levels completed the initial recruitment questionnaire. These individuals were assessed for eligibility to register in the workshop and the final 30 participants were enrolled using convenience sampling. Accordingly, the criterion for students' enrollment was their lack of previous familiarity with systematic reviews, which was assessed based on the absence of published systematic review and meta-analysis articles. The course plan, along with the final registration link, was emailed to the selected individuals for their final registration, in accordance with the course content and regulations. Eventually, 30 medical students from various academic levels (basic sciences, physiopathology, clerkships and internships) who were interested in learning systematic reviews and had not published any articles in this field, were selected as the final sample.

Instructional design and implementation of the course: Instructional contents were produced before starting the course. Multimedia contents, including instructional videos and narrated PowerPoint files alongside text and Excel files, were provided. Participants were also introduced to relevant books and online resources. Prior to the start of the course, a pre-test consisting of 20 multiple-choice questions based on the course content was administered to the participants. The questionnaire was assessed in terms of face validity and content validity (using content validity index (CVI) and content validity ratio (CVR)) according to the opinions of five medical education specialists, ensuring that the questions cover all aspects of the course content. The students were divided into five groups, each comprising six members, and a representative was appointed for each group. In line with the design, development and implementation steps of the ADDIE model, the training was delivered through a combination of in-person, synchronous online and asynchronous online methods over 12 sessions. The course plan is presented in Table 1. The last step of the ADDIE model was carried out according to Kirkpatrick’s model. Upon completion of the course, students' satisfaction (reaction) was evaluated using a 5-point Likert scale.

Table 1. An overview of the systematic review course plan

To assess students' knowledge (learning), the same pre-test questions were used as a post-test, and the average scores before and after the course implementation were compared. Additionally, the impact of different teaching methods alongside the participants' demographic characteristics (gender and academic level) were assessed on the final course outcomes. Lastly, to measure the skill of interested students (behavior), the protocol registration of a systematic review study was submitted to the International Prospective Register of Systematic Reviews (PROSPERO) website as an evaluation basis.

Statistical analysis: All analyses were performed using SPSS software, Version 24. To compare the results of the pre-test and post-test for assessing the changes in participants' theoretical knowledge, the data distribution was first determined using Shapiro-Wilk and Kolmogorov-Smirnov tests. If the distribution was normal, a paired t-test was used; otherwise, the non-parametric Wilcoxon test was employed. A p-value < 0.05 was considered significant.

Ethical considerations: This study was approved by the ethical committee of SBMU (ethics code: IR.SBMU.SME.REC.1402.039). The course was completely free of charge, and no fees were required from the students to participate in the course.

Results

General characteristics of the study participants: The sample age range varied from individuals born in 1994 to 2001 (mean age: 25.2, SD: 1.9). Among the participants, 17 (56.6%) were male and 13 (43.4%) were female. Interns had the highest participation rate, with 14 (46.6%) individuals, followed by clerkship students, with 12 (40%) participants. Additionally, there were two participants each from the basic science (6.6%) and physiopathology (6.6%) disciplines (Figure 1).

Out of the initial 30 participants, 3 individuals (one female from the clerkship level and two males from the internship level) dropped out during the course. The remaining 27 individuals remained enrolled in the course until its completion.

Half of the participants expressed a 100% need for the implementation of this course (Figure 2).

Participants’ satisfaction: The satisfaction survey, administered at the end of the course using a 5-point Likert scale, indicated that 85.2% of the students expressed complete satisfaction with the course, while the remaining participants indicated an 80% satisfaction rate.

Learning outcomes: The results of the pre-test and post-test are presented in Table 2. Based on the results of the Kolmogorov-Smirnov and Shapiro-Wilk tests, the data obtained from the pre-test and post-test were not normally distributed (p-value < 0.05). Therefore, the Wilcoxon test was used to compare the means.
The results of this test indicated that the training course significantly improved the participants' theoretical knowledge (p-value< 0.001). The training had a significant impact on learning in both male (p-value = 0.001) and female (p-value = 0.013) participants. Due to the small number of participants in the basic science (2 participants) and physiopathology (2 participants) levels, a comparison was made between the three groups: basic science/physiopathology, clerkship students and interns. The training resulted in a significant increase in theoretical knowledge for clerkship students (p-value = 0.004) and interns (p-value = 0.006), but no significant difference was found in the basic science/physiopathology group (p-value = 0.414).

The greatest improvement in test scores was observed among subjects that were instructed using the asynchronous online teaching method (p-value=0.001), presumably due to students' access to content and reviewing materials compared to the other two methods (synchronous online: p-value = 0.180/
in-person: p-value = 0.660). However, it should be noted that the statistical concepts that were taught through in-person methods, require more practice and repetition to achieve mastery due to their inherent complexity. Therefore, the absence of improvement in students' learning outcomes for these topics cannot be solely attributed to the teaching method.

Skill outcomes: In order to assess the practical skills of the students at the behavioral level, considering the time constraints of medical students, a survey was conducted to allow individuals with both interest and sufficient available time to participate in this stage. Based on the survey, eight individuals expressed their readiness to participate. They were divided into two groups of four, and two systematic review study protocols were submitted to the PROSPERO website and subsequently approved after peer review.

Discussion

Considering the existing gap in teaching systematic review methodology to medical students, this study focused on the design, implementation, and evaluation of a systematic review training course. To the best of our knowledge, this is the first study aiming to conduct and evaluate a systematic review training course in Iran. Overall, our findings demonstrate that the structured implementation of a training course for systematic review studies has a significant impact on enhancing the satisfaction, theoretical knowledge and skills of medical students.

In a study conducted by Patabendige et al. in 2023, which was similar to our study in terms of designing and implementing a systematic review training course, 29% of the students expressed their motivation for participating in these courses as a desire to acquire research principles, while another 29% mentioned their motivation to pursue systematic review research in the future. Additionally, 24% expressed the need to carry out these types of research as a college project (10).

Table 2. Pre-test and post-test results

These findings are consistent with the results of our study, indicating high students’ interest in learning the methodology of secondary studies.

The evaluation of the program at the reaction level among our study participants showed that 85.2% of the students were completely satisfied with the course, and others expressed an 80% satisfaction rate. Previous studies on designing interventions for research principles have also demonstrated that the structured design of such courses can be accompanied by positive participant reactions. For example, Mbuagbaw et al. designed a preliminary workshop on systematic review studies and demonstrated that the participants had a high level of satisfaction with the content of the course (11).

Previous studies have also shown that the appropriate design of educational interventions aiming at teaching research methodology can promote theoretical knowledge about research activities. For example, Shrivastava et al. conducted a four-day research methodology training course, including lectures, interactive sessions, and group work. Evaluating the participants’ knowledge indicated a significant improvement in post-test scores compared to pre-test scores (12). In another study by Kumar et al., a two-day educational intervention in research methodology resulted in a significant increase in the average post-test knowledge score compared to the pre-test score (13).

Investigation of the factors influencing participants' learning revealed that the increase in theoretical knowledge was not dependent on the participants' gender, as this improvement was observed in both male and female students. However, our findings indicated a more noticeable improvement in the level of theoretical knowledge among clerkship students and interns. This finding suggests that these students, due to their clinical exposure and acknowledgment of the significance of evidence-based research in medicine, prioritize research and recognize its importance in their medical practice.

Our preliminary findings demonstrated that students showed the greatest improvement in their level of theoretical knowledge in the topics taught using the asynchronous online approach. The asynchronous approach is a type of virtual learning in which learners engage in independent learning without the need for synchronous online presence. In this approach, educational materials are provided through a learning management system (14). Considering that the majority of our course participants were interns and clerkship students, who are often involved in clinical shifts, the use of an asynchronous online approach, due to the availability of educational content and the flexibility in study schedules, could effectively enhance individual learning outcomes. Nevertheless, this method has limitations due to restricted interaction with instructors and other learners (15). Therefore, in the present study, we adopted a blended learning approach to achieve maximum efficiency while considering the constraints of the students. In this approach, the use of traditional in-person teaching methods was combined with asynchronous and synchronous online approaches. This approach not only facilitated face-to-face interaction between instructors and learners but also mitigated the limitations of each method when used independently. In other words, the shortcomings of each individual approach were reduced, and their strengths were integrated (16).

This study has different limitations that should be considered in interpreting the results. The present study was conducted on a limited number of medical students of SBMU, and only one mentor was responsible for the implementation of the course. In order to generalize the results on a larger scale, other similar studies with increased sample sizes and expert mentors should be carried out in other institutions. On the other hand, the participation of basic science and physiopathology students in this training course was low, and distinct analysis for each group was not possible. This may have led to bias in analysis among these student subgroups.

Conclusion

Considering the results of this study, medical students felt a significant need to familiarize themselves with systematic reviews. The increase in knowledge after training course was more pronounced among students at higher academic levels, including interns and clerkship students, who have tangible exposure to the clinical field. By adopting instructional design models like the ADDIE model and employing diverse teaching methods, including the production of instructional videos within an asynchronous virtual learning environment, we could successfully design and implement a systematic review research training course tailored to the needs of medical students.