Background

The research plays a significant role in medicine development. Medical students need to keep themselves updated with the newest findings. Moreover, they should be able to contribute to their fields of interest by actively engaging in academic research (1). Several models have been offered for the assessment of academic training. One of the most recognized frameworks is Kirkpatrick’s evaluation model, which evaluates the effectiveness of education (2, 3). This model sorts the outcomes into four levels: reaction, learning, behavior changes, and results (4). In other words, it evaluates the outcome of any program in terms of its effect on the performance of those attending the program (2). The first level of the model (reaction) requires the participants to fill in a questionnaire after the course to express their overall impression of the program (5). This level of evaluation is supposed to assess how much the course fulfills the participants' expectations. The second level (learning) can be assessed utilizing questionnaires completed before and after participating in the program. This level evaluates how the participants can apply their learning in actual practice (6). Several workshops have been held in recent years to instruct medical students on research skills and scientific writing (7). The evaluation of such workshops is essential as it can demonstrate whether or not a particular workshop has led to improved performance among the participants (7). In this study, two levels of Kirkpatrick’s model were adopted to evaluate the effectiveness of a workshop on scientific writing held for medical students.

Objectives

The objectives of this study could be summarized as follows:

• To examine the background knowledge of students regarding research methodology;
• To assess the effectiveness of the workshop in improving the participants’ understanding of research processes;
• To determine any possible association between participation in a workshop on scientific research and an improvement in the level of knowledge among participants;
• To determine whether or not there is any significant difference between basic and clinical students in terms of their reaction to the workshop and the improvement in their knowledge.

Methods

Study Design: This was a cross-sectional study for 14 months (December 2021 to February 2023) conducted for medical sciences students. The sample size included all general medical students of Shahid Beheshti University of Medical Sciences who are in the basic level (from the 1st to 4th academic semester) and the clinical level (from the 5th semester to the end of the 13th semester). Data was collected through questionnaires completed by participants before and after the workshop (4, 8). Immediately after the workshop, the participants filled out a questionnaire on how satisfied they were with the workshop's content, design, and outcome. Two questionnaires (4-choice questions and self-assessment) were used to assess level two of the model. The questionnaire included several items concerning various aspects of paper composition. The score difference between the questionnaires for each participant demonstrated the efficiency of the training program.

Population: This study is approved by the ethics committee of Shahid Beheshti University of Medical Sciences (SBMU) IR.SBMU.SME.REC.1400.079. A total of seven workshops were held in this period, with
40 students attending each workshop, amounting to 280 students of medical sciences. All seven courses' teacher, conditions, and content were the same. Informed consent and being a student of medical sciences were the prerequisites for enrolment.

Workshop: The duration of each workshop was ten hours, which was held over two days for 5 hours to increase the workshop's efficiency. At the beginning of the first session, workshop outline was briefly introduced to ensure that the students had a general image of what they should expect from the training program. At the end of each workshop, a Q&A was held to discuss any issue that might have remained unclear for some students.

Sampling Tools: Before and after the workshop, evaluations were made to assess improvements in general understanding of the basic concepts of scientific research in medical sciences. The questionnaires were designed based on the content and objectives of the workshop. Regarding determining the validity and reliability of the questionnaire, after designing and preparing the questions, the questionnaire was approved by the supervisor. Therefore, it had validity, and due to the questions were designed according to the research objectives, it also had structural validity. Also, during the two stages of the experimental study and measurement on 20 students, the Cronbach's alpha value for the raw scores of the questions was determined to be 96%.

To evaluate the first level (reaction), a researcher-made questionnaire consisting of two parts was used. The first part was concerned with demographic information, such as name, age, and email of the participants, and the second part contained 12 questions. These questions concerned with the workshop's content, the lecturer, and the overall organization of the workshop. Answers were given on a scale of 1-5, with 1 being the highest and 5 being the lowest score based on Likert Scale.

To evaluate the second level (learning), two questionnaires were used. 1) A 4-choice question containing 25 questions about the workshop's content. 2) The self-assessment questionnaire based on the likert scale, which included 25 questions, was filled in before the workshop and one month later. Participants reported familiarity with these 25 questions on a scale of 1 to 9. If they did not know about the issue, they would score from 1 to 3. If they were familiar with the issue but not entirely, they would score from 4 to 6. Finally, if they were completely well-informed, they would have scored 7 to 9.

Statistical Analysis: Kolmogorov-Smirnov test assessed the normal distribution of reaction and learning levels items. A comparison of reaction and learning levels items between basic and clinical student groups was conducted using an Independent-Samples t-test. Also, comparing learning levels items before and after the workshop was conducted using a Paired-Samples t-test. The Significant level was assigned 0.05. The calculations were carried out using SPSS version 21.

Results

The demographics of the study. Of the 280 students of medical sciences participating in the current study, 162 (57.8%) were men (basic and clinical, 95 and
67 persons respectively), and 118 (42.2%) were women (basic and clinical, 65 and 53 persons respectively).

The normal test showed that reaction level items in each group had a normal distribution. Because p-values were above 0.05(P-Values for items 1 to 12 in per group: 0.23, 0.14, 0.20, 0.09, 0.08, 0.12, 0.34, 0.16, 0.20, 0.07, 0.32, 0.21; 0.11, 0.15, 0.27, 0.25, 0.19, 0.30, 0.26, 0.18, 0.23, 0.24, 0.19, 0.36). Also, scores of learning levels based on 4 choice questions and Self-assessment had normal distribution (P-Values for before and after per group: 0.20, 0.18, 0.20, 0.20; 0.20, 0.20, 0.20, 0.06).

Level 1: Reaction

As shown in Table 1, the average score for all the
12 questions concerning the first level is 1.59 ± 0.57, which shows that, in general, the workshop managed to leave a good impression on the basic students. In terms of the content of the workshop, the highest score was given to the applicability of the contents, with a mean score of 1.69 ± 0.58. Regarding the lecturer’s qualities, the highest score was obtained for the competency and expertise of the lecturer over the contents of the workshop, with a mean score of 1.83 ± 0.58. Finally, according to basic students, the maintenance of order and discipline throughout the workshop achieved the highest score concerning the organization of the workshop, with a mean of 1.69 ± 0.59. On the other hand, clinical students showed their overall reaction to the workshop with an average score of 1.63 ± 0.54, which shows a good level of satisfaction in general.

Regarding the workshop's content, the highest score was given to the diversity and novelty of the contents, with a mean score of 1.68 ± 0.48. Regarding the lecturer’s qualities, the highest score was given to the presentation and communication of the contents, with a mean score of 1.78±0.61. Finally, regarding the organization of the workshop, motivating and encouraging the learners to engage in more research activities received the highest score with a mean of 1.69 ± 0.48.

Level 2: Learning

As shown in Table 2, according to two questionnaires (4 choice and self-assessment), the values for the Paired-Samples t-test and p-value (P < 0.0001) indicate significant improvement within groups in the learning of both basic and clinical students. The results for the Independent-Samples t-test and p values
(P = 0.008, 0.011, and < 0.0001) didn’t indicate significant differences between basic and clinical students in the learning scores.

Table 1. Scores of Basic and Clinical Students for the First Questionnaire (Reaction Level)

*SD: Standard Deviation; ** Significant at 0.05

Table 2. Scores of Learning Levels

*SD: Standard Deviation;: **Significant at 0.05; P-Value <0.0001 is very small. For example, in self-assessment for clinical group, p-value for comparing of before and after workshop was 1.13×10-44; ***Independent-Samples t-test; ****Paired-Samples t-test

Discussion

Medical sciences and related areas of study are rapidly developing. To keep up with these developments, one needs to be actively involved in the academic discussion around their fields of interest by following the latest research. Workshops on academic research are frequently held to educate medical students on the basics of scientific writing and publication. The growing number of such workshops calls for objective methods to evaluate the efficiency of the training programs. The current study evaluated to assess of a workshop on academic research in terms of the participants’ reactions and learning.

Similar to our findings, other results were achieved by Pourjahromi et al., showing that age and sex had no significant impact on reaction, learning, and behavior changes (9).

The assessment of reaction in this study showed that participants were generally satisfied with the workshop's design, presentation, and content. This was reflected in the results obtained from the first questionnaire, which returned a mean value of 1.59 ± 0.57 (in basic students) and 1.63±0.54 (in clinical students). Dorri et al. reported the overall reaction of the nurses participating in training on cardiopulmonary resuscitation (CPR) as useful (10). Participants of a workshop held for a dental faculty on writing multiple-choice questions also reacted favorably to the overall design and content of the workshop (11). Similarly, a workshop for nurses on how to operate a DC shock device resulted in the general satisfaction of the participants with the content, lecturer, and educational facilities (9). The assessment of educational workshops for librarians by Shirazi et al showed the participants' general satisfaction, which encouraged the organizers to hold more workshops and encouraged more librarians to participate in the following workshops (12). In contrast, results from an evaluation of courses on coaching and refereeing by Bakhshandeh et al. showed that participants from some particular sports were not satisfied with the overall design and content of the programs, while courses on other sports had managed to elicit a more positive reaction from the participants. Bakhshandeh attributes the low level of satisfaction with some of the courses to the prioritization of financial issues over the quality of the education by some organizers (13).

The learning results showed significant improvement in the scores among both basic and clinical students. These results highlight the positive effect of participation in the workshop on the medical students’ knowledge of research methodology. To promote research, it is necessary to encourage "participation in the workshop" and increase research skills. The study on the efficiency of training on CPR indicated a desirable change in the staff's knowledge level (10).
The participants’ knowledge was also substantially increased due to a workshop on writing multiple-choice questions (11). The workshop held for librarians caused an increase in the general knowledge, expertise, and technical proficiency of the participants (12).

It can be argued that educational workshops generally improve participants’ knowledge, and participants react positively to such training programs. The only limitation of our study is the lack of a control group (the group that did not receive training through the workshop).

Conclusion

In the present study, all students showed a generally high level of satisfaction in terms of the content, the lecturer, and the workshop organization. As for the learning scores, both basic and clinical students significantly improved within groups. But there was no significant difference between the two groups (basic and clinical). A comprehensive evaluation of the quality of such training programs can help organizers improve the quality of their programs. Based on our findings, the high level of satisfaction experienced by the participants and their significantly improved knowledge of research methodology suggest that such workshops can indeed put medical students on the right track toward a productive academic and clinical career. On the other hand, more studies are needed to evaluate the efficiency of similar workshops to determine each program's positive and negative aspects.

In this study, a control group could have increased the validity of the results as participants could be compared with fellow students. Besides, more participants could increase the reliability of the survey by decreasing random errors. Finally, participants could be followed up for longer periods to evaluate the third and fourth levels of Kirkpatrick’s model (i.e., behavior change and results). For example, the number of papers published by the participants in the following months could indicate the workshop’s success in bringing about significant changes in behavior and tangible results among the participants.