Evaluation of Passing Scores in Semiotics: An Objective Structured Clinical Examination for Medical Students of Mashhad University of Medical Sciences, Mashhad, Iran, 2015

Makaren, Abbas; Mahdavifard, Hamid; Gholami, Hasan

doi:10.5812/sdme.59227

Evaluation of Passing Scores in Semiotics: An Objective Structured Clinical Examination for Medical Students of Mashhad University of Medical Sciences, Mashhad, Iran, 2015

Document Type : Original Article

Authors

Abbas Makaren

Hamid Mahdavifard

Hasan Gholami

Department of Medical Education, Mashhad University of Medical Sciences, Mashhad, IR Iran

https://doi.org/10.5812/sdme.59227

Abstract

Background Numerous exams are held at different levels and in different fields of medical sciences to evaluate students’ practical knowledge. In pass-fail exams where several examiners score the students, it is important to determine “the minimum passing score” or “the passing score” to determine whether students have passed or failed; this score is sometimes called the “cut-off point” or “standard score.” The objective structured clinical examination (OSCE) method is employed for the final assessment of medical students in Semiotics I in Mashhad University of Medical Sciences, Mashhad, Iran. The commonly used standard scoring method for this lesson is the fixed score method, which sometimes results in a discrepancy between educational management and the lecturers. Hence, the current study aims to compare 4 different methods—the Cohen, borderline-group, borderline regression, and Hofstee methods—of determining the passing score in the semiotics course and comparing the results with those of the fixed score method. Methods A 6-station OSCE was used to assess Semiotics I in Mashhad University of Medical Sciences in 2015. In the current study, in order to determine a standard scale for scoring the students, two forms, Forms 1 and 2, and a checklist were completed for each student. In Form 1, a 5-option Likert scale scoring system, graded from poor to excellent, was used. Data from Form 1 were analyzed using the borderline regression and borderline-group methods. Form 2 included 4 items and the collected data were analyzed using the Hofstee method. Data collected from both forms were analyzed, after the exams, using SPSS version 16. Results The cut-off point established by the Cohen method was very close to that of the common method. In other words, there was no significant difference between the cut-off point determined by the Cohen method (11.73) and that of the common method (12). The other study methods, however, such as borderline regression and borderline-group methods proposed higher cut-off points, which were significantly different from that of the common method: more students failed Semiotics I using these methods. The Hofstee method cannot be used in the OSCE, as the results were insignificant. Conclusions Because there was a significant difference in the number of students who passed the exam based on the fixed score and Cohen methods, and on the borderline-group and borderline regression methods, it is recommended that the latter methods not be widely employed. In addition, it is suggested that different methods should be used to define a mean standard passing score because, according to the statistics, an accurate and efficient estimator with minimum variances accuracy should be employed to evaluate population parameters, and the mean estimator would benefit from such advantages.

Keywords

Objective Structured Clinical Examination

Standard Score Determining Methods

Cut-Off Point

Mortaz Hejri S, Jalili M. Standard setting in medical education: fundamental concepts and emerging challenges. Med J Islam Repub Iran. 2014;28:34. [PubMed: 25250275].
Harden RM, Gleeson FA. Assessment of clinical competence using an objective structured clinical examination (OSCE). Med Educ. 1979;13(1):41–54. [PubMed: 763183].
Cusimano MD. Standard setting in medical education. Acad Med. 1996;71(10 Suppl):S112–20. [PubMed: 8940953].
Wayne D, Cohen E. Standard Setting in Competency Evaluation. ; 2010.
Norcini JJ. Setting standards on educational tests. Med Educ. 2003;37(5):464–9. [PubMed: 12709190].
Boulet JR, De Champlain AF, McKinley DW. Setting defensible performance standards on OSCEs and standardized patient examinations.Med Teach. 2003;25(3):245–9. doi: 10.1080/0142159031000100274.

[PubMed: 12881044].

Boursicot KA, Roberts TE, Pell G. Using borderline methods to compare passing standards for OSCEs at graduation across three medical schools. Med Educ. 2007;41(11):1024–31. doi: 10.1111/j.1365-

2923.2007.02857.x. [PubMed: 17973762].

Kane MT, Crooks TJ, Cohen AS. Designing and Evaluating StandardSetting Procedures for Licensure and Certification Tests. Adv Health Sci Educ Theory Pract. 1999;4(3):195–207. doi: 10.1023/A:1009849528247. [PubMed: 12386478].
Schoonheim-Klein M, Muijtjens A, Habets L, Manogue M, van der Vleuten C, van der Velden U. Who will pass the dental OSCE? Comparison of the Angoff and the borderline regression standard setting methods. Eur J Dent Educ. 2009;13(3):162–71. doi: 10.1111/j.1600-0579.2008.00568.x. [PubMed: 19630935].
Ben-David MF. AMEE Guide No. 18: Standard setting in student assessment. Med Teach.2009; 22(2): 120–30. doi: 10.1080/01421590078526.
Kaufman DM, Mann KV, Muijtjens AM, van der Vleuten CP. A comparison of standard-setting procedures for an OSCE in undergraduate medical education. Acad Med. 2000;75(3):267–71. [PubMed: 10724316].
Plake BS, Hambleton RK, Jaeger RM. A New Standard-Setting Method for Performance Assessments: The Dominant Profile Judgment Method and Some Field-Test Results. Educ Psychol Measure. 2016; 57(3): 400–11. doi: 10.1177/0013164497057003002.
Ricker KL. Setting cut-scores: A critical review of the Angoff and modified Angoff methods. Alberta J Educ Res. 2006;52(1):53.
Maurer TJ, Alexander RA, Callahan CM, Bailey JJ, Dambrot FH. Methodological and Psychometric Issues in Setting Cutoff Scores Using the Angoff Method. Personnel Psychol. 2006;44(2):235–62. doi:

10.1111/j.1744-6570.1991.tb00958.x.

Kramer A, Muijtjens A, Jansen K, Dusman H, Tan L, van der Vleuten C. Comparison of a rational and an empirical standard setting procedure for an OSCE. Objective structured clinical examinations. Med

Educ. 2003;37(2):132–9. [PubMed: 12558884].

Brandon PR. Conclusions About Frequently Studied Modified Angoff Standard-Setting Topics. Appl Measure Educ. 2004;17(1):59–88. doi:10.1207/s15324818ame1701_4.
Jalili M, Hejri SM, Norcini JJ. Comparison of two methods of standard setting: the performance of the three-level Angoff method. Med Educ. 2011;45(12):1199–208. doi: 10.1111/j.1365-2923.2011.04073.x.

[PubMed: 22122428].

Chinn RN, Hertz NR. Alternative Approaches to Standard Setting for Licensing and Certification Examinations. Appl Measure Educ. 2002;15(1):1–14. doi: 10.1207/s15324818ame1501_01.
Troncon LE. Clinical skills assessment: limitations to the introduction of an "OSCE" (Objective Structured Clinical Examination) in a traditional Brazilian medical school. Sao Paulo Med J. 2004;122(1):12–7. [PubMed: 15160521].
Wilkinson TJ, Newble DI, Frampton CM. Standard setting in an objective structured clinical examination: use of global ratings of borderline performance to determine the passing score. Med Educ.

2001;35(11):1043–9. [PubMed: 11703640].

Davison I, Bullock AD. Evaluation of the Introduction of the Objective Structured Public Health Examination. Birmingham: The University of Birmingham; 2007.
Smee SM, Blackmore DE. Setting standards for an objective structured clinical examination: the borderline group method gains ground on Angoff. Med Educ. 2001;35(11):1009–10. [PubMed: 11703635].
Humphrey-Murto S, MacFadyen JC. Standard setting: a comparison of case-author and modified borderline-group methods in a small-scale OSCE. Acad Med. 2002;77(7):729–32. [PubMed: 12114151].
Wood TJ, Humphrey-Murto SM, Norman GR. Standard setting in a small scale OSCE: a comparison of the Modified Borderline-Group Method and the Borderline Regression Method. Adv Health Sci Educ Theory Pract. 2006;11(2):115–22. doi: 10.1007/s10459-005-7853-1. [PubMed: 16729239].
Bandaranayake RC. Setting and maintaining standards in multiplechoice examinations: AMEE Guide No. 37. Med Teach. 2008;30(9-10):836–45. doi: 10.1080/01421590802402247. [PubMed: 19117221].
Searle J. Defining competency - the role of standard setting. Med Educ. 2000; 34(5):363–6. [PubMed: 10760120].
Jalili M, Mortazhejri S. Standard Setting for Objective Structured Clinical Exam Using Four Methods: Pre-fixed score, Angoff, Borderline Regression and Cohen’s. Strid Dev Med Educ. 2012;9(1):77–84.
Cizek GJ. Standard-Setting Guidelines. Educ Measure Issues Pract. 2005;15(1):13–21. doi: 10.1111/j.1745-3992.1996.tb00802.x.
Bay L. Standard Setting: A Guide to Establishing and Evaluating Performance Standards on Tests by Cizek, GJ, & Bunch, MB. Wiley Online Library; 2010.
Reid K, Dodds A. Comparing the borderline group and borderline regression approaches to setting Objective Structured Clinical Examination cut scores. J Contemp Med Educ. 2014;2(1):8. doi:

10.5455/jcme.20130921013523.

Barman A. Standard setting in student assessment: is a defensible method yet to come? Ann Acad Med Singapore. 2008;37(11):957–63. [PubMed: 19082204].
Taylor CA. Development of a modified Cohen method of standard setting. Med Teach. 2011; 33(12): e678–82. doi: 10.3109/0142159X.2011.611192. [PubMed: 22225450].