Background

Assessment is the requisite or essential part of medical education. Indian medical schools follow the traditional method of assessments, in which only a few structured cases, like long or short cases, are used to assess student's clinical skills (1), by which we cannot assess the total concepts covered in the curriculum framed and appears to be more theoretical and not aligned with the learning objectives. Introducing blueprint assessment as a technical component for skill assessment is an ideal solution, which helps reduce the standard drawbacks of assessment. A Blueprint is a calibrated or quality document that compiles the entire educational content and its outcome. It ensures to cover all the aspects of the curriculum and their educational domains during assessment (2). Blueprint specifies the subject content, topic, learning objectives, like skills, knowledge, and attitude, and the tools and methods to assess the content. Blueprinting in assessment is essential as it is a perfect source of evidence supporting the content validity. Hence, to assess fairly and provide clear guidelines to students, such as what is to be studied, what is to be learned, skills to be acquired, etc., blueprinting assessments could be a gold-standard method for evaluation (3). A well-structured blueprint refers to a valuable educational tool to improve the assessment quality in medical education. The present study was done to develop an assessment blueprint for clinical skill competencies in the Department of General Surgery for medical undergraduates. It included designing the blueprint for assessing clinical skills among undergraduates of general surgery and analysing the faculty's perception of the process of designing an assessment blueprint.

Objectives

The aim of this study was to develop a blueprint for the clinical assessment of undergraduate medical students in the department of general surgery at the Apollo Institute of Medical Sciences and Research. We aimed to improve undergraduate medical education by contributing to the development of standard assessment methods and acquiring feedback from the experts.

Methods

It is a one-year cross-sectional study. The curriculum coordinator and the faculty from the Department of General Surgery from two different medical schools (Apollo Institute of Medical Sciences and Research and Gandhi Medical College) participated in developing the blueprint. All the medical schools in India have adopted a competency-based curriculum, which was proposed by the National Medical Commission (NMC) in 2019. The medical school curriculum has three phases: phase 1 (one year of pre-clinical subjects, including anatomy, physiology, and biochemistry), phase 2 (one year of para-clinical subjects, including community medicine, forensic medicine, pathology, pharmacology, microbiology, and clinical rotations), phase three has two parts: two years of clinical subjects, including community medicine and medicine, and allied subjects, including psychiatry, dermatology, obstetrics and gynecology, and paediatrics.

Planning: This blueprint was framed to assess the clinical competencies of medical students for their complete clinical postings in general surgery (From phase 2 to phase 3) at Apollo Institute of Medical Sciences & Research, Hyderabad.

Sensitization workshop

With the consensus from the faculty, all the competencies suggested by the NMC for undergraduate standards were considered in developing the blueprint. The process started with a series of workshops conducted by the medical education team to sensitize the faculty following a template, which was provided to all the Faculty of General Surgery. The content details are described in Table 1.

The blueprint content was developed by ten specialists (from two different medical schools), the dean, and eight postgraduates.

Step 1: The subject experts defined the purpose and scope of the subject related to the blueprint. The curriculum contained competencies covering the entire syllabus included in phases 2 and 3. Firstly, the learning objectives were framed, and the subject experts prepared the teaching/learning (T/L) methods and the assessment tool for each competency.

Step 2: The weightage was identified for each competency based on two parameters, including the impact of the topic on health(I) and its frequency of occurrence(F) (I×F). Perceived impact of the topics on health was identified: (I)- 1) non-urgent, 2) serious but not life-threatening, and 3) life-threatening emergency- and frequency of occurrence of a particular disease (F): 1) rarely seen, 2) relatively common, and 3) very common. Based on the I×F, the topics were classified as “must know” with I×F of 6-9, “should know” with I×F of 3-4, and "good to know" with I×F of 1-2. The weightage coefficient for each competency was calculated as I×F/T. The sum of I and F is labelled as T. Number of questions of each competency was calculated by multiplying the weightage coefficient by the total number of items in the assessment. The total marks of each topic were calculated by multiplying the corresponding value of weightage (W) by the total marks the students were assessed as per the blueprint.

Step 3: Each competency was described in detail to ensure that the medical student is trained and assessed to meet the minimum level expected. The division of competency is described in Table 2. The assessment was divided into OSCE (Objective Structured Clinical Examination) and Case-Based Discussion. Further, OSCE was divided into history taking, physical examination, clinical procedure, clinical reasoning, and communication skills.

Step 4: A faculty feedback questionnaire on the assessment blueprint was prepared. The feedback questionnaire was peer-reviewed and validated by the medical education team and used a 5-point Linkert scale to grade their perceptions.

Statistical analysis was done using Cronbach’s alpha to test the scale's reliability. The value obtained was 0.585. The generally accepted rule is alpha between the 0.6-0.7 range indicating an acceptable level of reliability. To make an opinion about the blueprint designed, feedback was collected from the faculty of general surgery. The feedback form included certain standard validated questions to obtain valid perceptions of the faculty. The questions were framed to help us evaluate the importance or the need for the blueprint to assess the clinical skills of the medical students.

Results

Table 1 depicts a spreadsheet created with column 1showing the competency number followed by column 2 with the clinical competencies (systems). The competencies are part of the curriculum suggested by the NMC of India. The subject experts assigned the appropriate T/L method to each competency, which is mentioned in column 3, while two parameters were considered to calculate age weight: 1) the perceived impact of that competency in terms of its impact on health in society (I) (columns 4 and 2) Frequency of its occurrence (F) (column 5).

The product was calculated by multiplying I and F for each competency to give a weightage in column 6. For calculating the weightage in column 7, each competency's product (I×F) was divided into the total competencies (Tis the sum of I×F of all competencies).

In columns 8 and 9, the weightage of each competency is multiplied by the number of items for an assessment. Here, the assessment was an objective structured clinical examination (OSCE) with 15 stations with ten marks each, followed by the phase (Phases 2 and 3), at which the competency was tested in column 10. The competency can be taught and assessed in one or more phases. This covered the total syllabus designed as per NMC guidelines. It gathers certain information on whether the blueprint makes the examination fair, covers all the essential topics, and should be an integral part of the examination, giving students what to know and aligning with learning objectives, and helping as a guide to constructing clinical exam format. It has all the details of various competencies, which were aligned with the T/L method and the appropriate assessment, and the phase, in which that particular competency is taught and assessed. It also gives information on the marks allotted to a competency based on the competency's impact score and the frequency of its application in clinical practice.

Table 3 explains the curricular contents, based on which the product I×F was categorized into "must know," "should know," and "nice to know." Thus, in our study, 52% of the competencies fall into must know category, 31% should know, and 17% nice to know. This guides the examiners in selecting the tasks for assessment; in this phase, the curriculum should be analyzed, and the marks should be allotted.

In Table 2, the competencies (curricular content) are further divided based on various patient presentations and conditions. This will help the examiner to assess all the aspects of the curriculum using various domains of OSCE. Some of the patient presentations and conditions can be assessed by one or more domains of OSCE. The results of the feedback questions were measured in percentage (Figure 1). Thus, 97% of the faculty believed that blueprint makes examination fair, 93% agreed that it covered all the subject-related important topics and can make an integral part of the assessment, 90% believed that it improves the validity of exams, 87% expressed that blueprint acts as a guide to constructing clinical exam format and it is an assessment plan framed according to what one must know and learn, 73% believed that this blueprint is aligned with learning objectives and can be used to assess skills and in-depth knowledge, and70% felt that the weightage is given to public health important topics.

Figure 1: Measures of Faculty Feedback form in percentage

Table 1. Blueprint for Clinical Assessment in General Surgery for undergraduates

Table 2. Various patient presentations, conditions, and domains of OSCE

Table 3. Categorization of competences

Discussion

The present study was done as a practice to improve the validity of assessment and conduct a fair assessment with a standardized and guided blueprint in the Department of General Surgery. The results will help the faculty to make decisions on the student’s performance based on multiple data in formative assessments. Blueprint can be stated as a map for assessment ensuring the inclusion of all the aspects of the curriculum of different educational domains (4). Blueprint means “detailed action plan.” In brief, it acts as a link between assessment and learning objectives. Blueprint enables the faculty to set the question paper in such a way that it covers most of the important concepts and tests the students' in-depth knowledge. It gives a clear idea to frame the appropriate questions to test that particular objective, specific to the content unit, with specified marks (5). It matches the exact modality of assessment with different competencies of the course content. Blueprint helps to reduce major validity threats, such as underweighting national health important topics, assessment bias, such as question paper format being either too easy or too difficult or examiner bias.

In a study conducted by Patil et al. (2017) on introducing a blueprint in assessments in the Department of Pathology, the faculty believed that the blueprint helps as a guide in framing the question paper and improves the validity of assessments (100%) and about 89% of the faculty expressed that that blueprint must be a constitutive part of assessments, as it makes the assessment fair (6). Considering the
open-ended questions for both faculty and students, it was suggested that blueprinting must be used for all the examination phases, like summative, formative, internal, pre-final, and final university assessments. Our results are in accordance with this study. The result of the present study in the form of faculty’s feedback indicated that developing the blueprint for the assessment purpose is essential and will be a template that guides the faculty to design the question paper with aligned learning objectives, covering the curriculum content specified for the course. It was also stated that it highlights the must-know contents, helps assess the students' in-depth knowledge and gives equal weightage for all the topics to conduct a fair assessment. Our study results are in accordance with the previous studies (6). We finally found that the faculty believed that a blueprint can be made as an integral part of the assessment and helps fair examinations.

Conclusion

This study designed an assessment blueprint as a tedious process because it involved multiple stakeholders. It is a valid and reliable tool because it aligns clinical competencies and learning objectives with assessments and ensures that all topics are given appropriate weightage. The assessment blueprint makes the examination fair not only for the students but also for the examiner. This may contribute to quality medical education by improving assessment standards.