Document Type : Original Article
Authors
- Neetha Kundoor 1
- Faiz Hussain 2
- Sridhar Lingam 2
- Venkat Narsimha Reddy 3
- Rekha Arcot 4
- Harshita Bondugula 5
- Srilatha Bashetti 6
1 Department of Biochemistry, Apollo Institute of Medical sciences & Research, Hyderabad, Telangana, India
2 Department of General Surgery, Apollo Institute of Medical Sciences & Research, Hyderabad, Telangana, India
3 Department of General Surgery, Gandhi Medical College, Hyderabad, Telangana, India
4 Department of General Surgery, Dean, Apollo Institute of Medical Sciences & Research, Hyderabad, Telangana, India
5 Department of Dermatology, Apollo Institute of Medical Sciences & Research, Hyderabad, Telangana, India
6 Department of Biochemistry, Dr. Patnam Mahender Reddy Institute of Medical Sciences, Chevella, Telangana, India
Abstract
Background: Assessment is the requisite or essential part of medical education. A Blueprint is a calibrated or quality document that compiles the entire educational content and its outcome.
Objectives: The present study was done to develop an assessment blueprint for clinical skill competencies in the Department of General Surgery for medical undergraduates.
Methods: Blueprint was developed as an assessment tool for undergraduate medical students attending the third stage of general surgery, part 2. The feedback form was circulated among the departmental faculty to obtain their perception/ opinion about the designed blueprint. The feedback was then analysed.
Results: Feedback was 73- 83% satisfactory from the faculty’s opinion, and they stated that it aligned with learning objectives and public health. Important topics were considered, and they can be used to assess in-depth knowledge to improve clinical skills and be considered an integral part of assessments.
Conclusion: Blueprint could be an essential tool to conduct unbiased, ethical, and consistent examinations because it has a structured format that reflects the competencies that can be assessed. This may help quality medical education by improving assessment standards.
Highlights
Neetha Kundoor: (Google Scholar) (PubMed)
Faiz Hussain: (Google Scholar) (PubMed)
Sridhar Lingam: (Google Scholar) (PubMed)
Venkat Narsimha Reddy: (Google Scholar) (PubMed)
Rekha Arcot: (Google Scholar) (PubMed)
Harshita Bondugula: (Google Scholar) (PubMed)
Srilatha Bashetti: (Google Scholar) (PubMed)
Keywords
Background
Assessment is the requisite or essential part of medical education. Indian medical schools follow the traditional method of assessments, in which only a few structured cases, like long or short cases, are used to assess student's clinical skills (1), by which we cannot assess the total concepts covered in the curriculum framed and appears to be more theoretical and not aligned with the learning objectives. Introducing blueprint assessment as a technical component for skill assessment is an ideal solution, which helps reduce the standard drawbacks of assessment. A Blueprint is a calibrated or quality document that compiles the entire educational content and its outcome. It ensures to cover all the aspects of the curriculum and their educational domains during assessment (2). Blueprint specifies the subject content, topic, learning objectives, like skills, knowledge, and attitude, and the tools and methods to assess the content. Blueprinting in assessment is essential as it is a perfect source of evidence supporting the content validity. Hence, to assess fairly and provide clear guidelines to students, such as what is to be studied, what is to be learned, skills to be acquired, etc., blueprinting assessments could be a gold-standard method for evaluation (3). A well-structured blueprint refers to a valuable educational tool to improve the assessment quality in medical education. The present study was done to develop an assessment blueprint for clinical skill competencies in the Department of General Surgery for medical undergraduates. It included designing the blueprint for assessing clinical skills among undergraduates of general surgery and analysing the faculty's perception of the process of designing an assessment blueprint.
Objectives
The aim of this study was to develop a blueprint for the clinical assessment of undergraduate medical students in the department of general surgery at the Apollo Institute of Medical Sciences and Research. We aimed to improve undergraduate medical education by contributing to the development of standard assessment methods and acquiring feedback from the experts.
Methods
It is a one-year cross-sectional study. The curriculum coordinator and the faculty from the Department of General Surgery from two different medical schools (Apollo Institute of Medical Sciences and Research and Gandhi Medical College) participated in developing the blueprint. All the medical schools in India have adopted a competency-based curriculum, which was proposed by the National Medical Commission (NMC) in 2019. The medical school curriculum has three phases: phase 1 (one year of pre-clinical subjects, including anatomy, physiology, and biochemistry), phase 2 (one year of para-clinical subjects, including community medicine, forensic medicine, pathology, pharmacology, microbiology, and clinical rotations), phase three has two parts: two years of clinical subjects, including community medicine and medicine, and allied subjects, including psychiatry, dermatology, obstetrics and gynecology, and paediatrics.
Planning: This blueprint was framed to assess the clinical competencies of medical students for their complete clinical postings in general surgery (From phase 2 to phase 3) at Apollo Institute of Medical Sciences & Research, Hyderabad.
Sensitization workshop
With the consensus from the faculty, all the competencies suggested by the NMC for undergraduate standards were considered in developing the blueprint. The process started with a series of workshops conducted by the medical education team to sensitize the faculty following a template, which was provided to all the Faculty of General Surgery. The content details are described in Table 1.
The blueprint content was developed by ten specialists (from two different medical schools), the dean, and eight postgraduates.
Step 1: The subject experts defined the purpose and scope of the subject related to the blueprint. The curriculum contained competencies covering the entire syllabus included in phases 2 and 3. Firstly, the learning objectives were framed, and the subject experts prepared the teaching/learning (T/L) methods and the assessment tool for each competency.
Step 2: The weightage was identified for each competency based on two parameters, including the impact of the topic on health(I) and its frequency of occurrence(F) (I×F). Perceived impact of the topics on health was identified: (I)- 1) non-urgent, 2) serious but not life-threatening, and 3) life-threatening emergency- and frequency of occurrence of a particular disease (F): 1) rarely seen, 2) relatively common, and 3) very common. Based on the I×F, the topics were classified as “must know” with I×F of 6-9, “should know” with I×F of 3-4, and "good to know" with I×F of 1-2. The weightage coefficient for each competency was calculated as I×F/T. The sum of I and F is labelled as T. Number of questions of each competency was calculated by multiplying the weightage coefficient by the total number of items in the assessment. The total marks of each topic were calculated by multiplying the corresponding value of weightage (W) by the total marks the students were assessed as per the blueprint.
Step 3: Each competency was described in detail to ensure that the medical student is trained and assessed to meet the minimum level expected. The division of competency is described in Table 2. The assessment was divided into OSCE (Objective Structured Clinical Examination) and Case-Based Discussion. Further, OSCE was divided into history taking, physical examination, clinical procedure, clinical reasoning, and communication skills.
Step 4: A faculty feedback questionnaire on the assessment blueprint was prepared. The feedback questionnaire was peer-reviewed and validated by the medical education team and used a 5-point Linkert scale to grade their perceptions.
Statistical analysis was done using Cronbach’s alpha to test the scale's reliability. The value obtained was 0.585. The generally accepted rule is alpha between the 0.6-0.7 range indicating an acceptable level of reliability. To make an opinion about the blueprint designed, feedback was collected from the faculty of general surgery. The feedback form included certain standard validated questions to obtain valid perceptions of the faculty. The questions were framed to help us evaluate the importance or the need for the blueprint to assess the clinical skills of the medical students.
Results
Table 1 depicts a spreadsheet created with column 1showing the competency number followed by column 2 with the clinical competencies (systems). The competencies are part of the curriculum suggested by the NMC of India. The subject experts assigned the appropriate T/L method to each competency, which is mentioned in column 3, while two parameters were considered to calculate age weight: 1) the perceived impact of that competency in terms of its impact on health in society (I) (columns 4 and 2) Frequency of its occurrence (F) (column 5).
The product was calculated by multiplying I and F for each competency to give a weightage in column 6. For calculating the weightage in column 7, each competency's product (I×F) was divided into the total competencies (Tis the sum of I×F of all competencies).
In columns 8 and 9, the weightage of each competency is multiplied by the number of items for an assessment. Here, the assessment was an objective structured clinical examination (OSCE) with 15 stations with ten marks each, followed by the phase (Phases 2 and 3), at which the competency was tested in column 10. The competency can be taught and assessed in one or more phases. This covered the total syllabus designed as per NMC guidelines. It gathers certain information on whether the blueprint makes the examination fair, covers all the essential topics, and should be an integral part of the examination, giving students what to know and aligning with learning objectives, and helping as a guide to constructing clinical exam format. It has all the details of various competencies, which were aligned with the T/L method and the appropriate assessment, and the phase, in which that particular competency is taught and assessed. It also gives information on the marks allotted to a competency based on the competency's impact score and the frequency of its application in clinical practice.
Table 3 explains the curricular contents, based on which the product I×F was categorized into "must know," "should know," and "nice to know." Thus, in our study, 52% of the competencies fall into must know category, 31% should know, and 17% nice to know. This guides the examiners in selecting the tasks for assessment; in this phase, the curriculum should be analyzed, and the marks should be allotted.
In Table 2, the competencies (curricular content) are further divided based on various patient presentations and conditions. This will help the examiner to assess all the aspects of the curriculum using various domains of OSCE. Some of the patient presentations and conditions can be assessed by one or more domains of OSCE. The results of the feedback questions were measured in percentage (Figure 1). Thus, 97% of the faculty believed that blueprint makes examination fair, 93% agreed that it covered all the subject-related important topics and can make an integral part of the assessment, 90% believed that it improves the validity of exams, 87% expressed that blueprint acts as a guide to constructing clinical exam format and it is an assessment plan framed according to what one must know and learn, 73% believed that this blueprint is aligned with learning objectives and can be used to assess skills and in-depth knowledge, and70% felt that the weightage is given to public health important topics.
Figure 1: Measures of Faculty Feedback form in percentage
Table 1. Blueprint for Clinical Assessment in General Surgery for undergraduates
S.NO |
Competency No. |
SYSTEM/TOPIC |
T/L method |
Impact (I) |
Frequency of Occurrence (F) |
I×F |
W=I×F/T |
NUMBER OF QUESTIONS=W×15 |
Marks=W× 150 |
Rounding off (Culminate) closer to the next value |
Phase 2 |
Phase3 (Part 1) |
Phase3 (Part 2) |
1 |
SU2.3 |
Communication with and counsel patients and families about the treatment and prognosis of shock and show empathy and care |
Role Play |
3 |
2 |
6 |
0.052 |
0.78 |
7.8 |
8 |
|
|
√ |
2 |
SU3.3 |
Counseling patients and family/friends for blood transfusion and blood donation |
Role Play |
2 |
3 |
6 |
0.052 |
0.78 |
7.8 |
8 |
√ |
|
|
3 |
SU8.2 |
Showing professionalism and empathy with the patient during general surgery |
Role Play |
1 |
3 |
3 |
0.026 |
0.39 |
3.9 |
4 |
√ |
|
|
4 |
SU9.3 |
Informing the results of surgical examinations and proper counseling with the patient |
Role Play |
1 |
3 |
3 |
0.026 |
0.39 |
3.9 |
4 |
√ |
|
|
5 |
SU10.2 |
Describing the steps and obtaining informed consent in a simulated environment |
Role Play |
1 |
3 |
3 |
0.026 |
0.39 |
3.9 |
4 |
√ |
|
|
6 |
SU10.4 |
Performing basic surgical skills such as first aid, including suturing and performing minor surgeries in a simulated environment |
DOAP |
3 |
2 |
6 |
0.052 |
0.78 |
7.8 |
8 |
|
|
√ |
7 |
SU11.3 |
Demonstrating airway maintenance on a mannequin or similar |
DOAP |
3 |
2 |
6 |
0.052 |
0.78 |
7.8 |
8 |
|
|
√ |
8 |
SU13.4 (Organ Transplant) |
Counseling patients and relatives in the field of organ donation in a simulated environment |
Role Play |
1 |
1 |
1 |
0.008 |
0.12 |
1.2 |
2 |
√ |
|
|
9 |
SU14.4 (Suturing) |
Demonstrating asepsis and suturing techniques in a simulated environment |
DOAP |
2 |
3 |
6 |
0.052 |
0.78 |
7.8 |
8 |
|
|
√ |
10 |
SU17.2 |
Demonstrating the steps in Basic Life Support Transportation of an injured patient in a simulated environment |
DOAP, SNAPPS |
3 |
3 |
9 |
0.078 |
1.17 |
11.7 |
12 |
√ |
|
|
11 |
SU17.10 (Chest Trauma) |
Demonstrating airway maintenance Recognition and management of tension pneumothorax, hemothorax, and chest flap in a simulated environment |
DOAP |
3 |
3 |
9 |
0.078 |
1.17 |
11.7 |
12 |
|
√ |
|
12 |
SU18.3 (Swelling) |
Explaining and showing the clinical examination of the surgical patient, including swelling, and ordering the relevant tests for diagnosis. Describing and discussing an appropriate treatment plan. |
SNAPPS |
2 |
2 |
4 |
0.035 |
0.525 |
5.25 |
5 |
√ |
|
|
13 |
SU22.3 (Thyroid Disorders) |
Demonstrating and documenting the correct clinical examination of thyroid swellings and discussing the differential diagnosis and their management |
SNAPPS |
2 |
3 |
6 |
0.052 |
0.78 |
7.8 |
8 |
|
√ |
|
14 |
SU25.4 (Breast Counseling) |
Counseling the patient and obtaining informed consent for the treatment of malignant conditions of the breast |
Role Play |
3 |
2 |
6 |
0.052 |
0.78 |
7.8 |
8 |
|
√ |
|
15 |
SU25.4 (Breast Examination) |
Demonstrating the correct method of breast palpation for breast swelling on a mannequin or a similar condition |
DOAP |
1 |
2 |
2 |
0.017 |
0.255 |
2.55 |
2 |
|
√ |
|
16 |
SU27.8 (Lymphatic System) |
Demonstrating the correct examination of the lymphatic system |
SNAPPS |
1 |
2 |
2 |
0.017 |
0.255 |
2.55 |
2 |
|
√ |
|
17 |
SU27.8 (Hernia) |
Demonstrating the correct technique to examine the patient with the hernia and identify different types of hernias |
SNAPPS |
1 |
3 |
3 |
0.026 |
0.39 |
3.9 |
4 |
|
√ |
|
18 |
SU28.9 (Abdomen) |
Demonstrating the correct technique of examination of a patient with stomach disorders |
SNAPPS |
1 |
3 |
3 |
0.026 |
0.39 |
3.9 |
4 |
|
|
√ |
19 |
SU29.10 (Rectal) |
Demonstrating a digital rectal examination of the prostate on a mannequin or a similar condition |
DOAP SNAPPS |
2 |
2 |
4 |
0.035 |
0.525 |
5.25 |
5 |
|
√ |
|
20 |
SU30.5 (Scortal) |
Examination of scrotal swelling |
SNAPPS |
3 |
3 |
9 |
0.078 |
1.17 |
11.7 |
12 |
|
√ |
|
21 |
SU27.6 Vascular System |
Examination of the arterial system |
SNAPPS |
1 |
2 |
2 |
0.017 |
0.255 |
2.55 |
2 |
|
|
√ |
22 |
SU27.6 Vascular System |
Examination of venous system |
SNAPPS |
3 |
2 |
6 |
0.052 |
0.78 |
7.8 |
8 |
|
|
√ |
23 |
|
Examination of ulcer |
SNAPPS |
3 |
3 |
9 |
0.078 |
1.17 |
11.7 |
12 |
|
√ |
|
|
|
|
|
|
|
114 |
|
14.805 |
148.05 |
150 |
|
|
|
Table 2. Various patient presentations, conditions, and domains of OSCE
Outcomes |
Presentation |
Various patient presentations and conditions and the domains of OSCE |
|||||||
Conditions |
History Taking |
Physical Examination |
Clinical Procedure |
Clinical Reasoning |
Data Interpretation |
Communication/ |
Training Phase |
||
Communicating with and counseling patients and families about the treatment and prognosis of shock and demonstrating empathy and care |
|
Septic shock |
|
|
|
|
|
|
|
Counseling patients and families/ friends about blood transfusion and |
|
Counseling the patient/Family member on the significance of blood donation |
|
|
|
|
|
√ |
Phase 2 |
Obtaining consent for transfusion |
|
|
|
|
|
√ |
Phase 2 |
||
Showing professionalism |
Peri-operative care: Minor procedures |
Informing the patient |
|
|
|
|
|
√ |
Phase 3 |
Obtaining consent for surgery |
|
|
|
|
|
√ |
Phases 2 |
||
Informing the patient of the results of surgical examinations and advising the patient appropriately |
The routine investigation done before surgical procedure; Investigations related to any kind of malignancy |
Investigation of the results for minor procedures, major procedures, and malignancy |
|
|
|
|
|
√ |
Phases 2 |
Interpretation of Investigation results |
|
|
|
|
√ |
√ |
|
|
|
Performing basic surgical skills such as first aid, including suturing and performing minor surgeries in a simulated environment, Demonstration of asepsis and suturing techniques |
Minor Injury Abscess |
Hand washing Glowing and gowning Donning and doffing |
|
|
√ |
|
|
|
Phase 2 |
Preparation of antiseptic field |
|
|
√ |
|
|
|
Phase 2 |
||
Incision and closure of skin and subcutaneous tissue |
|
|
√ |
|
|
|
Phase 3 |
||
Intercostal drain |
|
|
√ |
|
|
|
Phase 3 |
||
Appropriate selection of instruments for various minor procedures |
|
|
√ |
√ |
√ |
√ |
Phase 3 |
||
Surgical drainage of abscess |
|
|
√ |
|
|
|
Phase 3 |
||
Catheterization (Male/Female) |
|
|
√ |
|
|
|
Phase 3 |
||
Wound dressing |
|
|
√ |
|
|
|
Phase 3 |
||
Transplantation |
Awareness of Organ Donation |
Discussion about organ donation with the family of a dying patient |
|
|
|
|
|
√ |
Phase 2,3 (Part 1) |
Trauma |
Injured patient |
BLS |
|
|
√ |
|
|
|
Phase 2/3 (Part 1) |
Airway management |
|
|
√ |
√ |
|
|
Phase 3 (Part 1,2) |
||
Skin and subcutaneous tissue |
Subcutaneous swelling Ulcer |
Lipoma, Neurofibroma keloid, Sebaceous cyst |
√ |
√ |
|
√ |
|
|
Phase -2,3 (Part 1) |
Breast |
Breast lumps and nipple discharge, Acute |
Breast lumps: benign and malignant Breast abscess |
√ |
√ |
|
|
√ |
|
Phase 3 (Parts 1 |
|
Patient counseling and obtaining informed consent for the treatment of malignant breast diseases |
|
|
|
|
|
√ |
Phase 3 (Part 1) |
|
Vascular and lymphatic diseases |
Leg ulceration, Varicose veins, Limb Ischemia: Acute and Chronic, Lymphadenitis: Acute and Chronic |
Diabetic ulcer, Venous insufficiency, Vascular injury, Thrombotic |
√ |
√ |
|
√ |
√ |
|
Phase 3 (Part 1,2) |
Abdomen |
Abdominal pain, Change in bowel habit |
Appendicitis |
√ |
√ |
√ |
√ |
|
√ |
Phase 2, 3 (Part 1,2) |
Hernias |
Abdominal swelling |
Inguinal hernias |
√ |
√ |
|
√ |
|
√ |
Phases 2 and 3 (Part 1,2) |
Rectal |
Abdominal pain, Change in bowel habit, Gastrointestinal, Hemorrhage |
Hemorrhoids and perianal disease, Anal fissures |
√ |
√ |
|
√ |
|
√ |
Phase 2, 3 (Part 1,2) |
Urinary system |
Loin pain, Haematuria, Lower urinary tract symptoms |
Urethral Strictures, Urinary calculus disease; Urinary tract infection |
√ |
√ |
√ |
√ |
|
√ |
Phase 3 (Part 1,2) |
Penis, testis, and scrotum |
Scrotal swellings, Testicular pain |
Hydrocele, Sebaceous cyst, Orchitis, Epididyamo orchitis |
√ |
√ |
√ |
√ |
|
√ |
Phase 3 (Parts 1 |
Table 3. Categorization of competences
I×F |
Category |
Percentage |
Phase |
||
Phase 2 |
Phase 3 (Part 1) |
Phase 3 (Part 2) |
|||
6-9 |
Must know |
52% |
2 |
5 |
5 |
3-4 |
Should know |
31% |
4 |
2 |
1 |
1-2 |
Nice to know |
17% |
1 |
2 |
1 |
Total |
|
100% |
|
|
|
Discussion
The present study was done as a practice to improve the validity of assessment and conduct a fair assessment with a standardized and guided blueprint in the Department of General Surgery. The results will help the faculty to make decisions on the student’s performance based on multiple data in formative assessments. Blueprint can be stated as a map for assessment ensuring the inclusion of all the aspects of the curriculum of different educational domains (4). Blueprint means “detailed action plan.” In brief, it acts as a link between assessment and learning objectives. Blueprint enables the faculty to set the question paper in such a way that it covers most of the important concepts and tests the students' in-depth knowledge. It gives a clear idea to frame the appropriate questions to test that particular objective, specific to the content unit, with specified marks (5). It matches the exact modality of assessment with different competencies of the course content. Blueprint helps to reduce major validity threats, such as underweighting national health important topics, assessment bias, such as question paper format being either too easy or too difficult or examiner bias.
In a study conducted by Patil et al. (2017) on introducing a blueprint in assessments in the Department of Pathology, the faculty believed that the blueprint helps as a guide in framing the question paper and improves the validity of assessments (100%) and about 89% of the faculty expressed that that blueprint must be a constitutive part of assessments, as it makes the assessment fair (6). Considering the
open-ended questions for both faculty and students, it was suggested that blueprinting must be used for all the examination phases, like summative, formative, internal, pre-final, and final university assessments. Our results are in accordance with this study. The result of the present study in the form of faculty’s feedback indicated that developing the blueprint for the assessment purpose is essential and will be a template that guides the faculty to design the question paper with aligned learning objectives, covering the curriculum content specified for the course. It was also stated that it highlights the must-know contents, helps assess the students' in-depth knowledge and gives equal weightage for all the topics to conduct a fair assessment. Our study results are in accordance with the previous studies (6). We finally found that the faculty believed that a blueprint can be made as an integral part of the assessment and helps fair examinations.
Conclusion
This study designed an assessment blueprint as a tedious process because it involved multiple stakeholders. It is a valid and reliable tool because it aligns clinical competencies and learning objectives with assessments and ensures that all topics are given appropriate weightage. The assessment blueprint makes the examination fair not only for the students but also for the examiner. This may contribute to quality medical education by improving assessment standards.
- Patil SY, Hashilkar NK, Hungund BR. Blueprinting in assessment: how much is imprinted in our practice? J Educ Res Med Teach. 2014;2(1):4–6.
- Sood R, Paul V, Mittal S, Adkoli B, Sahni P, Kharbanda O. Assessment in medical education: trends and tools. New Delhi: KL Wig CMET, AIIMS; 1995.
- Adkoli BV, Deepak KK. Blue printing in assessment. In: Singh T, Anshu, editors. Principles of assessment in medical education. New Delhi: Jaypee Publishers; 2012: 205–13.
- Adkoli B. Attributes of a good question paper. In: Sood R, editor. Assessment in Medical Education: Trends and Tools. New Delhi: KL Wig Center for Medical Education and Technology, AIIMS; 1995
- Hamdy H. Blueprinting in medical education. N Engl J Med. 2007;356:387‑95. doi:10.1056/NEJMra054784.
- Sunita Y Patil, Manasi G, Hema BB, Ashwini R. Blueprinting in assessment: Atool to increase the validity of undergraduate written examinations in pathology. Int J Appl Basic Med Res. 2015 Aug;5(Suppl 1):S76-9. doi: 10.4103/2229-516X.162286. [PMID: 26380218] [PMCID: PMC4552073]