Ranking of essential nursing skills from highest to lowest by average degree of complexity score in high‐complexity category.
Abstract
Introduction: The objective structured clinical examination (OSCE) has been adopted by many universities for the assessment of healthcare competencies and as a formative teaching tool in both undergraduate and postgraduate nursing education programs. This pilot study evaluates the validity of OSCE checklists to be used in first‐year undergraduate nurse practice education.
Keywords
- objective structured clinical examination
- development
- checklist
- content validity index
- nursing
1. Introduction
The objective structured clinical examination (OSCE) was originally developed for medical education in Scotland by Harden and colleagues in 1975 [1], but has now been widely accepted as a “fit‐for‐purpose” instrument for measuring clinical skills competency in healthcare education [2, 3]. The OSCE is defined as “an approach to the assessment of clinical competence in which the components of competence are assessed in a well‐planned or structured way with attention being paid to objectivity” [4].
The OSCE has been adopted by many universities for the assessment of healthcare competencies, and it is generally accepted as a valid assessment tool and as a formative teaching approach in both undergraduate and postgraduate nursing education programs [5–8]. The benefits accrued by using the OSCE tool include the development of students’ confidence [5]; the preparation of students for clinical practice [9]; the achievement of deeper and more meaningful learning [7]; the ability to provide students with feedback on their clinical skills performance; and additionally its enables students to identify their strengths and weaknesses in clinical skills [6].
The OSCE typically consists of a circuit or series of short assessment tasks, each of which is assessed by an examiner using a predetermined, objective marking scheme to make the assessment of clinical skills more objective rather than subjective [10]. In an OSCE, each student has to demonstrate specific skills and behaviors in a simulated environment. The OSCE acronym has itself evolved over the years, and there are now many variations, for example, Group Objective Structured Clinical Examination [11], Objective Structured Video Exam [12], Objective Structured Assessment of Technical Skill [13], Objective Structured Teaching Encounter [14], etc. The latter number of variations on the OSCE has evolved because of its utility and applicability as an assessment and teaching tool in nursing and interprofessional education [15].
The development of new criteria for assessing clinical skills requires critical scrutiny to ensure that the validity and reliability of each assessment are maximized [10]. Validity focuses on whether a test actually succeeds in addressing the competencies it is designed to test [16]. The assessment checklists used in the OSCE are developed according to evidence‐based practice guidelines and standards of nursing care to establish content validity [17]. Evaluating content validity is a critical early step in enhancing the construct validity of an instrument, and therefore, content validation is an important topic for clinicians and researchers who require high‐quality measurements [18]. The content validity index (CVI) based on expert ratings of relevance is the most widely used method among nurse researchers of quantifying content validity for multi‐item scales [10].
This study describes two interconnected methodological phases that could be considered and implemented when developing and establishing the CVI of a checklist, which is designed to measure nursing student performance during clinical skills assessment using OSCE. Checklists in an OSCE provide an ideal method for assessing skills that require a series of steps that should be completed with consistency and continuity each time the skill is performed [20].
2. Methods
The checklist for the OSCE was developed in three methodological and chronological phases (Figure 1).
In first phase, a comprehensive search of the literature relating to OSCE in Slovenian nursing was conducted and no published research examining the use of OSCE in Slovenian nursing curriculum was found.
In second phase, the degree of complexity (DOC) phase, a 10‐point scale was created and used to evaluate the DOC for each essential nursing skill as perceived by nursing educators. All essential nursing skills that were included in this study were part of a first‐year curriculum in the practical nursing education at one of University in Slovenia.
The DOC has 10 levels in which 1 represents “very low complexity” and 10 represents “very high complexity.” The DOC scores for each essential nursing skill were classified into three categories. A score between 1 and 4 belonged to the low complexity category; a score between 4 and 8 belonged to the medium complexity category, and a score between 8 and 10 belonged to the high‐complexity category.
Phase three, the content validity index (CVI) phase, included various measurements [18]. Educators evaluated each item in nursing procedures by using a four‐point Likert‐type ordinal scale in which 1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, 4 = highly relevant, nurse. Two metrics were calculated in the scope of CVI analysis: (1) Item Content Validity Index (I‐CVI) and (2) Content Validity Index Average (S‐CVI/Ave). The first metric (I‐CVI) represents the count of all items in the essential nursing skill, which were rated with 3 or 4 divided by the total number of nursing educators. The S‐CVI/Ave was calculated after summing all I‐CVI numbers and dividing them by the number of items in the essential nursing skill [18].
3. Results
Nursing educators (n = 12) systematically evaluated seventy‐two essential nursing skills using a 10‐point DOC scale. The average DOC score for each procedure was then rearranged into one of three categories (low, medium and high). Twelve essential nursing skills with an average DOC score of 3.44 (95% confidence interval (CI): 3.07–3.82) were ranked in the low complexity category, forty‐six essential nursing skills with an average DOC score of 5.84 (CI: 5.55–6.12) in the medium complexity category, and fourteen essential nursing skills with an average DOC score of 8.54 (CI: 8.36–8.71) in the high‐complexity category.
Table 1 presents the essential nursing skills (n = 14) with average DOC scores from highest to lowest in the high‐complexity category. Peripheral cannula insertion and female urinary catheterization were estimated with the highest average DOC score (9.00 or “very high complexity”). In the DOC, phase essential nursing skills were also arranged into areas in nursing.
Essential nursing skills | Areas in nursing | Average degree of complexity score |
---|---|---|
Peripheral cannula insertion | Diagnostic/therapeutic essential nursing skills | 9.00 |
Urinary catheterization: female | Elimination | 9.00 |
Suctioning the nasopharyngeal airway | Respiratory care | 8.75 |
Medication: injection of intravenous drugs | Medical management | 8.75 |
Suctioning the oropharyngeal airway | Respiratory care | 8.67 |
Tracheostomy: suctioning a patient | Respiratory care | 8.67 |
Endotracheal suctioning of the adult intubated patient with open suction systems | Respiratory care | 8.67 |
Venipuncture | Diagnostic/therapeutic essential nursing skills | 8.50 |
Cleaning infected wound | Diagnostic/therapeutic essential nursing skills | 8.50 |
Nursing care of tracheostomy | Respiratory care | 8.33 |
Insertion of a nasogastric tube | Nutrition | 8.25 |
Pressure Ulcer Treatment | Diagnostic/therapeutic essential nursing skills | 8.18 |
Mouth care in unconscious patients | Personal hygiene | 8.17 |
Rinsing infected wound | Diagnostic/therapeutic essential nursing skills | 8.08 |
Nursing essential nursing skills (n = 6) with the highest average DOC score in each separate nursing areas were used for further estimate by CVI phase. Eleven nursing educators estimate I‐CVI for each essential nursing skill with different number of items (range from 28 to 58). For peripheral cannula insertion, I‐CVI was calculated for 39 of items and ranged from 0.82 to 1.00, which represents a good content validity. None of the items were deleted during CVI because they met agreements recommended by Polit and colleagues (Table 2 and Figure 2) [19].
Item | E1 | E2 | E3 | E4 | E5 | E6 | E7 | E8 | E9 | E10 | E11 | Number of experts | I‐CVI |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | 4 | 2 | 4 | 4 | 4 | 4 | 4 | 4 | 1 | 4 | 4 | 11 | 0.82 |
2 | 4 | 2 | 3 | 4 | 4 | 4 | 4 | 4 | 1 | 4 | 4 | 11 | 0.82 |
3 | 4 | 3 | 4 | 4 | 4 | 4 | 4 | 4 | 2 | 4 | 4 | 11 | 0.91 |
4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 11 | 1.00 |
5 | 4 | 3 | 4 | 4 | 4 | 4 | 3 | 4 | 3 | 4 | 4 | 11 | 1.00 |
6 | 4 | 4 | 3 | 4 | 4 | 4 | 3 | 3 | 3 | 4 | 4 | 11 | 1.00 |
7 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 11 | 1.00 |
8 | 3 | 2 | 3 | 4 | 1 | 4 | 3 | 4 | 3 | 4 | 4 | 11 | 0.82 |
9 | 4 | 3 | 4 | 4 | 4 | 4 | 4 | 4 | 2 | 4 | 4 | 11 | 0.91 |
10 | 4 | 3 | 4 | 4 | 4 | 4 | 4 | 3 | 4 | 4 | 4 | 11 | 1.00 |
11 | 4 | 2 | 4 | 4 | 3 | 4 | 4 | 3 | 3 | 4 | 4 | 11 | 0.91 |
12 | 4 | 3 | 4 | 4 | 3 | 4 | 4 | 4 | 2 | 4 | 4 | 11 | 0.91 |
13 | 3 | 2 | 3 | 4 | 3 | 4 | 3 | 4 | 2 | 4 | 3 | 11 | 0.82 |
14 | 4 | 3 | 4 | 4 | 4 | 4 | 4 | 4 | 3 | 4 | 4 | 11 | 1.00 |
15 | 4 | 4 | 4 | 4 | 2 | 4 | 4 | 3 | 4 | 4 | 4 | 11 | 0.91 |
16 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 11 | 1.00 |
17 | 3 | 4 | 3 | 4 | 3 | 4 | 4 | 4 | 2 | 4 | 3 | 11 | 0.91 |
18 | 4 | 3 | 4 | 4 | 4 | 4 | 4 | 4 | 3 | 4 | 4 | 11 | 1.00 |
19 | 4 | 3 | 4 | 4 | 4 | 4 | 3 | 4 | 4 | 4 | 4 | 11 | 1.00 |
20 | 4 | 3 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 11 | 1.00 |
21 | 4 | 3 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 11 | 1.00 |
22 | 4 | 3 | 4 | 4 | 2 | 4 | 4 | 4 | 4 | 4 | 4 | 11 | 0.91 |
23 | 4 | 3 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 11 | 1.00 |
24 | 4 | 3 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 11 | 1.00 |
25 | 3 | 3 | 4 | 4 | 4 | 4 | 4 | 4 | 3 | 4 | 4 | 11 | 1.00 |
26 | 4 | 3 | 4 | 4 | 4 | 4 | 4 | 4 | 3 | 4 | 4 | 11 | 1.00 |
27 | 4 | 3 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 11 | 1.00 |
28 | 4 | 3 | 3 | 4 | 1 | 4 | 4 | 4 | 3 | 4 | 4 | 11 | 0.91 |
29 | 4 | 3 | 4 | 4 | 1 | 4 | 4 | 4 | 4 | 4 | 4 | 11 | 0.91 |
30 | 4 | 3 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 11 | 1.00 |
31 | 4 | 2 | 4 | 4 | 4 | 4 | 4 | 4 | 3 | 4 | 4 | 11 | 0.91 |
32 | 4 | 3 | 4 | 4 | 1 | 4 | 4 | 4 | 3 | 4 | 4 | 11 | 0.91 |
33 | 4 | 3 | 4 | 4 | 4 | 4 | 4 | 4 | 3 | 4 | 4 | 11 | 1.00 |
34 | 4 | 3 | 4 | 4 | 1 | 4 | 4 | 4 | 4 | 4 | 4 | 11 | 0.91 |
35 | 4 | 4 | 4 | 4 | 3 | 4 | 4 | 4 | 2 | 4 | 4 | 11 | 0.91 |
36 | 4 | 4 | 4 | 4 | 3 | 4 | 4 | 4 | 2 | 4 | 4 | 11 | 0.91 |
37 | 4 | 2 | 4 | 4 | 4 | 4 | 4 | 4 | 3 | 4 | 4 | 11 | 0.91 |
38 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 11 | 1.00 |
39 | 4 | 3 | 4 | 4 | 3 | 4 | 4 | 3 | 4 | 4 | 4 | 11 | 1.00 |
S‐CVI/Ave = 0.95 |
All I‐CVI scores were summated to calculate S‐CVI/Ave and then divided by the number of items: (0.82 + 0.82 + 0.91 + … + 1.00)/39 = 0.95. All calculated results of S‐CVI/Ave exceeded 0.90, which in combination with CVI (S‐CVI) levels above 0.78 (Table 2) represent excellent content validity (Table 3) [19].
Essential nursing skills | Areas in nursing | Number of items | S‐CVI/ave |
---|---|---|---|
Peripheral cannula insertion | Diagnostic/therapeutic essential nursing skills | 39 | 0.95 |
Urinary catheterization: female | Elimination | 54 | 0.95 |
Medication: injection of intravenous drugs | Medical management | 28 | 0.94 |
Mouth care in unconscious patients | Personal hygiene | 28 | 0.93 |
Insertion of a nasogastric tube | Nutrition | 36 | 0.93 |
Suctioning the nasopharyngeal airway | Respiratory care | 51 | 0.92 |
4. Discussion and conclusion
Methodological phases described in this pilot study could be considered and implemented when developing and establishing the checklist, which is designed to measure nursing students’ performance during clinical skills assessment using OSCE. The purpose of developing a DOC score in the study was to represent the range of complexity in essential nursing skills and to identify criteria for further research in the CVI phase. Results of the CVI analysis demonstrated a good content validity (I‐CVI and S‐CVI/Ave) for the essential nursing skills that were included in the evaluation.
The benefits of using CVI for OSCE checklists have to be considered in terms of how it might undermine the essential nursing skill as a whole. For example, the calculated CVI for some items in a procedure might be calculated as lower than recommended. That in turn questions the need for the item within an essential nursing skill, and yet it is argued that every item in an essential nursing skill has a purpose. Eliminating those items with a low CVI could therefore be detrimental to the whole OSCE essential nursing skill and presents a challenge to nurse educators. On the other hand, CVI is widely used for developing different methodological researching tools [21–25].
Using OSCE in undergraduate nursing education offers a fresh approach for nurse educators in Slovenia and provides a new opportunity for determining nursing students’ competency levels in simulated environment.
Findings from the CVI analysis are promising for developing OSCE checklist and are promising for further research using OSCE as an assessment modality.
References
- 1.
Harden RM, Stevenson M, Downie WW, Wilson GM. Assessment of clinical competence using objective structured examination. British Medical Journal. 1975; 5955 :447–451 - 2.
Pegram A, Fordham‐Clarke C. Implementing peer learning to prepare students for OSCEs. British Journal of Nursing. 2015; 24 :1060–1065. DOI: 10.12968/bjon.2015.24.21.1060 - 3.
Stunden A, Halcomb E, Jefferies D. Tools to reduce first year nursing students’ anxiety levels prior to undergoing objective structured clinical assessment (OSCA) and how this impacts on the student’s experience of their first clinical placement. Nurse Education Today. 2015; 35 :987–991.DOI: 10.1016/j.nedt.2015.04.014 - 4.
Harden RM. What is an OSCE? Medical Teacher. 1998; 10 :9–22 - 5.
Alinier G. Nursing students’ and lecturers’ perspectives of objective structured clinical examination incorporating simulation. Nurse Education Today. 2003; 23 :419–426 - 6.
McWilliam P, Botwinski C. Developing a successful nursing Objective Structured Clinical Examination. The Journal of Nursing Education. 2010; 49 :36–41. DOI: 10.3928/01484834‐20090915‐01 - 7.
Barry M, Noonan M, Bradshaw C, Murphy‐Tighe S. An exploration of student midwives’ experiences of the Objective Structured Clinical Examination assessment process. Nurse Education Today. 2012; 32 :690–694. DOI: 10.1016/j.nedt.2011.09.007 - 8.
Traynor M, Galanouli D. Have OSCEs come of age in nursing education? British Journal of Nursing. 2015; 24 :388–391. DOI: 10.12968/bjon.2015.24.7.388 - 9.
Mitchell ML, Jeffrey CA, Henderson A, Glover P, Nulty DD, Kelly MA, Groves M, Knight S. Using an Objective Structured Clinical Examination for Bachelor of Midwifery students’ preparation for practice. Women and Birth. 2014; 27 :108–113. DOI: 10.1016/j.wombi.2013.12.002 - 10.
Rushforth HE. Objective structured clinical examination (OSCE): Review of literature and implications for nursing education. Nurse Education Today. 2007; 27 :481–490 - 11.
Elliot DL, Fields SA, Keenen TL, Jaffe AC, Toffler WL. Use of a group objective structured clinical examination with first‐year medical students. Academic Medicine. 1994; 69 :990–992 - 12.
Humphris GM, Kaney S. The objective structured video exam for assessment of communication skills. Medical Education. 2000; 34 :939–945 - 13.
Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, Brown M. Objective structured assessment of technical skill (OSATS) for surgical residents. British Journal of Surgery. 1997; 84 :273–278 - 14.
Simpson D, Lawrence S, Krogull S. Using standardized ambulatory teaching situations for faculty development. Teaching and Learning in Medicine. 1992; 4 :58–61 - 15.
Harden RM, Lilley P, Norman G. The Definitive Guide to the OSCE: The Objective Structured Clinical Examination as a Performance Assessment. 1st ed. Churchill Livingstone: Elsevier Health Sciences; 2015. p. 42 - 16.
Mårtensson G, Löfmark A. Implementation and student evaluation of clinical final examination in nursing education. Nurse Education Today. 2013; 33 :1563–1568. DOI: 10.1016/j.nedt.2013.01.003 - 17.
Miller B, Carr KC. Integrating standardized patients and objective structured clinical examinations into a nurse practitioner curriculum. The Journal for Nurse Practitioners. 2016; 12 :e201‐e210 - 18.
Polit DF, Beck CT. The content validity index: Are you sure you know what’s being reported? Critique and recommendations. Research in Nursing & Health. 2006; 29 :489–497. DOI: 10.1016/j.nurpra.2016.01.017 - 19.
Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health. 2007; 30 :459–467 - 20.
Chumley HS. What does an OSCE checklist measure? Family Medicine. 2008; 40 :589–591 - 21.
Cooper S, Cant R, Porter J, Sellick K, Somers G, Kinsman L, Nestel D. Rating medical emergency teamwork performance: Development of the Team Emergency Assessment Measure (TEAM). Resuscitation. 2010; 81 :446–452. DOI: 10.1016/j.resuscitation.2009.11.027 - 22.
Beattie M, Shepherd A, Lauder W, Atherton I, Cowie J, Murphy DJ. Development and preliminary psychometric properties of the Care Experience Feedback Improvement Tool (CEFIT).BMJ Open. 2016; 6 :e010101. DOI: 10.1136/bmjopen‐2015‐010101 - 23.
Liaw SY, Scherpbier A, Klainin‐Yobas P, Rethans JJ. Rescuing A Patient In Deteriorating Situations (RAPIDS): An evaluation tool for assessing simulation performance on clinical deterioration. Resuscitation. 2011; 82 :1434–1439. DOI: 10.1016/j.resuscitation.2011.06.008 - 24.
Hsu LL, Hsieh SI. Development and psychometric evaluation of the competency inventory for nursing students: A learning outcome perspective. Nurse Education Today. 2013; 33 :492–497. DOI: 10.1016/j.nedt.2012.05.028 - 25.
Ulfvarson J, Oxelmark L. Developing an assessment tool for intended learning outcomes in clinical practice for nursing students. Nurse Education Today. 2012; 32 :703–708. DOI: 10.1016/j.nedt.2011.09.010