Abstract: Ophthalmology residency training programs need authentic methods of assessment to show that trainees have learned and can do what is expected upon graduation. Written and oral examinations are necessary to assess knowledge but other methods are needed to assess skill. Workplace-based assessments (WPBAs) should be utilized to observe resident skill in the clinic and during surgery. Several such assessment tools have been published and validated. These tools have the additional benefit of facilitating specific formative feedback and thus can be used for both teaching and assessing.
The expression “Teaching does not equal learning” means that competence must be assessed. One cannot assume that just because something is taught, it is also learned. Valid and reliable methods of assessment are thus required to ensure an ophthalmologist can competently evaluate and treat. WPBAs are crucial tools required to authentically evaluate the ophthalmologist’s performance in patient care and surgical skill. WPBAs simply provide a structure to assessing observed skill in the clinic and operating room. Most WPBAs are rubrics which contain “Dimensions”, the skills required to perform a task, “Ratings”, a method to grade the performance and “Descriptors”, an explanation of what it means to achieve a certain rating (Table 1). The dimensions could be the steps of a surgical procedure or components of performing an eye examination. The rating system could be numerical (e.g., 1–4) or a scale such as novice, beginner, advanced beginner and competent. The descriptors are very important from both a teaching and tool reliability standpoint. Descriptors should be a precise description of the behavior expected for each rating. Each rating for each dimension ideally has a description. These descriptors tell the student and the assessor what is expected to achieve a certain rating. Examples of dimensions, ratings and behavioral descriptors are shown in Figure 1, a portion of the ICO-OSCAR:phaco (1). Thus, the student can review the descriptions and learn what is expected. The descriptors standardize the assessment and help assessor’s rate similarly thus producing a more reliable tool.
| Dimension | Ratings | |||
|---|---|---|---|---|
| Bad | Better | Good | Excellent | |
| Step 1 of procedure | Behavior description in every rating box | Behavior description in every rating box | Behavior description in every rating box | Behavior description in every rating box |
| Step 2 of procedure | – | – | – | – |
| Step 3, etc. | – | – | – | – |
In this review we will consider existing WPBAs designed to assess clinical skills and surgical proficiency. Another article in this issue will describe the multisource feedback (360-degree assessment) WPBA designed to assess communication skills and professionalism.
Literature search through Google and PubMed was conducted using key words: workplace-based assessment, surgical rubric, clinical skills assessment, ocex, Oscar, ocat, ophthalmic surgical skill assessment.
A variety of clinical evaluation exercises exist for general medical care. This review will describe ophthalmology specific WPBAs. To my knowledge, the Ophthalmic Clinical Evaluation Exercise (OCEX) is the only ophthalmology specific WPBA that has gone through validity and reliability determinations and has been published (2,3). The OCEX is used by an assessor who observes the resident performing an entire history, examination and case presentation. Importantly, a rubric that describes the behavior necessary to achieve each grade on the OCEX was developed. As the assessor observes they simply circle the behavior described in the rubric that is demonstrated by the trainee. When completed, the assessor reviews the rubric with the student as specific formative feedback for improvement. The OCEX has been shown to be a valid and reliable tool for assessing skill in patient care, medical knowledge and communication. It is available on the International Council of Ophthalmology’s (ICO) website in multiple languages (www.icoph.org). However, it was not developed by an international panel and thus the ICO has created and validated an international OCEX that has been submitted for publication.
Paley and associates investigated using the OCEX to discriminate skill levels across years of training and found that it did not help monitoring longitudinal development of resident clinical performance across years of training (4). This is not surprising as the OCEX was designed to evaluate whether a resident can take a history, conduct an examination and provide a management plan. This should be accomplished by the end of the first year (or sooner). The value of the OCEX is assuring the beginning resident can perform these skills and provide important formative feedback to achieve this goal. Langue and associates created and validated the “PEAR”—Pediatric Examination Assessment Rubric to specifically assess the resident’s skills in this area (5). The PEAR is similar to the OCEX but includes more detail and is aimed specifically at testing required for pediatric patients. Reliability studies are still to be completed.
The On-Call Consultation Assessment Tool (OCAT) is a validated type of WPBA that is applied retrospectively during a chart review (6). It addresses several aspects of on-call performance: (I) quality of patient care, (II) consultation timeliness, and (III) the resident’s perception of problem urgency. The OCAT is completed periodically by reviewing on-call consultation notes. The OCAT is then reviewed with the student to provide specific formative feedback. Subsequent OCATs are monitored for improvement.
The UK Royal College of Ophthalmologists (RCOphth) have developed a variety of WPBAs (7). None of these WPBAs have been published in peer reviewed literature and it is unclear what validation process was used. The Clinical Rating Scales (CRS) and Direct Observation of Procedural Skills (DOPS) both are used for assessing clinical skills. There are 11 CRS forms that assess each part of the 8-step eye examination plus retinoscopy, Amsler grid and history taking. The CRS are rubrics with grading scale of poor, fair, good and very good. They include descriptions of what behaviors constitute “poor” and “very good.” However, no descriptions are given for fair and good. Thus, this leaves quite a bit of interpretation to the assessor. The DOPs are used for a wide variety of skills such as ocular irrigation, lacrimal function, local anesthesia, etc. There is one DOPS form and the type of procedure is written in at the top. The rating scale is the same as the CRS (poor to very good) but there are no behavioral anchors describing what it means to achieve each rating.
Trainees are often evaluated by an assessor who observes a surgical procedure and provides feedback. Well-written rubrics are an essential component of proving high quality feedback.
Cremers and associates developed the “Global Rating Assessment of Skills in Intraocular Surgery” (GRASIS) and showed face and content validity (8). The GRASIS includes scores from 1–5 for skills preoperative knowledge, microscope use, instrument handling, and tissue treatment in addition to seven other areas. Behavioral descriptors were written for gradings of 1, 3 and 5 leaving ratings 2 and 4 to be determined by the evaluators. Cremers and associates also developed the “Objective Assessment of Skills in Intraocular Surgery” (OASIS) (9). This tool facilitates collation of objective data such as wound placement and size, phacoemulsification time, and total surgical time and is meant to complement the GRASIS. They showed that the OASIS had both face and content validity.
Feldman and Geist developed the Subjective Phacoemulsification Skills Assessment that is designed specifically for phacoemulsification cataract extraction (PCE) surgery (10). This tool looks at each individual step of PCE in addition to rating overall performance. The evaluator rates on a 1–5 scale from ‘strongly agree’ to ‘strongly disagree’. They were able to show a degree of inter-rater reliability.
The UK RCOphth WPBA Handbook also describes several Objective Structured Assessments of Surgical Skill (OSATS) (7). OSATS 2 and 3 are specifically for use of the operating microscope and aseptic technique. OSAT 1 is generic for all surgical procedures. Thus, there are no procedure specific behavioral descriptors to standardize the assessment or to teach the student.
Saleh and associates developed the “Objective Structured Assessment of Cataract Surgical Skill” (OSACSS) (11). Phacoemulsification surgery is divided into 20 separate steps and each step is scored using a 5-point Likert scale. There is a rudimentary but very subjective rubric that assigns the following descriptions to 3 of the 5 Likert ratings: 1= “poorly or inadequately performed”, 3= “performed with some errors or hesitation” and 5= “performed well with no prompting or hesitation.” Scores of 2 and 4 are left to the evaluator to determine. This creates a degree of subjectivity and leads to lower reliability of the tool. Recognizing this, an international panel of authors modified the OSACSS by producing a globally-applicable rubric with levels based on a modified Dreyfus model of skill acquisition (novice, beginner, advanced beginner, and competent). No expert level was included considering that trainees do not achieve this level during their training. Behavioral anchors or descriptions of behavior representing each scoring level of each step were written to try and decrease subjectivity and provide instruction to trainees as to what is expected (Figure 1). Validity and inter-rater reliability were established by international panels of experts (1,12). To my knowledge, the ICO-OSCARs are the only WPBAs developed and validated for international use.
In a similar fashion internationally applicable assessment tools for many commonly done ophthalmology surgeries were developed and validated (13-21). ICO-OSCARS are being developed for intravitreal injection, corneal transplant, selective laser trabeculoplasty, dacryocystorhinostomy, DMEK, DALK, DSAEK, pterygium, glaucoma tube shunt, ruptured globe, and pediatric glaucoma procedures. The ICO-OSCARs serve as a toolbox of internationally applicable, valid and reliable assessment tools for surgical skill that of clearly communicate to the learner what is expected to attain competence.
It is essential to have methods demonstrating the ability of our trainees to competently perform the skills required of an ophthalmologist in the real world. WPBA must be used in addition to traditional tests of medical knowledge such as multiple-choice questions, oral examinations and objective structured clinical examinations. There are an increasing number of validated WPBAs available for clinical and surgical skill.
