| Sunday 16th May Afternoon Workshops |
Sunday 16 MayAfternoon workshops: 1.30pm to 5.00pmWorkshop 18 - WORKSHOP FULLWhy and how should we evaluate the curriculum?Ara Tekian (University of Illinois at Chicago, USA) Venue: Hyatt Regency Hotel Cost: $125 One of the essential tasks in curriculum development is evaluation of an educational program. This workshop will start with an exercise that highlights the need and importance of curriculum evaluation. It will be followed by an interactive session, where everyone will be involved in applying Kern and his colleagues’ 10 steps for curriculum evaluation. These include identifying the users, uses, and resources needed for evaluation; formulating evaluation questions; choosing appropriate designs and instruments for data collection; and analyzing the data. Ethical concerns as well as presentation of results will also be included. Finally, a few successful case studies will be discussed, emphasizing tips for successful curriculum evaluation at participants’ home institutions.Workshop 19Learning and assessment in the operating theatre: a multimodal perspectiveRoger Kneebone (Imperial College London, UK) Venue: Hyatt Regency Hotel Cost: $125 This workshop focuses on learning and assessment in clinical practice, taking the operating theatre as a case study. A multimodal perspective will highlight the modes in which signs of learning are made and recognised in clinical practice, and how discrepancies of interpretation may arise. These signs go beyond speech and writing to include body positioning and movement of head and hands. The workshop draws on two strands of ongoing work by our group: ethnographic research into the operating theatre as a pedagogic site, and innovative uses of simulation to recontextualise observed practice. Theoretical and practical implications for learning and assessment will be discussed. The workshop is aimed at clinicians and educators with an interest in assessment and an understanding of current issues around healthcare simulation.Workshop 20Virtual Patients for assessmentRachel Ellaway (Northern Ontario School of Medicine, Sudbury, Canada) Venue: Hyatt Regency Hotel Cost: $125 Virtual patients can take many forms and fulfill many functions including problem-based learning, developing decision-making and professionalism skills and supporting synthesis of knowledge in complex simulated situations. The use of virtual patients for assessment purposes has tended to be formative rather than summative largely due to the effort required to create suitable summative assessment virtual patient cases and significant challenges ensuring their reliability. Working with freely available tools this workshop will review a range of virtual patient modalities for assessment, explore approaches to authoring VPs for assessment and explore methods for ensuring validity and reliability.Workshop 21 - WORKSHOP FULLAssessing Professionalism: Linking evaluation with teachingSylvia Cruess, Richard Cruess and Yvonne Steinert (Centre for Medical Education, McGill University, Montreal, Canada) Venue: Hyatt Regency Hotel Cost: $125 The assessment of knowledge of professionalism and of professional behavior in students, residents, and practicing physicians has become an important issue in medical education. The workshop will propose that a first and essential step is to define professionalism after which a variety of tools can be selected to assess both knowledge and behaviors. The workshop will encourage participants to outline the methods of selecting a definition for their own institution, determining which behaviors reflect the attributes of professionalism which are derived from the definition, and to choose how best in their own milieu to evaluate these behaviors. The necessity of evaluating the professionalism of faculty members will be stressed.Workshop 22 - WORKSHOP FULLUse of Generalizability Theory in designing and analyzing performance-based testsDave Swanson and Brian Clauser (National Board of Medical Examiners, Philadelphia, USA) Venue: Hyatt Regency Hotel Cost: $125 Performance-based testing methods (e.g., OSCEs, oral exams) are commonly used for assessment of clinical competence. Because these methods involve multiple sources of measurement error (for example, rater stringency, task difficulty, content specificity), classical test theory does not furnish the tools needed for investigation of psychometric characteristics. In this workshop, participants will learn to view assessment situations from the perspective of generalizability theory, which does supply the necessary conceptual and statistical tools to estimate the reproducibility of scores on performance-based tests and evaluate alternate approaches to test design and use of testing resources. Participants will learn to: (1) View assessment situations from a G-theory perspective; (2) Interpret output from software commonly used for generalizability analysis; (3) Evaluate the impact of alternate test designs on the reproducibility of test scores. Participants will be provided with a handout including background information on each of the above topics and annotated excerpts from statistical package output. Familiarity with generalizability theory is not required; however, attendees should be familiar with analysis of variance for multifactor designs with repeated measures.Workshop 23Assessing assessment at the station level; an OSCE quality improvement workshopGodfrey Pell (University of Leeds, UK) Venue: Hyatt Regency Hotel Cost: $125 This interactive workshop is primarily focused at the improvement OSCE assessment quality at the station level. In the ideal OSCE assessment the difference in checklist marks achieved by candidates should be purely the result of different levels of clinical competency; however, in the real world errors creep in which can sometimes account for as much as 50% of the mark variation. In the workshop we will discuss a range of metrics and their role in helping to identify sources of error, we will also suggest remedies for the issues raised. Real data from a UK medical school will be used to demonstrate a number of issues, as will data collected in the workshop from delegate activities. Issues covered will include: (1) Fixed effects such as assessor training, gender interactions; (2) Checklist design; (3) Support material; (4) Diagnostic metrics and their interpretation; (5) Quality control by walking about; (6) SP training. The workshop is at intermediate level, and some knowledge of OSCEs and basic statistics is expected.Workshop 24A simulated ward exercise to assess performance of doctors early in their postgraduate careersJ Ker, F Anderson and K Stirling (Clinical Skills Unit, Medical School, University of Dundee, UK) Venue: Hyatt Regency Hotel Cost: $125 The workshop will enable participants to (a) recognize the use of a simulated ward exercise in diagnosing doctors in difficulty; and (b) evaluate the use of an observation tool to assess performance.There is a need to identify those doctors who are underperforming early in their careers(1). The ward simulation exercise(2) enables performance to be assessed using a validated tool(3). Participants will be given an overview of the elements of the exercise and will have the opportunity to assess 2 exercises as part of the diagnostic process. Discussion on transferability to participants’ different health care systems will be facilitated.
Workshop 25Setting performance standards for simulation-based exercisesJack Boulet (Educational Commission for Foreign Medical Graduates, Philadelphia) and David Murray (Washington University School of Medicine, St. Louis), USA Venue: Hyatt Regency Hotel Cost: $125 With the recent adoption of high-stakes performance-based assessments in medicine and other healthcare professions, including those used for credentialing, there has been a need to modify existing standard setting methodologies, including developing new techniques that can reliably delimit the point, or points, that separate adequate from inadequate performance. For simulated patient (SP) or mannequin-based simulations to be effectively used for summative decisions (e.g., licensure, maintenance of certification), setting defensible performance standards is paramount. Examinee-centered approaches have been employed to derive cut-scores for performance-based assessments. Here, performance samples (e.g., videotapes) are shown to the panelists (without the scores) and they are asked to make summary judgments (e.g., adequate, inadequate) for each. Then, the panelists’ judgments are regressed onto the actual scores to delimit the score point that maximally discriminates between adequate and inadequate performance. This technique, to be employed in the workshop, has been shown to yield valid and defensible standards for both SP- and mannequin-based assessments. After attending this workshop, the learner will be able to: (1) Choose an appropriate standard setting methodology for his/her particular needs; (2) Design a basic standard setting study; (3) Understand and evaluate the process of setting standards for performance-based assessments.Workshop 26Using simulation to measure outcomes with validity and reliabilityUniversity of Miami Gordon Center Faculty Venue: University of Miami Gordon Center for Research in Medical Education Cost: $125 Simulation technology is being used with increasing frequency as a tool to measure individual, team and curricular outcomes. Important questions remain regarding the added value of using this technology to measure a range of competencies previously ignored or limited with traditional methods. Critical to these issues is the development of robust outcome measures that enable educators to make important decisions reading the competence of trainees and researchers to conduct studies with sufficient rigor and generalizability. In this workshop, participants will address the relationship between reliability and validity, describe multiple forms of evidence for validity and select an approach to reliability and validity assessment for a sample simulation-based exercise.Workshop 27 (EN ESPAÑOL)Using simulation to measure outcomes with validity and reliabilityUniversity of Miami Gordon Center Faculty Venue: University of Miami Gordon Center for Research in Medical Education Cost: $125 Simulation technology is being used with increasing frequency as a tool to measure individual, team and curricular outcomes. Important questions remain regarding the added value of using this technology to measure a range of competencies previously ignored or limited with traditional methods. Critical to these issues is the development of robust outcome measures that enable educators to make important decisions reading the competence of trainees and researchers to conduct studies with sufficient rigor and generalizability. In this workshop, participants will address the relationship between reliability and validity, describe multiple forms of evidence for validity and select an approach to reliability and validity assessment for a sample simulation-based exercise. (EN ESPAÑOL)
|
   |