Academic Section of CAEP: Featured Education Innovation
Name of
Innovation
Competency-based Assessment of Resuscitation Skills by Simulation-based OSCE using
the Queen’s Simulation Assessment Tool (QSAT)
Lead Innovators
Andrew K. Hall, J. Damon Dagnone
Lead Innovator’s
email address
[email protected]
Contact for Lead
Innovator
Dr. Andrew Koch Hall
Department of Emergency Medicine
Queen’s University
Empire 3, Kingston General Hospital
76 Stuart Street
Kingston, Ontario, Canada K7L 2V7
(613) 548-2368
Division or
Department
website
http://emergencymed.queensu.ca/
Description of
the Innovation
Background and Goals: Assessment of clinical expertise in postgraduate medical
education is moving away from knowledge-based examinations towards competencybased assessments.1 The use of high-fidelity simulation is emerging as an effective
approach to competency-based assessments.2 Over a four year period, we developed
and validated a modifiable anchored global assessment scoring tool for simulation-based
Objective Structured Clinical Examinations (OSCEs) of resuscitation competence in
postgraduate emergency medicine (EM) trainees: the Queen’s Simulation Assessment
Tool (QSAT)
Methods: The Department of EM at Queen’s University implemented a longitudinal
simulation-based resuscitation curriculum that has employed bi-annual simulation-based
OSCEs to assess resident performance in resuscitation since 2009. In these exams, 20
to 25 EM residents are individually presented with 2 or 3 resuscitation scenarios and
debriefed by a faculty member immediately following their performance.
The QSAT was developed for use in these exams, using a modified Delphi technique
with a panel of EM physicians. It is a hybrid scoring tool comprised of four anchored
domain scores and an overall global assessment score. The QSAT is unique in its simple
and compact generic structure, which can be easily modified for utilization in any
resuscitation scenario. The figure below demonstrates the generic QSAT, and a specific
scenario QSAT with anchor modification.
Following an initial blueprinting technique, 10 standardized resuscitation OSCE scenarios
were developed and administered to EM trainees. Video-recorded resident performances
were scored by multiple blinded EM attending physicians trained in the use of the
scenario specific QSATs. Utilizing the “unified model” of an argument for validity,
originally proposed by Messick3 we designed the QSAT, OSCE stations, and review
process with principles of content and response process validity; and collected data
relating specifically to the internal structure validity, relations with other variables such as
level of training, and perceived benefit to learning.
Results: Discriminatory validity (Senior vs Junior) was excellent and inter-rater reliability
showed acceptable levels of agreement for each scenario. Generalizability studies
yielded G-coefficients ranging from 0.67 to 0.84. D-studies suggested that increasing the
number of scenarios per OSCE (>6) with a single examiner per station would produce Gcoefficients close to 0.90, which would be acceptable for high-stakes examinations.
Resident trainees reported comfort being assessed in the simulation environment and
found the simulation-based examination very valuable to their learning. Detailed
descriptions of the QSAT development and validation and our program of simulationbased OSCE assessment are available4,5.
Reflective critique: In summary, these OSCEs have become an important part of the
assessment system within our EM program at Queen’s University and will help us as we
transition to a fully integrated competency-based medical education (CBME) curriculum
by July 2017. This assessment methodology has subsequently been evaluated in a multicenter study including 4 other Canadian sites.6 Looking forward, our next step is the
administration of single-center 8-stations simulation-based resuscitation OSCE
examining EM residents from across the country. It is our hope this form of resuscitation
skills assessment will be included in the Royal College’s Competency by design (CBD)7
project for emergency medicine across Canada.
References
1.
2.
3.
4.
5.
6.
7.
Who wrote this
summary?
Hamstra SJ: Keynote address: the focus on competencies and individual learner
assessment as emerging themes in medical education research. Acad Emerg Med
2012; 19: 1336-43
Sherbino J, Bandiera G, Frank J: Assessing competence in emergency medicine
trainees; an overview of effective methodologies. Can J Emerg Med 2008; 10: 365
Messick S: Validity, Educational Measurement, 3rd edition. Edited by Linn RL. New
York, Macmillan, 1989, pp 13-103
Hall AK, Dagnone JD, Lacroix LL, Pickett W, Klinger DA. Queen’s Simulation
Assessment Tool (QSAT): Development and Validation of an Assessment Tool for
Resuscitation OSCE Stations in Emergency Medicine. Simul Healthc 2015; 10: 67132.
Hagel CM, Hall AK, Dagnone JD. Queen’s Emergency Medicine Simulation OSCE:
an advance in competency-based assessment. Canadian Journal of Emergency
Medicine (in press)
Dagnone JD, Hall AK, Woolfrey K, Davison C, Moore SM, McNeil G, Ross JR, Pickett
W. QSAT - Validation of a Competency-based Resuscitation Assessment Tool - A
Canadian Multi-Centered Study [abstract]. Can J Emerg Med 2014; 15: S1
Harris, KA, Frank JR. eds. Competence by Design: Reshaping Canadian Medical
Education. The Royal College of Physicians and Surgeons of Canada 2014;
Retrieved Mar 25, 2015, from:
http://www.royalcollege.ca/portal/page/portal/rc/common/documents/educational_initi
atives/rc_competency-by-design_ebook_e.pdf
Andrew K. Hall and J. Damon Dagnone