1. No category

Dr. et MME Sandy Kujumdshiev, Zurich, Switzerland
How to Evaluate the Role-play
of Simulated Patients:
Development and Validation of a
New Questionnaire.
Sandy Kujumdshiev,
Christina Conrad, Michaela Zupanic, TOF Wagner,
Martin Fischer
25.05.2013
Questionnaire to evaluate the role-play of simulated patients
Introduction
Dr. et MME Sandy Kujumdshiev, Zurich, Switzerland
The reliability and also the validity of an OSCE
depend on the examiner rating,
the standardization of the stations and
on the highly standardized role-play of the
simulated patient.
Yet only few studies address the quality
assessment that is crucial for the outcome (1).
1 Cleland JA at al. AMEE Guide No 42. Med Teach 2009
25.05.2013
Questionnaire to evaluate the role-play of simulated patients
Introduction
Dr. et MME Sandy Kujumdshiev, Zurich, Switzerland
Mainly realism, little communication, not
enough items at all, teaching situation
Questionnaire for individual and specific quality
2 Wind La et al. MEDICAL EDUCATION 2004
management
25.05.2013
Questionnaire to evaluate the role-play of simulated patients
Hypothesis / Research Questions
Dr. et MME Sandy Kujumdshiev, Zurich, Switzerland
H: The role-play of simulated patients can be
assessed in a valid and reproducible way by a
questionnaire.
What are the criteria for an authentic role-play of
simulated patients in an OSCE?
How is it possible to operationalize these criteria
in a short questionnaire to be able to assess the
role-play of simulated patients?
Is the developed questionnaire valid and reliable
and the rating reproducible?
25.05.2013
Questionnaire to evaluate the role-play of simulated patients
Methods:
1 Collection, Prioritization
M
Dr. et MME Sandy Kujumdshiev, Zurich, Switzerland
We asked three experts, teachers,
simulated patients and students what
April/May 2011
March 2011
• Training
are the criteria for good
role-play.
February 2011
• Training
M
before
October 2010 Pretest 1
• Opera• Pretest 1
tionalization
April 2010
• Revision
• Quali
tative
Survey
• Training
before
Pretest 2
We collected criteria.
• Prioritization
25.05.2013
• Pretest 2
before
OSCE
• Use
in the
real
OSCE
Three other experts, teachers,
simulated patients and students
prioritized them.
Questionnaire to evaluate the role-play of simulated patients
Methods: 2 Operationalization
M
Dr. et MME Sandy Kujumdshiev, Zurich, Switzerland
Communication
• 5 Items
M
Information March
• 4 2011
Items
M February 2011
• Training
before
Realism
October 2010 Pretest 1
• Opera• Pretest 1
tionaliRecurrent
zation
April 2010
Play
• Revision
• Quali
tative
Survey
Global
• Prioritization
25.05.2013
Rating
• Training
before
Pretest 2
April/May 2011
• Training
before
OSCE
• Use
•
7
Items
in the
• Pretest 2
real
OSCE
• 1 Item
• 1 additional
Item
Questionnaire to evaluate the role-play of simulated patients
Methods: 3 Pretest A
M
18 Raters,
90 questionnaires
Dr. et MME Sandy Kujumdshiev, Zurich, Switzerland
M
5 Roles
M
History
March 2011
Taking
• Training
before
Patient
Pretest 2
Management
February 2011
• Training
before
October 2010 Pretest A
• Pretest 2
Physical
• OperaExamination
• Pretest A
tionalization
Breaking Bad
April 2010
• Revision
News
• Quali
tative
Survey
Psychiatric
Exploration
• Prioritization
25.05.2013
Univer- Subject
sity
April/May 2011
Göttin- General
• Training
Medicine
before
gen
OSCE
Frank- Internal
• Use Medicine
furt
in the
Frankreal Surgery
OSCE
furt
Mainz
Med.
Psych.,
Sociology
Dres- Psychiatry
den
Questionnaire to evaluate the role-play of simulated patients
Methods: 3 Results Pretest A
M
Improvement
Cronbach’s alpha:
Dr. et MME Sandy Kujumdshiev, Zurich, Switzerland
One item was removed becauseM
of a low item difficulty p < .70
M
and a discriminatory power
under r = .30
February 2011
• Training
before
October 2010 Pretest A
• Opera• Pretest A
tionalization
April 2010
• Revision
• Quali
tative
Survey
• Prioritization
25.05.2013
April/May 2011
• Training
before
OSCE
commun. α=0.798
March 2011
• Training
before
Pretest 2
information
α=0.583 • Use
• Pretest 2
in the
real
OSCE
realism α=0.894
entire questionnaire was highly
reliable with α=0.894
Questionnaire to evaluate the role-play of simulated patients
Methods: 4 Results Pretest B
intern
consistency:
M
M
Dr. et MME Sandy Kujumdshiev, Zurich, Switzerland
April/May 2011
• Training
before
OSCE
comm.
α=0.775
February 2011
• Training
informat.
• Use
before
in the
October 2010 Pretest A
• Pretest B α=0.819
real
• OperaOSCE
• Pretest A
tionalirealism
zation
April 2010
• Revision
α=0.836
• Quali
tative
entire questionnaire stayed
Survey
• Priorihighly reliable with α= 0.892
tization
March 2011
• Training
before
Pretest B
25.05.2013
Questionnaire to evaluate the role-play of simulated patients
The Final Questionnaire
Dr. et MME Sandy Kujumdshiev, Zurich, Switzerland
Communication
• 5 Items
Information
Items
• 44 Items
Realism
• 7 Items
Recurrent Play
• 1 Item
Global Rating
• 1 additional Item
25.05.2013
N
o
t
r
a
t
e
a
b
l
e
Questionnaire to evaluate the role-play of simulated patients
Methods: 5 OSCE
B
From 89 questionnaires1:
Dr. et MME Sandy Kujumdshiev, Zurich, Switzerland
34 questionnaires for history taking
April/May
2011
47 questionnaires for phys. examination
März 2011
• Training
• Schulung
before
8 questionnaires for patient management
direkt vor
Februar 2011
• Schulung
direkt vor
Pretest 1
Pretest 2
in parallelOktober
use with
2010 the already validated
• Operationa- (2)
MaSP questionnaire
• Pretest 1
lisierung
April 2010
• Qualitative
Befragung
• Pretest 2
OSCE
• Use
in the
real
OSCE
• Überarbeitung
All examiners were trained for the MaSP
questionnaire,
our questionnaire and the OSCE rating
• Priorisierung 2 Wind La et al. MEDICAL EDUCATION 2004;
Internal Medicine OSCEs, Frankfurt
before.
1
25.05.2013
Questionnaire to evaluate the role-play of simulated patients
Results
Dr. et MME Sandy Kujumdshiev, Zurich, Switzerland
Subscal
e
Communication
Subscale
Information
Subscale
Realism
New
Questi
onnair
e
MaSP
Mean (SD)
4.34
(1.10)
4.0
(1.50)
4.19
(0.83)
4.17
(0.88)
3.25
(0.52)
Cronbach‘s α
0.766
0.852
0.686
0.863
0.630
Spearman
Rho Correlationscoefficient
0.285
0.229
0.431
0.386
-
Significancelevel
0.05
0.05
0.01
0.01
-
Subscale realism had the highest agreement to the MaSP.
25.05.2013
Questionnaire to evaluate the role-play of simulated patients
Discussion / Take Home Message
Dr. et MME Sandy Kujumdshiev, Zurich, Switzerland
Our new questionnaire showed very good
statistical results, in comparison to the MaSP
even better.
The questionnaire provides a widespread
performance evaluation in “high stakes”
assessment.
The appropriateness of our questionnaire for
different types of role-play may be an idea for
further research.
25.05.2013
Questionnaire to evaluate the role-play of simulated patients
Dr. et MME Sandy Kujumdshiev, Zurich, Switzerland
Special thanks to Sabine Fischbeck, Martin
Fischer, Anne Simmenroth-Nayda, Reinhard
Strametz, Dagmar Kujumdshiev, Benjamin
Weih, Patricia Winkelmann, Uwe Zinßer and
Michaela Zupanic.
Correspondence should be addressed to:
e-mail:
[email protected]
25.05.2013
Questionnaire to evaluate the role-play of simulated patients