30/03/2015
Outline
Teacher Malpractice in
Assessment: the International
Context
Ardeshir Geranpayeh, Ph.D.
Head of Psychometrics & Data Services
Oxford, 26 March 2015
• Introduction
• Cheating (Malpractice):
- What, Where, How, Why, Who
- Consequences: Threat to Test Validity
• Standards and Regulatory Requirements
• How to avoid Cheating:
– Prevention vs Detection
– Policies on punishment
• School Collusion: Teacher Malpractice
© UCLES 2015
Cheating in the News
Cheating in the News
Driving test fraud:
More learner drivers
caught cheating,
BBC News, 26 Jan 2015
• 2,440 Chinese students caught cheating
in latest high-tech scam
Chinese state television reported on Sunday
that invigilators detected abnormal radio
signals from an illegal frequency during
national licensing tests for pharmacists in
Shaanxi province. The independent
Wednesday, 28 October 2014
1
30/03/2015
Cheating in the News
Cheating in the News
According to The Harvard Crimson, a survey
of this year's incoming freshmen revealed
that 42 percent had cheated on high school
homework before arriving at the Ivy League
institution.
The Huffington Post,
18 February 2015
More than 50,000 English language tests
taken by overseas students to extend their
British visas have been declared invalid or
questionable as a result of an official
investigation into cheating on a huge scale.
Cheating in the News
Cheating in the News
Cheating in Bihar:
India arrests
Hundreds over Bihar
Blatant Cheating
In Schools
'Cheating teachers have lost morality':
Dirty tricks on
the rise to boost
exam grades
BBC News
21 March 2015
The Guardian, 24 June 2014
MailOnline
13 June 2014
2
30/03/2015
abcNews
abcNews Poll
1 in 3 they
themselves
have cheated.
Rising to 43%
of older teens.
Most say
cheaters don't
get caught.
What is Cheating?
“Any action that violates the rules for
administering a test” Cizek, 1999:3
Where does it happen?
It can take a variety of forms
3
30/03/2015
Pen Scanners
There are YouTube
videos on How to Scan a
Document on the Go
iPod
Song names are renamed
with notes or test answers
for viewing on the screen.
Text files can be stored.
Text Messaging
Examinee can ask
questions and get
answers from friend
during test via text
messaging.
Toilet Breaks!
Calculator
Notes are entered
into calculators that
have memory for
storing notes.
Audio notes can be
stored.
Video notes can be
stored.
4
30/03/2015
Camera Phone
Examinee takes
pictures of a test with
a camera phone and
sends picture to
another person who
can text message
correct answers back.
Why does it happen?
– Material rewards such as
• Access to life chances
• Competitiveness
• Lack of self confidence
• Publication of league tables
(Schools)
– Unintentional Cheating
• Students, material sharing
• Teachers and Schools
New Technology
Google Glass
Share your
adventures instantly
to Google+, Facebook,
YouTube …
Apple Watch
Who is involved?
•
•
•
•
•
Maladministration
Imposters
Leaked paper / answers
Prohibited material
Copying
5
30/03/2015
What is at stake?
– Threat to test validity
– Score obtained by fraudulent means is not
valid
– Has negative impact on the validity of scores
obtained by other candidates
– Denying opportunities to others
– Where cheating is seen to be widespread,
even honestly obtained test results may lose
credibility and certificates become devalued
Standards for Prevention of
Cheating
Explicit statements in the Standards for
Educational and Psychological Testing
(1999)
What is at stake?
• Test Security is now seen as one of the most
important aspects of test Validity
• A Test may not be Valid if it is not secure
Standards for Prevention of
Cheating
1. Protect the security of tests (standard 11.7)
2. Ensure that individuals who administer the tests are
proficient in administration procedures and understand
the importance of adhering to directions provided by the
test developer (standard 13.10)
3. Inform examinees that it is inappropriate for them to
have someone else take the test, for them to disclose
secure test materials, or engage in any other form of
cheating (standard 8.7)
6
30/03/2015
Standards for Prevention of
Cheating
Cheating detection:
Regulatory requirements
4. Ensure that test preparation activities and materials
provided to students will not adversely affect the validity
of test score inferences (standard 13.11) and
5. Maintain the integrity of test results by eliminating
practices designed to raise test scores without
improving students, real knowledge, skills, or abilities in
the area tested (standard 15.9)
Ofqual’s conditions around
malpractice
• Ofqual is a risk-based regulator of qualifications
(other than degrees) that sets the standards
that awarding organisations need to meet when
they design, deliver and award regulated
qualifications, via a number of conditions of
recognition.
• Ofqual doesn’t tell AO’s how to manage
malpractice. It tells AO’s what must result from
their behaviour.
Condition A8: Malpractice and
maladministration
Condition A8 of the General Conditions of Recognition
covers:
 Preventing malpractice and maladministration
 Investigating and managing the effect of malpractice and
maladministration
 Procedures relating to malpractice and maladministration
 Dealing with malpractice and maladministration
7
30/03/2015
Examples of ‘Positive indicators’
The awarding organisation:
 has ways of working that reduce risks and incidents
occurring
 makes sure that individuals involved in the delivery of its
qualifications understand and routinely follow these ways
of working
 takes all reasonable steps to provide information which
is not inaccurate or misleading
 has policies, practices and/or procedures that reduce the
risk of malpractice and maladministration
When Ofqual monitored Cambridge English Language
Assessment they were happy with how we are managing
malpractice.
How to avoid Cheating?
Examples of ‘Negative indicators’
 Using an individual or Centre to assist with an
investigation when there is a suspicion or allegation
that they were connected to, or responsible for, the
case being investigated
 Not demonstrating steps taken to prevent repeat
incidents of previous cases
 not following its own procedures during investigation
 not properly evaluating or validating evidence collected
during investigation
 not keeping records/documents relating to the incident
for an appropriate period
 not being able to enforce action taken against those
involved, and cannot demonstrate a good reason why it
has been unable to do so
How to avoid Cheating?
• The Best practice is “Prevention”
• No Statistical magic can replace good
invigilation
• Tension between Faster/on demand results
and proper data forensic processes
• Use variety of ID checks
– Candidate ID Verification
– Biomedical checks
– Hand writing
8
30/03/2015
Policies on punishment
Once a cheating is detected, an action has to
be put in place to
• Stop fraudulent use of test results
• Deter future cheaters
• Punishment is dependent on the level of
cheating, which in turn can depend on 5 levels
of cheating detection
• Individual candidates, Group of candidates,
Teacher Malpractice, Test Centre collusion
and Widespread cheating
Level of punishment
•
What about School collusion:
Teacher Malpractice
• Students implicated may not have been
involved in the cheating
• Whilst the candidate’s results may be
cancelled if school was to be blamed, no
further action will normally be taken
against candidates
•
•
•
•
•
Withdrawing results/certificate (individual)
Re-taking the exam (suspect results)
Life Ban (if stakes is high or imposters)
Informing stake holders (regulator)
Legal action (insider)
Teacher Malpractice: the Atlanta case
• In 2009, Beverly Hall, Ex-Atlanta school
superintendent, was named national
superintendent of the year largely based
on her district’s improved standardised
test scores
• She was once credited with turning
around some of the city’s lowestperforming schools
9
30/03/2015
Teacher Malpractice: the Atlanta case
• Biggest US cheating scandal in US History
• Cheating detected on a 2009 standardized
state test involving 178 teachers and
principles, 56 schools investigated cheated,
43 people were indicted
• Prosecutors set out to reveal a culture of fear
and pressure existed under Hall
• Georgia Governor determination to trace its
source
• Cheating traced back to 2001
Teacher Malpractice: the Atlanta case
• The scandal testifies that cheating is no
longer seen as an old-fashioned battle
between teachers and students
• When the stakes are high, teachers would
also be willing to cheat
Final Remarks
• Use new psychometric techniques to
detect school collusion
• Erasure analysis detection
• Multifaceted approach to monitor teachers’
assessment
• Malpractice awareness among teachers
• Collaborative dialogue with teachers as
the guardians of students’ achievement
10
30/03/2015
Thank you for listening.
[email protected]
11