Examining washback:
What do we know
and what is there left to explore?
Dianne Wall
Lancaster University
UK
Definitions
High-stakes tests
Tests whose results are seen – rightly or
wrongly – by students, teachers,
administrators, parents or the general
public, as being used to make important
decisions that immediately and directly
affect them.
(Madaus, 1988)
Definitions
Impact
Any of the effects that a high-stakes test
may have on individuals, policies, or
practices – within a classroom, a school,
an educational system or society as a
whole.
(Wall 1997)
Definitions
Washback
... a type of impact, which relates to
the effects of high-stakes tests on
classroom practices – particularly
teaching and learning.
Types of washback
… can be positive or negative, to
the extent that it either promotes
or impedes the accomplishment
of educational goals held by
learners and/or programme
personnel.
(Bailey, 1996)
Positive washback
If a test has positive washback,
‘there is no difference between
teaching the curriculum and teaching
to the test’.
(Weigle & Jensen, 1997, p. 205)
Negative washback
- a mismatch between the stated goals of
instruction and the focus of assessment,
- may lead to the abandonment of
instructional goals in favour of test
preparation
Outline
•
Development of the concept
•
Overview of research
early studies
overview of what we know now
•
Critical issues
General education
Positive views
Tests can be a powerful, low-cost
means of influencing the quality of what
teachers teach and what learners learn
at school.
(Heyneman & Ransom, 1992)
General education
Positive views
‘measurement-driven instruction’
(MDI)
If tests are ‘properly conceived and
implemented’, then focusing teaching
on what they assess is a beneficial
activity.
(Popham, 1987)
General education
Positive views
Systemic validity
…when a test induces in the education
system curricular and instructional
changes that foster the development of
the cognitive skills that the test is
designed to measure.
(Fredericksen & Collins, 1989)
General education
Negative views
‘psychometric imperialism’
- Leads to cramming
- Narrows the curriculum
- Focuses the attention on skills that are easy to
test
- Restricts teacher and student creativity
- Demeans the professional judgement of teachers
… transfers control over the
curriculum to the agency which
sets or controls the exam.
(Madaus, 1988)
Views from language education
If it is a good examination, it will have a useful
effect on teaching; if bad, then it will have a
damaging effect on teaching.
(Heaton, 1990)
- Aspirations
- Assertions (little evidence)
Foundation papers
•
•
•
•
Alderson & Wall 1993
Hughes 1994
Bailey 1996
Messick 1996
Alderson & Wall 1993
• What washback is and what it is not
• Need to be specific: Washback Hypotheses
• Need to use a variety of methods when
investigating – with emphasis on observation
• Need to learn from research in other disciplines:
e.g innovation theory, motivation theory
Washback Hypotheses
•
•
•
•
•
•
A test will influence teaching.
A test will influence learning.
A test will influence what teachers teach.
A test will influence what learners learn.
A test will influence how teachers teach.
A test will influence how learners learn.
Washback Hypotheses (continued)
• A test will influence the rate and sequence, and
the degree and depth of teaching.
• A test will influence the rate and sequence, and
the degree and depth of learning.
• A test will influence attitudes to the content,
method, etc … of teaching and learning.
• Tests will have washback on all teachers and
learners.
or
• Tests will have washback on some teachers and
some learners but not on others.
Hughes 1994
Need to look at washback on
- Participants
- Processes
- Products
Participants
Teachers
Learners
Administrators
Materials writers
Curriculum designers
Processes
For teachers
What they teach, how they teach, the intensity
of their teaching, whether they give extra
classes…
For learners
Practising the target language and skills,
deciding what to study and what not to study,
memorising, worrying, looking for shortcuts,
cheating…
Products
New curricula?
New timetables?
New materials?
Better learning?
Bailey, 1999
Basic model of washback
Messick 1996
Evidential link
It is problematic to claim evidence of test
washback if a logical or evidential link
cannot be forged between the teaching or
learning outcomes and the test properties
thought to influence them.
Messick 1996
Validity by design
…rather than seeking washback as a sign of
test validity, seek validity by design as a
likely basis for washback.
Messick 1996
Validity by design
•
•
•
•
direct testing
authenticity (input, tasks)
avoid construct under-representation
avoid construct-irrelevant variance
Early research
Wall & Alderson 1993
different amounts of washback – on content,
methods, means of assessment
Alderson & Hamp-Lyons 1996, Watanabe 1996
teachers are affected by tests in different
ways
Shohamy, Donitsa-Schmitt & Ferman, 1996
the washback of tests can change over time
Areas where washback can appear
(Spratt 2005)
Curriculum
contents of curriculum, timetabling
Teaching materials
choice of textbooks, use of past papers, teacher-made
materials
Teaching methods
choice of methods, teaching of test-taking skills
Attitudes and feelings
of learners and teachers
Learning
Do test results improve?
Does learning improve?
Factors influencing washback
(Spratt 2005)
The exam
-its proximity
-its stakes
-the status of the language it tests
-its purpose
-the formats it employs
-the weighting of individual papers
-when the exam was introduced
-how familiar the exam is to teachers
Teacher beliefs about
- the reliability and fairness of the exam
- what constitute effective teaching methods
- how much the exam contravenes their
current teaching practices
- the stakes and usefulness of the exam
- their teaching philosophy
- the relationship between the exam and the
textbook
- their students’ beliefs
Teachers’ attitudes towards
- The exam
- Preparation of materials for exam classes
- Lesson preparation for exam classes
Teachers education and training
- Teachers’ own education and educational experience
- The amount of general methodological training they
have received
- Training in teaching towards specific exams and in how
to use exam-related textbooks
- Access to and familiarity with exam support materials
such as exam specifications
- Understanding of the exam’s rationale or philosophy
Resources
- The availability of customised materials and exam
support materials such as exam specifications
- The types of textbooks available
School
- Its atmosphere
- How much the administrators put pressure on
teachers to achieve results
- The amount of time and number of students in
exam classes
- Cultural factors such as learning traditions
Recent investigations
Teacher beliefs – Huang 2009
Washback on learners –
Huhta, Kalaja, & Pitkänen-Huhta 2006
Tracing the evidential link –
Wall & Horak 2006, 2008 and 2011
Critical issues
• For teachers - How to find the right balance
between teaching and exam preparation?
• For exam developers – How to strengthen the
influence of the exam when so many other
factors affect teaching?
• For testing community - Who should accept
responsibility if an exam is mis-used, and has
unintentional negative washback and impact?
• For educational authorities – How to convince
examination boards to do impact studies?
Importance of washback studies
EALTA Guidelines for Good Practice
Considerations for test development in national or
institutional testing units or centres
1. Is the test intended to initiate change(s) in
the current practice?
2. What is the washback effect? What
studies have been conducted?
Survey on Impact Studies
(Wall & Horak 2006)
Survey of EALTA members via discussion list
– Have you been involved in the introduction of a
new exam or an examination reform?
– What was the intended impact of the
new/reformed exam?
– Was an impact study of any sort conducted
(however small)?
– Methodology – who, when, what, how?
Survey on Impact Studies
Reasons given for not conducting an impact study
• Lack of awareness of value
‘The new system is simply assumed to be a
substantial improvement …’
• Lack of interest
‘…we can’t imagine who would have been
interested in the results except perhaps
ourselves.’
Reasons given for not conducting impact studies
• Lack of expertise
‘…because there is no expertise available and
there is not a scientific language testing tradition in
the region.’
• Lack of time and energy
‘We knew it would be hard enough to introduce
reforms; we did not have the time or energy to
conduct any further tests of the kind you mentioned.’
Reasons given for not conducting impact studies
• Lack of resources
• Fear of results
‘Ministries are notoriously reluctant to let
outsiders see if their measures have measurable
outcomes.’
‘…fear to stir the wasps’ nest – the reality might
be too terrible.’
Why aren’t there more washback studies?
Lack of
• Awareness of the value
• Interest
• Expertise
• Time and energy
• Resources
and
• Fear of results
Thanks, and over to you…
[email protected]