The generalizability of a construct driven approach to SJTs Eugene Burke, Chief Scientist

The generalizability of
a construct driven
approach to SJTs
Eugene Burke, Chief Scientist
Carly Vaughan, Managing Consultant
April 2011
©2011 SHL Group Limited
What we are going to cover
The largest multinational, multi-language and
multicultural SJT programme that we are aware of
Two projects that provide evidence of proof of concept
Highlights in terms of the evidence from those projects
A tentative answer to the question what do SJTs measure
©2011 SHL Group Limited
2
Part I: Development of SJTs for
The European Union
©2011 SHL Group Limited
3
Imagine you are asked to:
Design 1,625 SJT items over 3 years
125 items to be delivered every 6 weeks
That map to five competencies
And have to operate in 3 languages
To produce 40 equivalent tests
That operate at two different job levels
©2011 SHL Group Limited
4
And the items and tests:
Have to be calibrated and pre-trialled
Shown to be effective psychometrically
Shown to be effective linguistically
Fair by gender, age and nationality
©2011 SHL Group Limited
5
An example item
You work as part of a technical support team that produces
work internally for an organisation. You have noticed that
often work is not performed correctly or a step has been
omitted from a procedure.
You are aware that some
individuals are more at fault than others as they do not make
the effort to produce high quality results and they work in a
disorganised way.
a.
Explain to your team why these procedures are important and
what the consequences are of not performing these correctly.
b.
Try to arrange for your team to observe another team in the
organisation who produce high quality work.
c.
Check your own work and that of everyone else in the team to
make sure any errors are found.
d.
Suggest that the team tries many different ways to approach
their work to see if they can find a method where fewer mistakes
are made.
©2011 SHL Group Limited
6
Competencies covered
• Analysis and problem solving: Identifies the critical facts in
complex issues and develops creative and practical solutions
• Prioritising and organising: Prioritises the most important
tasks, works flexibly and organises own workload effectively
• Delivering quality and results: Takes personal responsibility
and initiative for delivering work to a high standard of quality
within set procedures
• Resilience: Remains effective under heavy workload, handles
organisational frustrations positively and adapts to a changing
working environment
• Working with others: Works co-operatively with others in
teams and across organisational boundaries, and respects
differences between people
©2011 SHL Group Limited
7
Building blocks for the solution
Leading &
Deciding
Deciding &
Initiating
Action
Leading &
Supervising
Supporting &
Cooperating
Working with
People
Adhering to
Principles &
Values
Interacting &
Presenting
Relating &
Networking
Persuading &
Influencing
Presenting &
Communicating
Analysing &
Interpreting
Writing &
Reporting
Applying
Expertise &
Technology
Analysing
Creating &
Conceptualising
Learning &
Researching
Creating &
Innovating
Formulating
Concepts &
Strategies
Organising &
Executing
Planning &
Organising
Delivering &
Meeting
Expectations
Following
Instructions &
Procedures
Adapting &
Coping
Adapting &
Responding to
Change
Coping with
Pressure
Enterprising &
Performing
Achieving
Goals &
Objectives
Entrepreneurial
& Commercial
Thinking
Universal
Competency
Framework
©2011 SHL Group Limited
8
Mapping to client’s competencies
Leading &
Deciding
Supporting &
Cooperating
Working With
Others
Interacting &
Presenting
Analysing &
Interpreting
Analysing & Problem
Solving
Creating &
Conceptualising
Organising &
Executing
Prioritising &
Organising
Adapting &
Coping
Resilience
Enterprising &
Performing
©2011 SHL Group Limited
Delivering Quality
& Results
Delivering Quality
& Results
9
Behavioural blueprint for items
Behaviour matrix
Analysing & Problem Solving
EPSO Competency
Making Judgements
UCF Behavioural
Component
Positive
indicator
Negative
indicator
Takes all available
information into account
when making judgements
Makes quick judgements
based upon incomplete
information
Passage with statements
©2011 SHL Group Limited
10
Delivering quality and results
You work as part of a technical support team that produces
work internally for an organisation. You have noticed that
often work is not performed correctly or a step has been
omitted from a procedure.
You are aware that some
individuals are more at fault than others as they do not make
the effort to produce high quality results and they work in a
disorganised way.
a.
Explain to your team why these procedures are important and
what the consequences are of not performing these correctly.
b.
Try to arrange for your team to observe another team in the
organisation who produce high quality work.
c.
Check your own work and that of everyone else in the team to
make sure any errors are found.
d.
Suggest that the team tries many different ways to approach
their work to see if they can find a method where fewer mistakes
are made.
©2011 SHL Group Limited
11
Language equivalence
©2011 SHL Group Limited
12
Evidence of the process
• Trialled on a total of 70,000
• This includes a competition with real 38,000 applicants in
2010
• Full range of p-values
• Ranging from 0.4 to 0.8 with a mean of 0.6
• Internal consistency reliabilities averaging at around
0.8
• Multiple test forms are equivalent in terms of reliabilities, means and
SDs as well as subscore correlations
• Competency domain scores that are coherent (see
next slide)
• Correlations with other constructs that are consistent
with published research (see nest slide)
©2011 SHL Group Limited
13
Correlations between test scores
Test
Verbal
Numerical
Abstract
Reasoning Reasoning Reasoning
SJT
0.36
0.23
Average
0.17
Prioritising
&
Organising
0.25
Score
Delivering
Quality
Results
Working
With
Others
Prioritising &
Organising
0.27
Working With
Others
0.28
0.31
Resilience
0.24
0.29
0.26
Analysis &
Problem Solving
0.27
0.33
0.30
Resilience
0.22
Source EPSO 01
©2011 SHL Group Limited
14
Part II: Development of SJTs for
a Global Retailer
©2011 SHL Group Limited
15
Imagine you are asked to:
Design an SJT for a global retailer
Who wants to start with a pilot in the UK
Then evaluate the test in Turkey and the US
And then roll it out globally in local languages
©2011 SHL Group Limited
16
Some choices for that client
• Use the classic inductive approach
– One option is a local instrument per country and language
– In addition to time and cost, how would you know that the
instrument is assessing common qualities across populations?
– How would you ensure a consistent talent acquisition strategy
across geographies?
• Use a forced etic approach
– Develop in UK English and then “localise” into other
languages
– How does this ensure that the instrument is effective across
all cultures and languages
• Use the combined emic-etic approach combined with
the criterion-centric approach
– Which is the strategy that we used
©2011 SHL Group Limited
17
Deductive approach to SJTs
The criterion-centric approach is founded on a theory
of the job
That uses a behavioural taxonomy to define the critical
behaviours that underpin effective performance
By using the links between behaviours and attributes
of people the constructs for the SJT are defined
Combine this with the combined emic-etic approach to
evaluate the generalisability of these constructs
across cultures and languages
©2011 SHL Group Limited
18
Of these competency dimensions
Leading &
Deciding
Deciding &
Initiating
Action
Leading &
Supervising
Supporting &
Cooperating
Working with
People
Adhering to
Principles &
Values
Interacting &
Presenting
Relating &
Networking
Persuading &
Influencing
Presenting &
Communicating
Analysing &
Interpreting
Writing &
Reporting
Applying
Expertise &
Technology
Analysing
Creating &
Conceptualising
Learning &
Researching
Creating &
Innovating
Formulating
Concepts &
Strategies
Organising &
Executing
Planning &
Organising
Delivering &
Meeting
Expectations
Following
Instructions &
Procedures
Adapting &
Coping
Adapting &
Responding to
Change
Coping with
Pressure
Enterprising &
Performing
Achieving
Goals &
Objectives
Entrepreneurial
& Commercial
Thinking
©2011 SHL Group Limited
19
These were identified
Supporting &
Cooperating
Working with
People
Interacting &
Presenting
Relating &
Networking
Delivering &
Meeting
Expectations
Organising &
Executing
Adapting &
Coping
©2011 SHL Group Limited
Adhering to
Principles &
Values
Adapting &
Responding to
Change
Coping with
Pressure
20
An example item
A customer has accidentally broken a bottle of oil.
There is broken glass and the oil is quickly spreading
across the floor. One of your colleagues is helping the
customer. You still have to re-fill a lot of shelves before
your break.
Watch your colleague for a moment to see if your help is
needed.
Help your colleague deal with the situation by calling the
cleaning staff.
Stay focused on completing your current task of re-filling
the shelves in time.
21
Overlap between SJTs forms
65%
UK
Turkey
65%
88%
71%
US
©2011 SHL Group Limited
22
An example change to an item
A customer has accidentally broken a bottle of oil.
There is broken glass and the oil is quickly spreading
across the floor. One of your colleagues is helping the
customer. You still have to re-fill a lot of shelves before
your break.
Watch your colleague for a moment to see if your help is
needed.
Help your colleague deal with the situation by calling the
cleaning staff.
Stay focused on completing your current task of re-filling
the shelves in time.
23
Cross validation
UK
Predictive
(N=70)
Working with
People
0.35
0.38
0.31
0.25
0.32
Adhering to
Principles & Values
0.28
0.18
0.55
0.21
0.32
Relating and
Networking
0.37
0.52
0.36
0.29
0.38
Meeting Customer
Expectations
0.46
0.28
0.22
0.26
0.30
Adapting to Change
0.38
0.19
0.26
0.32
0.29
Coping with
Pressure
0.38
0.18
0.43
0.10
0.28
Overall Rating
0.47
0.40
0.36
0.28
0.37
Criterion
Measure
Turkey
US
Concurrent Concurrent
(N=93)
(N=101)
Sample
Weighted
Average
(N=342)
UK
Concurrent
(N=78)
All correlations are corrected for range restriction
but not attenuation in the criterion measures
©2011 SHL Group Limited
24
Differences in performance
70%
63%
60%
Lowest
scorers on
the SJT
5 : 1 being
lowest
performers
on the job
50%
40%
30%
20%
46%
Lowest
Quartile Mgr
Ratings
13%
4%
10%
0%
Lowest Quartile
SJT Scores
©2011 SHL Group Limited
Highest
scorers on
the SJT
12 : 1
being
highest
performers
on the job
Highest
Quartile Mgr
Ratings
Highest Quartile
SJT Scores
25
Key messages from our presentation
©2011 SHL Group Limited
26
Common problems with SJTs
No clear logic in design and construction of content
Empirical rather than construct driven scoring
Not designed to anticipate localisation and
generalizability needs
These are the problems we have tried to
address through a criterion-centric and
combined emic-etic approach
©2011 SHL Group Limited
27
Thank you
[email protected]
[email protected]
You will also find us on
©2011 SHL Group Limited
28