1. No category

ROI on a Shoestring: How to
Measure More with Less
Session SU316
2007 ASTD International Conference & Exposition
Learning Objectives
Upon completion of this session, participants will be able to:
• Apply 10 cost-effective methods for measuring impact at
multiple levels, including ROI
• Leverage resources for collecting, analyzing, and
presenting results
• Define criteria for how and when to integrate cost-savings
approaches into existing performance improvement
initiatives
by
Holly Burkett, MA, SPHR, CPT
Certified ROI Professional
Principal, Evaluation Works
Davis, California 95616
Phone: 530-400-8875 Fax: 530-756-1906
Email: [email protected]
www.evaluationworks.net
“ROI on a Shoestring: How to Measure More with Less”
for ASTD 2007 International Conference and Exposition
Welcome, Introductions, Agenda
Opening Comments
Ten Shoestring Approaches
1. Plan for evaluation early in the process
2. Build evaluation into the performance improvement process
3. Share the responsibilities for results
4. Require participants to conduct major steps
5. Use short-cut methods for major steps
6. Use sampling to select appropriate programs
7. Use estimates in the data collection & analysis
8. Develop internal capability in results-based evaluation
9. Streamline the reporting process
10. Utilize web-based technology
Adapted from: Phillips, Jack J., (2003) Return on Investment in Training and Performance Improvement
Programs 2nd edition, Butterworth-Heinemann Boston, MA.
ROI is Reported in the Context of Five Levels of Evaluation
™ Jack Phillips, ROI Institute
Evaluation Levels
Level 1 Reaction &
Planned Action
Level 2 Learning
Level 3 Application
Level 4 Business Impact
Level 5 Return on Investment
Level 4

Level 5
1.
2.
3.
4.
5.
Measurement Focus
Measures participant satisfaction with the program or solution and
captures planned actions
Measures changes in knowledge, skills, and attitudes
Measures changes in on-the-job behavior
Measures changes in business impact variables
Compares program benefits to the costs
Moving from Impact to ROI Requires Five Steps
™ROI Institute
Identify program benefits
Convert benefits to monetary value
Tabulate program costs
Identify intangible benefits
Compare monetary benefits to program costs
Notes:
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
by Holly Burkett, MA, SHPR, CPT
[email protected]
All Rights Reserved
Page 1 of 15
Business Impact
To measure extent to which applied
critical skills/knowledge impacted
strategic goal of increasing labor
efficiency
Increased accuracy, quality
ROI
To measure return on investment with
performance improvement strategy
Achieve 120% ROI or greater
Reaction/Satisfaction /Planned Action
To measure participant satisfaction
with career development training
Achieve 4.0 on Overall Satisfaction
and Relevance rating(s)
85% identify planned actions
Learning
Identify individual skills, talents, and
development opportunities per selfassessment guidelines
Demonstrate proficiency with
development discussion steps
Application/Implementation
Complete Development Discussion
with manager within 30 days of
program completion
Apply Critical skills/knowledge to
designated job assignments within 30
days of program completion
Objective(s)
Skill Practice
exercises,
simulations
Skill assessment
(manager,
participant)
Individual
Action Plans
Team project
Follow up
session(s)
Cost benefit
analysis
Impact
Questionnaire
Estimates
Historical data
Performance
monitoring of
Impact
Questionnaire
Reaction
Questionnaire
Impact
Questionnaire
Data Collection
Method
Participants
Managers
HRD
facilitator
Participants
Steering
Committee
Managers
Estimates
Productivity,
labor
efficiency
(cycle time,
rework)
Steering
Committee
Department
Recorder data
Participants
Participants
Managers,
supervisors
Steering
Committee
Data Sources
During action
plan
implementation
2 months after
program
2 months after
action plan
completion
3 months after
program
2 months after
action plan
implementation
During session
Before/during
One week after
After each
session
During session
30, 60 days
Timing
HRD consultant
Subject matter experts
Participants
Managers
Steering Committee
HRD consultant
Project Sponsor
Steering Committee
Participants
Managers
HRD consultant
Subject matter experts
Participants
Managers
Participants
Managers
HRD consultant
HRD consultant
Participants
Managers
Steering Committee
Responsibilities
Adapted from: Burkett, H. “Evaluation: How do you Know your HPI Project was Worth the Effort?”, HPI Essentials, American Society for Training and
Development (2002)
5
4
3
2
1
Evaluation
Level
HRD Initiative/Performance Improvement Program: BETA Career Development Initiative Responsibility: HRD
Plan for Evaluation Early
SAMPLE DATA COLLECTION PLAN
“ROI on a Shoestring: How to Measure More with Less”
for ASTD 2007 International Conference and Exposition
Shoestring Approach 1: Plan for Evaluation Early


Develop objectives of solution
o Strategic business intent
o “Line of sight” between
 Needs
 Causes
 Solution
Develop evaluation plans
o Data Collection Plan
o Use of Baseline Data
Notes:
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
Tool: Data Collection Plan
See Case Example for Career Development Initiative
Shoestring Approach 2: Build Evaluation into the Performance Improvement
Process
A. Develop an Evaluation Policies and Purpose Statement
Tool: Sample Policy and Purpose Statement
The purpose of workplace learning and development initiatives at
Company X is to increase organizational, operational, and individual effectiveness.
Initiatives will offer tangible and intangible returns to customers and will assist Company X
in addressing all factors influencing organizational, operational, and individual performance.
Notes:
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
by Holly Burkett, MA, SHPR, CPT
[email protected]
All Rights Reserved
Page 3 of 15
“ROI on a Shoestring: How to Measure More with Less”
for ASTD 2007 International Conference and Exposition
Shoestring Approach 2: Build Evaluation into the Performance Improvement
Process (continued)
B. Initiate Transition Planning
Sample Transition Planning Actions





Establish milestones
o By the end of 2007, 50% of all programs sponsored by the Leadership Academy
will implement a measurement plan for assessing results with Level 3 objectives.
Develop a communication plan
Solicit an internal task force
Engage a strong, credible sponsor
Determine roles and responsibilities
o WLP Department: The WLP department has overall responsibility to manage the
entire evaluation process from the needs assessment to communication of
results
o Instructional Designer: Ensure that application, impact objectives are developed
for each new program before the design process begins
o Participants: Partner with manager to identify and remove barriers to application
(Adapted from: Phillips, J., Phillips, P,, Stone, R., & Burkett, H. (2006). The ROI Fieldbook: Strategies for
Implementing ROI in HR and Training. Elsevier. Butterworth Heinemann. Burlington, MA)
C. Clarify the Linking Process – Connect the “Value Dots”
Case Example: The Linking Process
for Career Development Initiative
Value Chain
Evaluation
Target
Business Impact
(Level 4)

Application
(Level 3)

Learning
(Level 2)

Reaction,
Planned Action
(Level 1)
Business Need
Increase operational
capacity in meeting strategic
goals; increased
organizational agility
Managers, supervisors will
apply Critical Skills towards
execution of prioritized tasks
Solution Objective
Increased labor efficiency
Managers, participants will
complete Development
Discussion within 60 days of
program completion
Managers, participants will
Participants will demonstrate
define Critical Skills required ability to conduct
for improved labor efficiency
Development Discussion with
their manager
Managers, participants will
 80% identify planned
identify planned intent to
intent to apply learned
apply relevant Critical Skills
skills
to prioritized job tasks
 4.5 rating out of 5.0 on
Overall Satisfaction
Alignment
by Holly Burkett, MA, SHPR, CPT
[email protected]
All Rights Reserved
Evaluation
Target
Business Impact
(Level 4)

Application
(Level 3)

Learning
(Level 2)

Reaction, Planned
Action
(Level 1)
Page 4 of 15
® Link objectives to identified
needs
® Customize curriculum to meet
desired objectives
® Incorporate benchmarked
transfer strategies into course
design
® Design data collection
instruments, evaluation plan(s)
® Assist HRD in job/task analysis
® Attend briefing sessions
® Complete pre-assessment survey
and Pre-work
Human Resource
Development
(HRD)
Adapted from Broad and Newstrom (1992)
Trainees
® Support HRD in defining
performance standards
® Attend briefing sessions prior to
implementation
® Reinforce trainee participation
Managers,
Supervisors
® Attend full program
® Complete self-assessment inventories
® Demonstrate active participation in
skill practices
® Complete Development Discussion
planner
® Communicate importance of learning,
performance, & business objectives
® Assess trainees for reaction, learning,
and skill/knowledge transfer
® Facilitate Pre-work
® Introduce Action Plan
® Implement evaluation plan/tools;
collect, analyze, convert data
® Remove barriers to trainees’
attendance
® Provide coverage for team in training
® Attend sessions as available
® Directly discuss action plan
® Ask trainees about training progress
® Attend, co-facilitate select training
® Co-facilitate meetings and briefings
® Communicate importance of learning,
performance, & business objectives
® Assist in collecting, analyzing,
converting data
During
Before
® Help define performance,
business objectives
® Participate in assessing skill gaps
® Co-facilitate meetings, briefings
® Require attendance at scheduled
briefings
Role
Steering
Committee,
Ops Staff.
Identified Trainers
and/or Consultants
IMPLEMENTATION
PLANNING
Continue implementing evaluation plan
Conduct Action Planning sessions
Facilitate 30/60 day follow up sessions
Report results to key stakeholders
Use results for continuous improvement
® Apply critical skills on the job
® Seek support from Supervisor in
implementing Developing Plan
® Implement Development Plan
® Identify barriers to application
® Complete 30/60 day Impact
Questionnaires
®
®
®
®
®
® Reinforce follow-up and application of
Action Plans
® Assist in removing barriers to application
® Ensure resources are available
® Monitor performance
® Participate in reviewing evaluation plan
® Reinforce follow-up and application of
Action Plans
® Recognize individuals for successful
completions
® Assist in removing barriers to application
® Provide incentives
® Serve as mentor, resource
After
EVALUATION
Case Example: Transfer Strategy Matrix for Career Development Program
Approach 3: Share Responsibilities for Major Steps
“ROI on a Shoestring: How to Measure More with Less”
for ASTD 2007 International Conference and Exposition
Shoestring Approach 3: Share Responsibilities for Major Steps
Notes:
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
Tool: Transfer Strategy Matrix
Shoestring Approach 4: Require Participants to Conduct Major Steps, Including...




Providing, collecting data during solution implementation
Providing, collecting data after solution implementation
Identifying impact on organizational performance due to application of new knowledge/skills
Communicating results of data collection
Case Example: Action Plan for Career Development Initiative
Action Steps
As a result of this program, what specific actions will you
apply based upon what you have learned:
Analysis
What specific business measure will be influenced by your
actions? Labor Efficiency
1.
Conduct Development Discussion with manager,
within 2 weeks of program completion.
1.
What is the unit of measure? Time savings due to less
supervision
2.
Initiate monthly developmental meetings to monitor
progress towards goals.
2.
3.
Provide status reports on Performance Priorities on a
weekly basis, or as determined
As a result of the anticipated changes, please
estimate the monetary benefits to your line or
department over a one month period.
$42 per hour for time saved on less supervision ($504
or $42 x 3 hrs. weekly x 4 weeks)
3.
What is the basis of this estimate? (how did you arrive
at this value?)
Average individual salary & benefits; reduced average
weekly time savings on follow-up meetings regarding
performance priorities/department projects (due to
clearer goal-setting & improved status reporting)
4.
What level of confidence do you place on the above
information? (100%=Certainty and 0%=No
Confidence) 85%
5.
What percentage of the changes above are the direct
result of applying skills learned in the Career
Development Program (0% - 100%)? 60%
6.
What other influences, besides training, might have
influenced these improvements? Improved time
management
Intangible Benefits:
Improved cooperation and teamwork. Better team focus
on performance priorities and the “big picture.” More pride
in my accomplishments, allows more self-direction.
Improved visibility to upper management -- allows me
more time to be strategic for career mobility
by Holly Burkett, MA, SHPR, CPT
[email protected]
What barriers, if any, may prevent you from applying what
you have learned?
Lack of follow up due to work volume, conflicting priorities
from senior management, corporate
All Rights Reserved
Page 6 of 15
“ROI on a Shoestring: How to Measure More with Less”
for ASTD 2007 International Conference and Exposition
Action Planning Considerations:
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
Tool: Action Plan
Shoestring Approach 5: Use Short-cut Methods for Major Steps
A. Just-in-Time Gap Analysis
 Process of bringing key players together to answer critical front-end questions

Puts potential solution systems on the table and surfaces assumptions
o Will this solution successfully bring about the change we’re seeking?
o What could get in the way? Whose support do we need?
o What would it take to roll it out successfully?
 Guiding Principles: __________________________________________________________
_______________________________________________________________________
Case Example: Just-in-Time Gap Analysis for Process Improvement Initiative
Hypothesis 1
Sub Hypothesis
Operators do not get
timely documentation
about process
changes
The
downtime on
subassembly
station 2 is
due to
operator
error
Operators are placed at
the station without
adequate time or
experience to gain task
proficiency
Operators have to wait
for parts or materials to
complete the subassembly in this station
Operators at the
previous station do
not consistently
perform an outgoing
inspection
Key Findings of Focus Group
Process deviations need to be
communicated across all shifts

Address conflicting messages about
this work process in next white board
meeting
This is an intermediate task requiring high
level decision making and repetitive practice
and experience. Errors have major risk
potential.


No new hires at this station
Develop training plan and qualification
to measure proficiency needs and
gaps
Downtime due to parts shortage not being
properly captured/documented by line
supervisors.

Implement training on Downtime
Recorder

Monitor through Daily Production
Checklist
No evidence for this assumption.
Source: Phillips, P. and Burkett, H. (2001) “Managing Evaluation Shortcuts” Info Line Issue 0111. American Society for
Training and Development, Alexandria, Va.
by Holly Burkett, MA, SHPR, CPT
[email protected]
All Rights Reserved
Page 7 of 15
“ROI on a Shoestring: How to Measure More with Less”
for ASTD 2007 International Conference and Exposition
Shoestring Approach 5: Use Short-cut Methods for Major Steps (continued)
Notes:
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
B. Forecast the ROI by:
 Defining the costs

Defining the benefits
o What tangible benefits are required given the expected program costs?
o Is the expectation realistic for obtaining a positive ROI?
o What non-monetary benefits can be expected?

Projecting the ROI

Assessing risks and contingencies: _____________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
Case Example: Forecasting ROI in a Manufacturing Environment
Unit of Measure: Absenteeism
Source of Data: Supervisors, Production Recorder (historical data)
Step 1: Define the cost
o V = $1,061.00 cost of 1 unplanned absence a day x 25 a week = $26,525.00 x 52
weeks = $1,379,300.00 annual cost of unplanned absences
Step 2: Define the benefits
o ∆P = 5 out of 25 unplanned absences decreased per week due to Supervisory
program = 20 a week
o A∆P = $1,061 cost of 1 unplanned absence a day x 20 a week = $21,220 x 52
weeks = $1,103,440 annual cost of unplanned absences
o Non-monetary benefits
Step 3: Project the ROI
o Cost Savings/Benefits = $1,379,300 (25) - $1,103,440 (20) = $275,860
o Solution Costs = $75,000
o ROI = Benefits ($275,860) – Costs ($75,000) = $200,860
Costs ($75,000)
x 100 = 268% ROI
Step 4: Risks, Contingencies
Notes:
________________________________________________________________________
________________________________________________________________________
Tools:
Just-in-Time Gap Analysis
Forecasting
by Holly Burkett, MA, SHPR, CPT
[email protected]
All Rights Reserved
Page 8 of 15
“ROI on a Shoestring: How to Measure More with Less”
for ASTD 2007 International Conference and Exposition
Mid-Point Review: Ten Cost Savings Approaches
1. Plan for evaluation early in the process
2. Build evaluation into the performance improvement process
3. Share the responsibilities for evaluation
4. Require participants to conduct major steps
5. Use short-cut methods for major steps
Shoestring Approach 6: Use Sampling to Select Appropriate Programs




Determine resources available for measuring results and impact
Match resources with the number of evaluations targeted at each level
Develop criteria for selecting programs at higher level evaluation
Select the specific courses and review with advisors/managers
Example: ROI Targets
Evaluation Levels
Level 1
Level 2
Level 3
Level 4
Level 5
Measures
Reaction, Satisfaction
Learning
On-the-job application
Business Impact
Return-on-investment
Percent
100%
60%
30%
20%
5-10%
Tools:
Selection Criteria
Transition Plan
Sample Transition Plan for Increasing Evaluation Targets
Year
0
Year
1
Year
2
Year
3
Year
4
Year
5
Reaction and
Planned
Action
85%
90%
95%
100%
100%
100
Learning
30%
35%
40%
45%
50%
60%
Application
5%
10%
15%
20%
25%
30%
Business
2%
4%
5%
9%
12%
15%
2%
4%
6%
8%
10%
Result 0%
OI
Notes:
s
_____________________________________________________________________________
ROI
_____________________________________________________________________________
_____________________________________________________________________________
Shoestring Approach 7: Use Estimates in the Data Collection and Analysis
Sample Impact Questions Showing Use of Estimates
 How often will you apply this behavior?
 What specific business unit of measure will be influenced by your actions?
by Holly Burkett, MA, SHPR, CPT
[email protected]
All Rights Reserved
Page 9 of 15
“ROI on a Shoestring: How to Measure More with Less”
for ASTD 2007 International Conference and Exposition
Shoestring Approach 7: Use Estimates in the Data Collection and Analysis
(continued)






As a result of these changes, please estimate the monetary benefits to your work unit over a
one month period.
What is the basis for your estimate?
What level of confidence, expressed as a percentage, do you place on the above estimate?
(100% = Certainty and 0% = No Confidence)
What other factors, besides training, may contribute to benefits associated with your applied
skills or knowledge?
Other factor
____%
What specific actions will you apply based upon what you have learned?
Case Example: Participant Estimates for Career Development Initiative
Calculation: A X B X C X D = E (Total adjusted monthly benefit)
E X 12 = F (Annualized benefit)
#
Unit of
Measure
Monthly
Improvement
Value
Confidence
Level
%
Change
Due to
Program
(C) X
50%
% Time
Applied to
Productive
Tasks
(D)
60%
Total
Adjusted
Monthly
Benefit
= (E)
$5,400
(A) X
(B) X
Error
$36,000
50%
rates
2
Quality
$24,000
60%
45%
70%
$4,536
3
Time
$18,000
85%
80%
75%
$9,180
4
Rework
$15,000
70%
85%
90%
$8,032.50
Total Annualized Benefit Directly Attributable to Training
ROI Calculation: Net Benefits Attributable to Training – Training Costs = ($195,700)
Training Costs ($83,300) x 100 = ROI %
1
Annualized
Program
Benefit
(use 12)
(F)
$64,800
$54,432
$110,160
$96,390
$325,782
235% ROI
Guidelines for the Use of Estimates:
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
Tools/Data Sources:
Action Plan, Impact Questionnaire
Shoestring Approach 8: Develop Internal Capability for Results-Based Evaluation





Provide on-line needs assessments, surveys, tools/templates
Promote education, cross-training, sponsorship
Publish testimonials using newsletters, e-mail, Web pages, or on-line Discussion Groups
Create an internal listserv to promote dialogue
Involve multiple stakeholders in evaluation planning and project Action Plans
by Holly Burkett, MA, SHPR, CPT
[email protected]
All Rights Reserved
Page 10 of 15
“ROI on a Shoestring: How to Measure More with Less”
for ASTD 2007 International Conference and Exposition
Shoestring Approach 9: Streamline the Reporting Process
Case Example: Streamlined Report
Background
 Business Objectives:
 Enhance individual and organizational bench strength
 Increase labor efficiency
 Improve job satisfaction
 Enhance organizational “brand” as employer of choice
 Increase organizational agility and flexibility

Performance Objectives:
 Implement development discussions with immediate supervisor
 Implement career enrichment action plan
 Integrate Career Development strategies with department, team, and business
performance plans
 Apply critical skills to defined performance priorities

Learning Objectives:
 Define critical skills required for job effectiveness
 Define skill gaps
 Identify talents
 Identify developmental needs
Tool:
Streamlined Impact Report
Shoestring Approach 10: Utilize Web-based Technology



For data collection, data analysis
For benchmarking
o ASTD’s Evaluation & ROI Learning Community at www.astd.org
o ASTD ROI Network
Considerations
Notes:
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
Tools: _______________________________________
by Holly Burkett, MA, SHPR, CPT
[email protected]
All Rights Reserved
Page 11 of 15
“ROI on a Shoestring: How to Measure More with Less”
for ASTD 2007 International Conference and Exposition
Approach 9: Streamline the Reporting Process
Sample Streamlined Impact Report
ROI Impact Study
Program Title: Career Development Initiative
Target Audience: First and second line Managers/Supervisors; Professional staff
Duration: 1 day
Technique to Isolate Effects: Participant estimation, trend analysis
Technique to Convert Data to Monetary Value: Historical costs, internal experts, estimates
Fully Loaded Program Costs: $83,300
Reaction,
Planned
Action
Results
Overall
satisfaction
rating: 4.8
I would
recommend
this program
to others: 4.8
Program was
relevant to my
daily job: 4.5
92% reported
intention to
conduct a
Development
Discussion
with their
manager
within 30 days
of the
workshop
Learning
Results
Application
Results
Participants
completed selfassessment of
• skills
• gaps
• talents
• development
needs
94% conducted
a Development
Discussion with
their manager
within 60 days of
workshop
I have a better
understanding of
my Performance
Priorities for the
next 6-12
months: 4.48
I have a better
understanding of
my Development
Needs as they
relate to my
current position:
4.37
I having a better
understanding of
my Talents as
they relate to my
current position:
4.26
76% apply
Critical Skills
from
Development
Discussion Plan
52% are
enhancing
development
through
development
plans approved
by their manager
Business
Impact
(directly
attributable to
training)
Monetary
benefits of
increased
productivity:
$187,000
Monetary
benefits from
increased
efficiencies:
$ 92,000
Return on
Investment
Intangible
Benefits
ROI (%) =
Improved
relationship with
immediate
manager
Net
Benefits - Costs
Costs
$195,700
$83,300
=
235% ROI
Increased
perception that
employer cares
about
employee’s
career growth
Increased
confidence in job
role
Improved ability
to view
performance
priorities in
relation to “big
picture”
Anticipated plan
to stay with
employer for
next 12 months
Burkett, H. ”Evaluating a Career Development Initiative“ in Kirkpatrick, Donald L. Evaluating Training Programs:
The Four Levels. 3rd Edition San Francisco, California Berrett-Koehler Publishers, Inc., 2005.
“ROI on a Shoestring: How to Measure More with Less”
for ASTD 2007 International Conference and Exposition
Summary: “Shoestring” Strategies for Building and Sustaining Organizational
Support
1. Assess _________________
2. Promote internal capabilities
3. Build business partnerships
4. Clarify the effort

Define results-based evaluation as a compelling element of organizational strategy

Position the Workplace Learning function as _________________

Focus on a few key measures
5. Consolidate, compress steps
6. __________________________
7. Commit to ____________________
Notes:
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
Closing Review of Key Concepts: Ten Cost Savings Approaches
6. Use sampling to select appropriate programs
7. Use estimates in the data collection & analysis
8. Develop internal capability in results-based evaluation
9. Streamline the reporting process
10. Utilize web-based technology
About the Presenter
Holly Burkett MA, SPHR, CPT, Principal of Evaluation Works, has more than 20 years
experience assisting public and private sector organizations design and implement evaluation
systems, processes, and tools to align program, client, and business outcomes for a wide range
of initiatives. Formerly with Apple Computer, she led the operation’s first HRD impact studies in a
resource constrained, dynamic environment. Sample clients include: the National Security
Agency (NSA): the State of California; the University of California Davis; Premera Blue Cross; the
Community HealthCare Association of the Dakotas (CHAD); Intel; and Chevron.
Certified as an ROI professional since 1997, Holly facilitates ROI on-line courses offered through
Villanova University and the ROI Institute. Editor-in-Chief of ISPI’s Performance Improvement
Journal, she is a frequent conference presenter, workshop leader, and author on performance
measurement topics. Most recent publications include co-authoring The ROI Fieldbook with Jack
rd
and Patti Phillips and Ron Stone (2006) as well as an ROI case study in the 3 edition of Donald
Kirkpatrick’s Evaluating Training Programs: The Four Levels (2005). As a certified Senior
Professional in Human Resources (SPHR), she also serves as a select Item Writer for the Human
Resource Certification Institute’s (HRCI) SPHR/PHR credentialing exams. Holly earned her
Masters in Human Resources and Organization Development from the University of San
Francisco and is currently pursuing doctoral studies in Human Capital Development. She can be
reached at [email protected].
by Holly Burkett, MA, SHPR, CPT
[email protected]
All Rights Reserved
Page 13 of 15
Job Aid: Cost Savings Approaches for Measuring More with Less
Approach 1. Plan for evaluation early in the
process.
Approach 6. Use sampling to select appropriate
programs.
Guideline: Define business needs, establish evaluation
purposes, determine target evaluation levels, develop
project objectives, and determine evaluation timing.
Guideline: Only specific types of programs should be
selected for a comprehensive, detailed analysis. Set
targets for the number of programs to be evaluated at
each level.
Tool(s): Data Collection Plan.
Approach 2. Build evaluation into the performance
improvement process
Guideline: Link business needs, program objectives, and
evaluation targets through-out the entire cycle of training
needs assessment, instructional design, delivery, and
evaluation. Establish an infra-structure of evaluation
policies, procedures, guidelines, and operating standards.
Tool. Selection Criteria, Transition Plan.
Approach 7. Use estimates in the data collection
and analysis.
Guideline: Estimates can save a great deal of
resources in the isolation and data conversion steps.
Use the most credible and reliable sources for
estimates, take a conservative approach, and develop a
culture for the estimation process.
Tool(s): Policy and Purpose Statements, Transition
Planning, Linking Process, Data Collection Plan.
Tool(s): Action Plans, Impact Questionnaire.
Approach 3. Share responsibilities for evaluation.
Approach 8. Develop internal capability.
Guideline: Invite managers and performers to provide
input on performance and skill deficits; ask stakeholders
to review and approve evaluation plans; and collect
participant and key stakeholder feedback data after the
program.
Guideline: Communicate purpose and scope of ROI
process as a continuous learning tool that will help
assess program priorities and areas of impact.
Develop staff capability and shared ownership
through education and training and targeted
development plans.
Tool(s): Transfer Strategy Matrix, Impact Questionnaire.
Approach 4. Require participants to conduct major
steps.
Guideline: Hold participants accountable for learning and
applying new skills and knowledge and for identifying
enablers and barriers to planned application.
Tool(s): Professional Development, Outsourcing, ROI
Certification, Task Forces, Case Studies.
Approach 9. Streamline the reporting process.
Tool: Action Plan.
Guideline: Once management is comfortable with the
process of ROI evaluation and a results-based
measurement focus has integrated into the
organization, a streamlined approach to reporting
results may be more appropriate and cost-effective.
Approach 5. Use shortcut methods for major steps.
Tool: Streamlined Impact Report.
Guideline: Use just-in-time solutions for gap analysis,
solution design, and data collection. Caution against an
over-reliance on short-cut methods and a “quick-fix”
mentality.
Tool(s): Just-in-Time Gap Analysis, Forecasting, Impact
Questionnaire.
Approach 10. Utilize web-based technology.
Guideline: Use suitable software packages to speed up
various aspects of ROI analysis, design, evaluation,
and reporting. Use technology to enhance internal
capability by offering on-line needs assessment, selfassessments, or evaluation templates for key
stakeholders.
Resources: Knowledge Advisors, Survey Monkey,
Survey Pro.
Adapted from: Phillips, J., Phillips, P. Burkett, H. and Stone, R. The ROI Fieldbook, Boston, MA: ButterworthHeinemann, 2005.
Shoestring Evaluation Approach
No
No
No
No
No
No
No
No
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
No
No
No
No
No
No
No
No
Adapted from: Burkett, H. and Phillips, P., Managing Evaluation Shortcuts
1. Plan for Evaluation Early
a. Objectives of solution clearly defined?
b. Evaluation plan developed?
2. Build Evaluation into the Performance Improvement Process
a. Evaluation policies, procedures defined?
b. Transition plan, with specific, achievable milestones, developed?
c. Link between business needs, program objectives, and evaluation levels?
3. Share Responsibilities for Results
a.Transfer strategies defined?
b.Participant responsibilities defined?
c. Manager/supervisor responsibilities defined?
d. WLP responsibilities defined?
e. Sponsor responsibilities defined?
4. Require Participants to Conduct Major Steps
a. Participants involved in performance analysis?
b.Participants involved in content review, scenario design?
c. Action planning in place?
5. Use Short-cut Methods for Major Steps
a. Just-in-time gap analysis conducted?
b. Data collection steps compressed?
6. Use Sampling to Select Appropriate Programs
a.Criteria established for evaluating programs for impact at Level 3 or above?
b. Criteria matched to available resources?
7. Use Estimates in Data Collection & Analysis
a.Participants asked to estimate impact of applied knowledge/skills upon
targeted business measures?
b. Participants asked to estimate impact of other influences upon performance
improvement?
c. A culture for use of WLP estimates established?
d. Credible data sources utilized for providing estimates?
8. Develop Internal Capability in Results-based WLP Evaluation
a. Resuls-based education, cross-training programs in place?
b. On-line needs assessment or data collection tools available?
c. Internal task force or steering committee in place?
9. Streamline the Reporting Process
a. Streamlined report process adopted?
b. Mechanisms in place to communicate results through-out initiative?
c. Report content specific to stakeholder needs?
d. Results used to influence decisions about WLP resource allocation?
10.Utilize Web-based Technology
a. Technology used for data colletion, analysis, or reporting?
Program or WLP Initiative
1
2
3
Yes
No
No
Comments
Job Aid: A Checklist for Matching Cost-Effective Evaluation Strategies to WLP Programs/Initiatives