Evidence-based? Promising? What
is the Difference and Why Does it
Matter?
David W. Test & Ruth Allison
National Technical Assistance Center on
Transition
Purposes
• To overview concept of evidence-based
practices (EBPs)
• To overview different definitions for EBPs,
including NTACT’s
• To introduce NTACT’s Best Available Evidence
Matrix
• To introduce NTACT’s “Other Topic” annotated
bibliographies
Federal Mandates
• NCLB (2001) requires schools ensure students
have access to effective scientifically based
instructional strategies and defines “scientifically
based research” as “research that…involves the
application of rigorous, systematic, and objective
procedures to obtain reliable and valid
knowledge relevant to education activities and
programs”[20 U.S.C. § 7901(37)].
• IDEA (2004) calls for the use of scientifically
based instruction with students with disabilities
Federal Mandates (cont.)
Workforce Innovation Opportunity Act (WIOA)
allows for the implementation of evidence based
findings to improve policy, procedure, practice,
and the preparation of personnel in order to
improve the transition of students with
disabilities from school to postsecondary
education or and employment outcome.
[Proposed §361.48 (a)(3)(vi)]
Scientifically Based Research
(NCLB, 2001)
The term “scientifically based research”• (A) means research that involves the application
of rigorous, systematic, and objective procedures
to obtain reliable, and valid knowledge relevant
to education activities and programs: and
• (B) includes research that• (i) employs systematic, empirical methods that
draw on observation or experiment
• (ii) involves rigorous data analyses that are
adequate to test the stated hypothesis and justify
the general conclusions drawn
Scientifically Based Research
(NCLB, 2001)
• (iii) relies on measurements or observational methods
that provide reliable and valid data across evaluators
and observers, across multiple measurements and
observations, and across studies by the same or
different investigators:
• (iv) is evaluated using experimental or
quasiexperimental designs in which individuals,
entities, programs, or activities are assigned to
different conditions and with appropriate controls to
evaluate the effects of the condition of interest, with a
preference for random-assignment experiments, or
other designs to the extent that those designs contain
within-condition or across-condition controls:
Scientifically Based Research
(NCLB, 2001)
• (v) ensures that experimental studies are presented in
sufficient detail and clarity to allow for replication or, at a
minimum, offer the opportunity to build systematically
on their findings; and
• (vi) has been accepted by a peer-reviewed journal or
approved by a panel of independent experts through a
comparably rigorous, objective, and systematic review.
Council for Exceptional Children
Published 3 special issues of Exceptional
Children
a. 2005; 71(2); quality indicators and
criteria for EBPs*
b. 2009; 75(3); examples of using quality
indicators and EBP criteria
c. 2013; 79(2); implementing and
sustaining EBPs
Other Terms.1
(US Office of Management and Budget, 2015)
• “Evidence-Based” level is consistent with the What Works
Clearinghouse standards and includes evidence based on
randomized experimental methods or quasi-experimental strategies
that use propensity matching or other strategies to insure a
matched comparison group as well as meta-analyses of such
studies.
• “Evidence-Informed” level refers to evidence gathered through
longitudinal studies and meta-analyses. Although the evidence is
primarily correlational in nature, the evidence has either been
demonstrated as stable over time or across multiple studies and is
therefore deemed robust enough to make conclusions that can
guide practice and policy-making efforts.
• “Promising” level relies on correlational data that is consistent with
existing theory.
Other Terms.2
(RRTC on Evidence-Based Practice in VR)
• Emerging offers knowledge about what works and does
not work, but doesn’t have evaluation data to
demonstrate its effectiveness( Puddy & Wilkens, 2011;
Twyman & Sota, 2008)
• Promising includes emerging practice elements and adds
programmatic quantitative data that demonstrates
positive outcomes, but doesn’t contain research data to
support its replication (De Palma, 2002)
• Evidence-based is a practice with statistical and clinical
significance and encompasses emerging, to promising, to
evidence-based (Driever, 2002)
Other Terms.3
(What Works Clearinghouse, Version 3.0)
• Rates studies as having:
–
–
–
–
–
–
Positive effects
Potentially positive
No discernible effects
Mixed effects
Potentially negative effects
Negative effects
• Then determines “extent” of evidence
(medium to large or small) based on # of
studies, settings, and participants
What are we talking about?
– Terms are not at all consistent.
– All combinations have been used:
A
B
C
Evidence
Based
Practice(s)
Research
Validated
Treatment(s)
Scientifically
Empirically
Program(s)
Supported
Intervention(s)
– As a speaker and listener, remember:
• Usage is NOT CONSISTENT across the field of
education
– We cannot assume anything from the term alone
– We must attend to the procedures, not the label
(Detrich, Spencer, & Slocum, 2009)
Levels of Practice
(Helsel, Hitchcock, Miller, Malinow, & Murray, 2006; Twyman, 2008)
Evidence-Based
Practices
• Are based on rigorous research designs
• Have demonstrated a record of success for improving
student outcomes
• Have undergone systematic review process using quality
indicators to evaluate level of evidence
Research-Based
Practices
• Are based on rigorous research designs
• Have demonstrated a record of success for improving
student outcomes
Promising
Practices
Unestablished
Practices
• Are based on research
• Have demonstrated limited success
• Have used a ‘weak’ research design
• Are not based on research
• Have no data to support effectiveness
• Based on anecdotal evidence and/or professional judgment
How Do You Identify an EBP, RBP,
PP,…?
• Start by identifying a “practice.”
• Horner et al. (2005) defined it as “A practice
refers to a curriculum, behavioral intervention,
systems change, or educational approach
designed for use by…with the express
expectation that implementation will result in
measurable educational, social, behavioral, or
physical benefit.” (p. 175).
ID'ing Practices continued…
• Once a practice is identified, it can be
useful to also identify a DV and the target
population
• For example: “The use of video modeling
to teach job skills to students with
cognitive disabilities.”
• Doing this will help narrow the coming
literature search
Once You Have Reviewed the Literature and Identified
Articles That Meet Your Inclusion Criteria,
How do You Decide if it is Evidence-Based?
• First, you must determine the quality of the
research studies that used the practice
– See NTACT’s Quality Indicator Checklists for
group, single-subject, correlational, and
qualitative research (ask us)
• Then, you must determine if you have enough
studies to meet some standard
– See NTACT’s Definitions for Levels of Evidence
(presented later)
Now that that is clear, let’s muddy
the water a bit…
• The field is now talking about “evidence-based
practice” as a way of doing business, as a
decision-making process. For example…
RRTC on EBP in VR
Evidence Based Practice (EBP):
“…should involve the integration of the best and most
current research evidence with clinical/educational expertise
and relevant stakeholder perspectives in the pursuit of
making the best possible decisions for a particular
consumer.”
From:
Integrating Research into VR Practice: A guide to integrating research into
Vocational Rehabilitation Practice (2015)
http://www.research2vrpractice.org/training/EBPintoVRPracticePUB2/index.htm
RRTC on EBP in VR
4 steps to Integrating Research into VR Practice:
1. Formulate well-defined, answerable questions
2. Seek the best evidence available to answer your
questions
3. Critically evaluate the evidence
4. Apply the evidence to your individual consumer
From: Integrating Research into VR Practice: A guide to integrating research into
Vocational Rehabilitation Practice (2015)
http://www.research2vrpractice.org/training/EBPintoVRPracticePUB2/index.htm
Evidence-Based Practice…
…is a decision-making process that
integrates:
– Top-tier research
– Relevant experience-based knowledge
– Current policy for improving outcomes for
children and youth with disabilities and their
families
(Buysse, Wesley, Snyder, & Winton, 2006; Turnbull et al., 2010)
Another Definition
 Evidence-Based Practice is a process of
making decisions informed by four sources
of influence:
a) frameworks or principles
b) best available research evidence
c) professional judgment
d) student needs and values
(Detrich, Spencer, & Slocum, 2009)
So, is Evidence-Based Practice a
“Noun” or a “Verb”?
• Noun = an intervention; based on high quality
research; an evidence-based practice; Slocum et
al. (2014) used the term “Empirically Supported
Treatment”
• Verb = a way of acting; Evidence-based practice;
evidence-informed practice
• Either way, where should a professional start?
With a practice that has shown a greater chance
of success, or with a practice where outcomes are
unknown?
What is NTACT Doing About EBPs?
Teaching practitioners to use evidence-based
practice (as a verb; as a way of acting) by
providing practitioners with access to the
best available evidence on practices (as a
noun; a specific practice) that ensure all
students with disabilities are prepared to
succeed in postsecondary education and
employment.
Best Available Evidence Matrix
(see handout)
Level of Evidence
Secondary Special Education
Evidence-based Practices
•
Are based on rigorous
research designs
•
Have demonstrated a
record of success for
improving student
outcomes
•
Have undergone systematic
review process using
quality indicators to
evaluate level of evidence
•
•
Research-based Practices
•
Are based on rigorous research
designs
•
Have demonstrated a record of
success for improving student
outcomes
•
•
VVocational Rehabilitation
•
1 Student-Focused Planning
5 Student-Development (Life
Skills)
(NSTTAC, Practice Descriptions)
1 Student-Focused Planning
26 Student-Development
(Academic, Employment, and
Life Skills)
(NSTTAC, Practice Descriptions)
•
6 Predictors (e.g., inclusion,
paid employment)
(NSTTAC, Predictor Resources)
Graduation/ School Completion/
Dropout Prevention
Career and Technical Education and
Career Development
10 Dropout Prevention
Programs (e.g., Check &
Connect, RENEW)
(Wilkins & Huckabee, 2014. NDPC-SD,
Literature Map)
•
Counseling and working
alliance between counselor and
consumer
•
Skills training
•
Supported Employment (for
people with ID/DD and severe
mental illness; individual
placement and support model;
IPS)
•
Overall Effect of VR
Interventions
(Journal of Vocational Rehabilitation,
2014)
•
Counseling and working
alliance between counselor and
consumer
•
Skills training
•
Supported Employment (for
people with ID/DD and severe
mental illness; individual
placement and support model;
IPS)
•
Overall Effect of VR
Interventions
(Journal of Vocational Rehabilitation,
2014)
Criteria for Levels of Evidence
(see handout)
Evidence-Based Practice
Group Experimental Design
•
•
•
Single-Case Design
•
•
•
Correlational
•
•
•
•
1
Two high quality1 or a combination of four high and acceptable quality2 studies
using rigorous research designs demonstrating positive effects
Studies must calculate effect size or report data that allows for calculation
There is no evidence from a high or acceptable quality study demonstrating
negative effects
A combination of five high3 or acceptable4 quality studies using rigorous research
designs demonstrating a functional relation
Three independent research teams
There is no evidence from a high or acceptable quality study demonstrating
negative effects
Two high quality a priori (planned, hypothesis stated) studies5 demonstrating
consistent significant correlations between predictor and outcome variables
Studies must use high quality6 propensity score modeling/matching
Studies must calculate effect size or report data that allows for calculation
There is no evidence from a high a priori study demonstrating negative
correlations between predictor and outcome variables
High quality group experimental study must meet 1, 2, 3, 4, 6, 8, 9 & 10 and 5 or 7 of EQIs and at least 4 of the DQIs Quality Indicator Checklist
for Group Experimental Research
2
Acceptable quality group experimental study must meet 1, 2, 3, 4, 6, 8, 9 & 10 and 5 or 7 of EQIs and at least 1 of the DQIs Must calculate effect
size or report data that allows for calculation Quality Indicator Checklist for Group Experimental Research
3
High quality single-case study meets all quality indicators Quality Indicator Checklist for Single-Case Research
4
Acceptable quality single-case study meets all QIs except 2 & meets one of 17-20 Quality Indicator Checklist for Single-Case Research
5
High quality a priori studies must meet all quality indicators for correlational research
6
High quality propensity score modeling studies must meet all quality indicators for correlational research
Other Topics
(see handout)
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Literature Review/ Resource Identification
Other Topics
Professional Development/ Technical Assistance/ Coaching
Data-based decision making – student level/ systems level
Collaboration/Consultation/Co-teaching
Universal Design for Learning (UDL)
Peer Supports
Positive Behavioral Interventions and Supports (PBIS) -secondary level
Response to Intervention (RTI) - secondary level
Interagency Collaboration
Academic Instruction – secondary level
Mental Health
Culturally Responsive Instruction
Business Connections
Family Engagement and Family Expectations
Assistive Technology
College Ready Programs
Assessment Practices – career, statewide data
Healthcare
Transportation
Supporting Students with Complex Support Needs
Pre-Employment Transition Services
What are we doing
with the “Other Topics”?
• Updating or creating Annotated Bibliographies for each
topic
• Creating 1-2 page summaries for each topic, with links to
possible resources
Your Turn!
• Will the Best Available Evidence Matrix be
useful?
• What more is needed to support
implementation of these practices?
• What additional “Other Topics” are
needed?
Thank You!
David W. Test: [email protected]
Ruth Allison: [email protected]