The Weak Links in RtI & How to Do It Right 2 RtI How Do the Pieces Fit Together ? How Could This Work in Your School ? Meet Exit Criteria Tier 1 Instruction for All Students Do you have at least the 13 Measurable Components in Place ? Check Integrity ? Universal Screening of Core Content (3/4#) Grade Level Learners Struggling Learners who : No Previous Intervention How are Things Working ? Tier 2 Intervention ‘A’ Tier 2, Intv ‘A’ Not Effective Tier 2 Intervention ‘B’ Problem Solving Comm Tier 2, Intv ‘A’ & ‘B’ Not Effective, Tier 3 Intervention (Custom) Check Integrity ? Meet Exit Criteria Image modified - Thanks to Sharon Vaughn for the original image Suspect a Disability ? If RtI2 has a Weak Link : What Must be Done 8 Interchangeable Terms: • Integrity • Fidelity • Procedural Reliability Definition: “the degree to which an intervention program is implemented as planned” (Gresham et al., 2003) Facilitating Factors Intervention Implementer Student • Acceptability • Level of training • Motivation • Rate of Change Produced • Motivation • Cooperation Perepletchikova and Kazdin, 2005 Discouraging Factors Intervention Implementer • Complexity • Resistance • Multiple resources • Student diversity • Time required • Exposure to other interventions Student • Difficult behavior • Severity/ duration of problem Perepletchikova and Kazdin, 2005 Why is Tier One Integrity Important → Need to be sure Tier 1 functions properly Before deciding Who Needs More Intervention (Tiers 2 & 3) → Too Many students (more than 30%) may Need Additional Services (Tiers 2 & 3) if Tier 1 is not functioning properly → Accountability Wickstrom’s Wisdom • What Happens When You Ask Teachers if They are Implementing an Intervention? report “Yes” • What is observed of the Same Teachers?  (actually Doing It) 11 How Do You Measure Implementation Integrity ? Measuring Implementation • Self-Monitoring and SelfReport • Direct Observation • Student Performance/ Permanent Products • Behavioral Data • Academic Data 13 Self-Monitoring/Self-Report • Main Question: Did you do the intervention? • Rating Scales • Checklists • Interviews • Some Pitfalls to Self-Reporting – People may do a little bit and give themselves credit for actually doing it – It is difficult for people to report accurately when they lack integrity/fidelity Research on Self-Report Witt et al. studied 33 intervention cases • Teachers agreed to do an intervention and were then observed in class • _____on a self report measure indicated that they had used the intervention as specified by the team • _____ Teachers had integrity/fidelity above ____ What Do WE Know about Assessing and Improving Fidelity of RTI, Joseph C. Witt, LSU, Senior Scientist, iSTEEP Learning Observations Direct Observation • An independent observer observes school wide implementation • Walk –Thrus (2,010,000 hits) • Peer Observations • Peer Monitoring Teacher Behaviors in Math Teacher follows script sufficiently to ensure integrity/fidelity of implementation Teacher implements each step (modeling, GP, IP) sufficiently to ensure integrity/fidelity of implementation Teacher implements self-correct/ EC following IP to ensure students learn IC Teacher maintains brisk pace Teacher provides corrective feedback immediately as needed Teacher talk is kept to a minimum and is characterized with short requests "What answer?" "How many?" Teacher engages students throughout lesson with a response that is verbal, written, or hands-on Teacher models using "think aloud" Most Some of the of the Time time Rarely Not at 3 2 1 all 0 http://www.centeroninstruction.org/files/Adol%20Lit%20Walk%20Thro ugh.pdf Teacher Behaviors in Math Most Some of the of the Time time Rarely Not at 3 2 1 all 0 Teacher follows script sufficiently to ensure integrity/fidelity of implementation Teacher implements each step (modeling, GP, IP) sufficiently to ensure integrity/fidelity of implementation Teacher implements self-correct/ EC following IP to ensure students learn IC Teacher maintains brisk pace Teacher provides corrective feedback immediately as needed Teacher talk is kept to a minimum and is characterized with short requests "What answer?" "How many?" Teacher engages students throughout lesson with a response that is verbal, written, or hands-on Teacher models using "think aloud" 21 Permanent Product Review Another Preferred Option: Permanent Products • Permanent products – Research – Practice • Leaves traces that it was done • Example: Computer and/or Paper Record – Number of tardies – Number of absences – Correct and incorrect responses on academics Review Permanent Products  Task Completion  Frequency  Be Cautious www.edhelper.com/math Assessment Fidelity Deborah Reed et al What If • Need data on literacy of Middle Schoolers • Paid 29 educators to test • Trained for ½ day (expert prepared content) • Each tester certified 99% accurate How Accurate would their Data be? Here’s What Happened • _____had ‘correctable errors’ in scoring • Note: ____ of test packets thrown out - ‘significant errors’ DIBELS data had correctable errors* • Another Study: ____ * Sentinel Schools submitted to U of OR Center A Practitioner’s Experiment How Many Checklists Found in 1 hour ? Usual Suspects “Taken for Granted” Please Add More How to Create An Integrity Checklist. . . 36 Core Questions • What does the intervention look like when it’s in use? • What would be seen in classrooms where it is used? • What will teachers and students be doing when the intervention is in use? (Hall & Hord, 2001) Making the Checklist • Identify the steps in the intervention plan • Create an item for each step in the plan • Yes/No response for each item Scoring the Checklist • Observe the intervention in action • For each checklist item evaluate - Was that step completed? • Total “Yes” Responses • Divide the number of “Yes” responses by the total number of items on the checklist • Integrity percentage yielded! 1. Select a Measure 2. Ask Teachers for Feedback on Social Validity 3. Teachers use the Measure with each other 4. Principal/PLCs use the Implementation Integrity Measure Let’s Talk About Tier 2 & Tier 3 Integrity You have a Result, But Can You Make a Judgment? 25 20 15 10 5 0 14-Jan 21-Jan 28-Jan 4-Feb 11-Feb 18-Feb Samples High School Tier 1 Instruction for All Students English I - Grade Level Learners Universal Screening - Benchmark Test Struggling Learners who : No Previous Intervention Meet Exit Criteria Image modified by Coulter- Thanks to Sharon Vaughn for the original concept/image Determine Goal Benchmark Assessment 100 80 60 Universal Screening Data 40 24 Class: 24 (Expected Entry score = 25) 11 Egberta : 11 20 0 Weeks 0 1 2 3 4 5 6 7 8 9 Reschly RTI, 2008 10 12 14 16 18 20 Samples High School Tier 1 Instruction for All Students English I Universal Screening of WCPM Grade Level Learners Struggling Learners who : No Previous Intervention How are Things working ? Tier 2 ‘A’ Your Favorite Meet Exit Criteria Image modified by Coulter- Thanks to Sharon Vaughn for the original concept/image Monitor Egberta’s Progress Relative to Goal Words Correct Per Minute 100 Class Egberta Goal Egberta Baseline 2 5 10 Egberta Int 80 60 40 24 20 11 0 Weeks 0 1 3 4 6 7 8 9 Reschly RTI 12 14 16 18 20 Samples High School Tier 1 Instruction for All Students English I Universal Screening of WCPM Grade Level Learners Struggling Learners who : No Previous Intervention How are Things working ? Tier 2 ‘A’ “Your Favorite” Integrity Check Meet Exit Criteria Image modified by Coulter- Thanks to Sharon Vaughn for the original concept/image 32 Monitor Egberta’s Progress Relative to Goal Words Correct Per Minute 100 Class Egberta Goal Egberta Baseline Egberta Int 10 16 80 60 Integrity 83% 40 24 20 11 0 Weeks 0 1 2 3 4 5 6 7 8 9 Reschly RTI 12 14 18 20 Samples High School Tier 1 Instruction for All Students English I Universal Screening of WCPM Grade Level Learners Struggling Learners who : No Previous Intervention How are Things working ? Tier 2 ‘A’ “Your Favorite” Tier 2, Intv ‘A’ Not Effective Tier 2 ‘B’ “Buddy’s Favorite” Meet Exit Criteria Image modified by Coulter- Thanks to Sharon Vaughn for the original concept/image Formative Evaluation: Change Intervention Class Egberta Int 1 Words Correct Per Minute 100 Egberta Goal Egberta Int 2 Egberta Baseline 80 Intervention Change 60 40 24 20 11 0 Weeks 0 1 2 3 4 5 6 7 8 9 Reschly RTI 10 12 14 16 18 20 Samples High School Tier 1 Instruction for All Students English I Universal Screening of WCPM Grade Level Learners Struggling Learners who : No Previous Intervention How are Things working ? Tier 2 ‘A’ “Your Favorite” Tier 2, Intv ‘A’ Not Effective Tier 2 ‘B’ “Buddy’s Favorite” Integrity Check Zoom Ahead Alan Meet Exit Criteria Image modified by Coulter- Thanks to Sharon Vaughn for the original concept/image Gap Still Not Closing: Alt. Int. & Consider Eligibility Words Correct Per Minute 100 Class Egberta Goal Egberta Baseline Egberta Int 1 Egberta Int 2 Egberta Int 3 80 Intervention Change 60 Integrity 83% Integrity 87% Integrity 100% Integrity 70% 40 24 20 11 0 Weeks 0 1 2 3 4 5 6 7 8 9 Reschly RTI 10 12 14 16 18 20 Samples High School Tier 1 Instruction for All Students English I Universal Screening of WCPM Grade Level Learners Struggling Learners who : No Previous Intervention How are Things working ? Tier 2 ‘A’ “Your Favorite” Tier 2, Intv ‘A’ Not Effective Tier 2 ‘B’ “Buddy’s Favorite” Check Problem Solving Integrity ? Comm Tier 2, Intv ‘A’ & ‘B’ Not Effective, Tier 3 “Peer Tutoring” Suspect a Disability ? Meet Exit Criteria Image modified by Coulter- Thanks to Sharon Vaughn for the original concept/image What Works to Improve Integrity? • Training, including Modeling, Coaching, & Feedback • Routine Integrity Checks with Feedback • Routine Progress Monitoring with Feedback Performance Feedback Increases Implementation Integrity Investing in Coaches: Avg. Referrals per Day Average Major Discipline Referrals per Day per Month Coach returns from leave 7 6 5 4 3 2 1 0 Sep 05-06 Oct Nov 06-07 One School’s Example Dec Jan Feb Mar Apr May Source: McGlinchey & Metcalf, 2009 Ingredients Needed to Solve the Integrity/Fidelity Problem • Begin with a specific, research proven, process for what to do. • Train teachers and provide ongoing support- RTI Lead, Coach, etc • Supportive but firm administration. – Expectations and assessment of integrity/fidelity and outcomes. • Support and Leadership from State DOE • Stay the course – Monitor outcomes and tweak What Do WE Know about Assessing and Improving Fidelity of RTI, Joseph C. Witt, LSU, Senior Scientist, iSTEEP Learning