Measurement What is the message? How Do We Know if a Change is an Improvement? “You can’t fatten a cow by weighing it” - Palestinian Proverb “If you can’t • Improvement is measure NOT about it, measurement you can’t IMPROVE it” • However… Differences between research and improvement Measurement for Research Measurement for Learning and Process Improvement Purpose To discover new knowledge Tests One large blind test To bring new knowledge into daily practice Many sequential, observable tests Biases Control for as many biases as Stabilize the biases from test to test possible Gather as much data as possible, just Gather just enough data to learn and in case complete another cycle Data Duration Can take long periods of time to obtain results Institute for Healthcare Improvement © AQuA Academy Small tests of significant changes accelerate the rate of improvement Types of Measures • Outcome Measures – Where are we ultimately trying to go? – Point to qualities that are valuable to stakeholders i.e. how is the system performing? What is the result? • Process Measures – Are we doing the right things to get there? – Track that steps in the system are performing as planned. – Help identify if changes are leading to improvement. • Balancing Measures – …………………………………………… © AQuA Academy Balancing Unintended Consequences “First, do no harm!” Balancing Measures help us ‘keep an eye on’ other aspects of the system as we focus on improving one part. © AQuA Academy How am I going to present my data? • What will you be reporting? • Who is your target audience? • How often will you need to communicate data? • What is the purpose of your analysis? © AQuA Academy 6 Number of Complaints Per Year 400 350 300 250 200 150 100 50 0 2010 2011 What conclusions can we draw from this? Snapshot in time 70 60 50 40 30 20 10 0 Jan-10 May 2010 Jan-11 May 2011 Displaying Data Run Charts • Plot data in a timely order • Can analyse by studying where values lie and why they are distributed around the median • Helps differentiate between Random and Special Cause Variation • Illustrates the impact of an improvement intervention © AQuA Academy 10 Number of Complaints per Quarter 140 No of complaints 120 100 80 60 40 20 0 Q1 Q2 Q3 2009 2010 Q4 Q1 Q2 Q3 2010 2011 Q4 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 70 Number of Complaints per Month 60 50 40 30 20 10 0 2010 2009 2011 2010 Audit: Year on year comparison 70 60 2010 2011 50 40 30 20 10 0 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Knowing What to Look For: Rules • Runs: One or more data points which lie on the same side of the line. Each time median is crossed, a new line is identified. • Trends: 5 or more points all going up or all going down. • Astronomical Data Points: Outlier that is significantly higher or lower than median. • Shift: 6 consecutive data points either above or below the line. Number of Complaints per Month 70 60 50 Astronomical Data Points Runs 40 Median 30 20 10 Shift Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 0 2009 2010 2010 2011 Does one way of presenting data work for everyone? Tips for Effective Measurement • Plot data over time • Seek usefulness, not perfection • Use sampling • Integrate measurement into the daily routine (and is someone already collecting the data?) • Use qualitative and quantitative data • Accept Variation © AQuA Academy Benchmarking • • • • • • • • Continuous search for better practices Understanding practices behind performance gaps Looking outside the organisation Provides realistic & achievable targets Challenges complacency Supports continuous improvement Visualise the improvement Identify weak areas & what needs to be done to improve Any Questions?