This webinar is hosted by:
Capacity for Health at APIAHF, a Capacity
Building Assistance Program, with offices in San Francisco and Washington D.C.
Who we are:
Our program is funded by the Centers for
Disease Control and Prevention (CDC) to provide
free trainings and one‐on‐one technical
assistance for community‐based organizations:
• Organizational Infrastructure and Program Sustainability
• Evidence‐Based Interventions (EBIs) and Public Health Strategies
• Monitoring and Evaluation Webinar: Pre/Post Tests
Housekeeping





All participants are automatically muted by webinar administrators.
For discussion, please “raise” your hand. You will be called on and un‐muted so that you can ask your question live. If your audio is through a telephone, please make sure to enter your audio PIN, otherwise we are unable to unmute you.
You may ask questions by typing them in using the questions feature.
If anyone is watching the webinar with multiple people around one computer, please type in your contact information in the questions box so that we can include you on any follow up.
Today’s webinar will be recorded. The recording will be available on our website.
Webinar: Pre/Post Tests
Presenter‐Kristin Fernandez, MPH
Kristin has spent the majority of her career working with HIV and other health programs as a health educator, research associate and program manager. Currently, she consults with various health agencies as a grant writer and program planning and evaluation specialist. When not working, she is chasing around her 2 year old daughter.
Webinar: Pre/Post Tests
Today’s Agenda
• Presentation (50 minutes):
1. What are pre/post tests?
2. How to design pre/post tests
3. Beyond the questionnaire: alternative styles of pre/post tests
• Question and Answer (10 minutes)
Webinar: Pre/Post Tests
Quick Poll
• Have you ever used a pre/post test as part of a program evaluation?
Webinar: Pre/Post Tests
Learning Objectives
By the end of this presentation, participants
will…
 Understand what pre/post tests are, and how they are useful
 Understand the limitations of pre/post tests
 Be able to form questions for pre/post tests based on program‐specific objectives
 Be able to identify several alternatives to questionnaires
 Feel confident designing a pre/post test
 Be able to define next steps for their agency
Webinar: Pre/Post Tests
Today’s Agenda
• Presentation (50 minutes):
1. What are pre/post tests?
2. How to design pre/post tests
3. Beyond the questionnaire: alternative styles of pre/post tests
• Question and Answer (10 minutes)
Webinar: Pre/Post Tests
What are Pre‐ and Post‐ Tests?
Instruments that measure behavioral
determinants about a subject the program targets to change Webinar: Pre/Post Tests
What are Pre‐ and Post‐ Tests?
These measures are captured at 2 points in time
Pre‐test of participant group
Post‐test of participant group
Program Activities
Baseline
measures
End‐line measures
Adapted from http://designmonitoringevaluation.blogspot.com/2010/03/pre‐and‐post‐test‐project‐group‐no.html
Webinar: Pre/Post Tests
What are Pre‐ and Post‐ Tests?
Comparing participants’ results produces an estimation of the program’s effect
Pre
Webinar: Pre/Post Tests
+
Post
= difference?
Pre/Post Tests…
• Are used to evaluate program outcomes
• Are best for measuring short‐term impact
• Can be used for routine monitoring
• Are often in short questionnaire format
• Alternatives will be discussed at the end of this presentation
Webinar: Pre/Post Tests
Evaluation Framework
Step 1 Determine Purpose and Uses
Step 2 Develop Measurable Objectives
Step 3 Develop Evaluation Questions
Step 4 Collect/Gather Credible Evidence*
Step 5 Analyze Info/Develop Conclusion
Step 6 Report Findings
*Pre/Post Tests used here
Source: APLA Shared Action
Webinar: Pre/Post Tests
Step 4: Collect/Gather Credible Evidence
1) Identify types of information needed
2) Determine sources for information 3) Select methods to collect information 4) Define procedures to collect information
Webinar: Pre/Post Tests
 What questions do we want answered?  Who will give us this these answers?
 How will we gather this information?
 When and where will we collect this information? Step 4: Collect/Gather Credible Evidence
 What questions do we want answered? Is the program WORKING? How?
 Who will give us this these answers?
PARTICIPANTS
 How will we gather this PRE and POST tests
information?
 When and where will we collect this information? Webinar: Pre/Post Tests
BEFORE the program begins and AFTER it ends
Benefits of Pre/Post Tests
They’re Cheap
They’re Easy
They’re Useful
Webinar: Pre/Post Tests
Drawbacks of Pre/Post Tests
• Other factors in participants lives could also be influencing the changes seen
• Rely on self‐reported information
Webinar: Pre/Post Tests
Pre/Post tests CAN help us say:
“Based on pre‐ and post‐test results,
change happened, and we think the
program may have influenced this.”
Webinar: Pre/Post Tests
They CAN’T help us say:
“Based on pre‐ and post‐test results, the
program alone caused the change.”
Webinar: Pre/Post Tests
Questions?
• Please “raise” your hand to be un‐muted or type into the Questions feature
Webinar: Pre/Post Tests
Today’s Agenda
• Presentation (50 minutes):
1. What are pre/post tests?
2. How to design pre/post tests
3. Beyond the questionnaire: alternative styles of pre/post tests
• Question and Answer (10 minutes)
Webinar: Pre/Post Tests
Case Study
Your agency provides weekly after‐school HIV
risk‐reduction workshops to teens. These
workshops take place over 10 weeks. During
these workshops, teens learn how reduce their
risk for HIV and other STDs through activities, skill‐building exercises and verbal and visual presentations. Webinar: Pre/Post Tests
Case Study Cont’d
You will use pre‐ and post‐tests in order to quickly and cheaply gather info and assess the program’s impact. The results will be used to refine the program and will be included in an upcoming grant application.
Where do you begin?
Webinar: Pre/Post Tests
Designing Pre/Post Tests
Step 1: Look at your program’s intended objectives
Step 2: Design test questions based on objectives & indicators
Step 3: Organize questions appropriately
Step 4: Pilot & revise
Webinar: Pre/Post Tests
Designing Pre/Post Tests: Step 1
Step 1: Look at your program’s intended:
goals
objectives
indicators
Webinar: Pre/Post Tests
Quick Poll
• Do you know where your program’s goals and objectives are located?
Webinar: Pre/Post Tests
Evaluation Framework
Step 1 Determine Purpose and Uses
Step 2 Develop Measurable Objectives
Step 3 Develop Evaluation Questions
Step 4 Collect/Gather Credible Evidence*
Step 5 Analyze Info/Develop Conclusion
Step 6 Report Findings
*Pre/Post Tests used here
Go BACK and look at Step 2 and Step 3
Webinar: Pre/Post Tests
Case Study: Step 1
Objective: “By end of program year, at least 80% of participating teens will demonstrate increased knowledge of HIV risk‐reduction strategies.”
Indicators to this objective:
1. Can name the 5 body fluids that transmit HIV
2. Can list 2 strategies to increase condom use Webinar: Pre/Post Tests
Caution!
Objective: “By end of program year, at least 80% of participating teens will demonstrate increased knowledge of HIV risk‐reduction strategies.”
This does NOT mean this objective will be met at the end of each session!
Webinar: Pre/Post Tests
Designing Pre/Post Tests: Step 2
Step 2: Design test questions based on objectives & indicators • Try to design at least one question around each program objective in your pre‐ and post‐test
• Use indicators to guide wording, question type • Don’t ask questions that require participants to remember picky details
• Be sure to ask questions for which a clear answer was given over the course of the program!
Webinar: Pre/Post Tests
Common Question Styles
• Ones in which a participant marks an answer
What day is it today? (CIRCLE ONE)
Monday Tuesday Wednesday Thursday Friday
• Ones in which a participant rates a certainty of knowledge, feeling, or opinion
How well do you agree with the following statement?
“Condoms should be available in schools” (CIRCLE ONE)
Strongly Agree
Agree
Disagree
Strongly Disagree
Webinar: Pre/Post Tests
Common Question Types
• Open‐ended
Describe your day:_____________________ (WRITE IN)
• Closed‐ended
Did you use a condom today? (CIRCLE ONE): Yes No
• Partially open‐ended
I usually forget to use a condom when (CHECK ALL THAT APPLY):  I’m drunk or high  I’m not at home  Other:________ (write in)
Webinar: Pre/Post Tests
Common Answer Types
• Ordinal
• Interval
Strongly Agree, Agree, Disagree,
Strongly Disagree
0‐10, 11‐25, 25‐50, 50+
1 to 5 with 1 = LOW and 5 = HIGH
• Nominal
• Numerical
Yes/No
Male/Female/Transgender
165 lbs
39 yrs old
11th grade
Webinar: Pre/Post Tests
Case Study: Step 2
Which of the following body fluids transmit HIV? (Check all that apply):
 Blood
 Saliva (spit)
 Breast milk
 Semen (cum)
 Vaginal fluids
 Vomit (puke, barf)
 Pre‐ejaculatory fluid
 Sweat
Webinar: Pre/Post Tests
 This question addresses one of the program’s stated objectives:
By end of program year, at least 80% of participating teens will demonstrate increased knowledge of HIV risk‐reduction strategies
 It uses the objective’s specific indicator as a guide for the question
Can name the 5 body fluids that transmit HIV
Case Study: Step 2
What 2 strategies will you try in order to help you use condoms every time you have sex?
1.____________________
___________________
2.____________________
___________________
 This question addresses one of the program’s stated objectives
By end of program year, at least 80% of participating teens will demonstrate increased knowledge of HIV risk‐reduction strategies  It uses the objective’s specific indicator as a guide for the question
Can list 2 strategies to increase condom use (participants write in answers)
Webinar: Pre/Post Tests
Designing Pre/Post Tests: Step 3
Step 3: Organize questions appropriately
Logical order of questions:
1. Sensory/physical: What do you see when you walk into a clinic…? Where do you go when you need STD testing…?
2. Knowledge/feeling: How do you feel when you have unprotected sex…? What is the ‘window period’ for HIV…?
3. Opinion/values: What is your opinion of sex education in schools…? 4. Analytic/action: How long does it take to put on a condom properly…? How would you prepare to have safe sex on a date…?
Webinar: Pre/Post Tests
Case Study: Step 3
Which of the following body fluids transmit HIV?
 Blood
 Saliva
 Breast milk
 Semen
 Vaginal fluids
 Vomit
 Pre‐ejaculatory fluid
 Sweat
= Knowledge question
Webinar: Pre/Post Tests
What 2 strategies will you try in order to help you use condoms every time you have sex?
1._______________________________
_________________________________
2._______________________________
_________________________________
= Analytic/Action question
Designing Pre/Post Tests: Step 4
Step 4: Pilot & revise
• Pilot the tests with staff and participants
• Ensures the questions are well written and make sense
• Ensures correct answers are due to actual
knowledge, not flaw in test design
• Eliminate or change any unclear questions
Webinar: Pre/Post Tests
Case Study: Step 4
1. Copies of the pre/post test piloted with staff
2. First revision
3. Pre/post test piloted with participants
4. Second revision
5. Process repeated
Webinar: Pre/Post Tests
Questions?
• Please “raise” your hand to be un‐muted or type into the Questions feature
Webinar: Pre/Post Tests
Common Styles of Pre/Post Tests
•
Taken at 2 points in time
•
Taken at 1 point in time
•
“Post‐only”
•
“Retrospective pre‐test”
Webinar: Pre/Post Tests
Retrospective Pre‐Test Design
1) Participants rate their knowledge, skill, attitude or behavior AFTER completing the program, then
1) Reflect back to how they were BEFORE the program and report and rate the same knowledge, skill, attitude and behavior
Webinar: Pre/Post Tests
Case Study: Retrospective Pre‐Test
Please check the box that describes how well you knew this information BEFORE participating in this program:
“The 5 fluids that transmit HIV”
 I knew this very well
 I knew this pretty well
 I knew this a little
 I did not know this
Webinar: Pre/Post Tests
Case Study: Retrospective Pre‐Test
Please check the box that describes how well you know this information NOW/AFTER participating in this program:
“The 5 fluids that transmit HIV”
 I know this very well
 I know this pretty well
 I know this a little
 I do not know this
Webinar: Pre/Post Tests
Case Study: Retrospective Pre‐Test
Please check the box that describes how well you could do the following task BEFORE participating in this program:
“I can use strategies that will help me use a condom”
 I could do this very well
 I could do this pretty well
 I could do this a little
 I could not do this
Webinar: Pre/Post Tests
Case Study: Retrospective Pre‐Test
Please check the box that describes how well you could do the following task NOW/AFTER
participating in this program:
“I can use strategies that will help me use a condom”
 I can do this very well
 I can do this pretty well
 I can do this a little
 I cannot do this
Webinar: Pre/Post Tests
Case Study: Retrospective Pre‐Test
Please rate your level of knowledge, skill and confidence in the following areas BEFORE and AFTER this program
A LITTLE
SOME
A LOT
Confidence to say no to sex NONE
Confidence in protecting myself
A LOT
Skill in negotiating condom use
SOME
Knowledge about condoms
A LITTLE
Knowledge about HIV
AFTER
NONE
1 is LOW and 4 is HIGH
BEFORE
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
Benefits of Retrospective Pre‐Test
• Program doesn’t start with a test
• Same “frame of reference”
• Convenient
Webinar: Pre/Post Tests
Limitations of Retrospective Pre‐Test
• Doesn’t measure actual knowledge, skills, etc. (only perceptions!)
• Recall may be difficult for some
• Culture, language skills, literacy level, age
Webinar: Pre/Post Tests
Questions?
• Please “raise” your hand to be un‐muted or type into the Questions feature
Webinar: Pre/Post Tests
Today’s Agenda
• Presentation (50 minutes):
1. What are pre/post tests?
2. How to design pre/post tests
3. Beyond the questionnaire: alternative styles of pre/post tests
• Question and Answer (10 minutes)
Webinar: Pre/Post Tests
Beyond the Questionnaire
• Questionnaires are not the best format for all target populations
• Some things are better SHOWN
Webinar: Pre/Post Tests
Alternatives to Questionnaires
• Card‐sorting
• Temperature gauge
• Labels, stickers, dots
• Observation
…and more!
Webinar: Pre/Post Tests
Case Study: Card‐Sorting
• Participants are given answer options on index cards • They choose the correct answers and group them together under categories Webinar: Pre/Post Tests
“Transmits HIV”
“Doesn’t transmit HIV”
Case Study: Temperature Gauge
“I feel confident that I can • Participants put a sticker on a picture of a increase my condom use.”
thermometer or other AGREE
image to represent how they feel about something DISAGREE
After program
Before program
Webinar: Pre/Post Tests
Case Study: Temperature Gauge
• As a group, participants “I feel confident that I can put a dot next to the increase my condom use.”
thermometer to AGREE
represent how they feel
• Dots can be color‐coded for variables like age, gender, zip code, etc. DISAGREE
Webinar: Pre/Post Tests
Case Study: Labeling
Please label the following:
• Participants label a diagram or picture
1.
• Labels contain answer choices plus a few extra choices
vagina
ovary
follicle
endometrium
uterus
2.
3.
4.
cervix
fallopian tube
Webinar: Pre/Post Tests
clitoris
5.
6.
Case Study: Observation
• Participants are observed by program staff and behaviors are noted
• Great for skill demonstration!
Webinar: Pre/Post Tests
“Pair off with the person next to you. Show them how to put a condom on the condom demonstrator correctly from start to finish.”
Further Resources
Slides, a recorded version of this webinar,
and links to further resources
will be available on our website at:
www.capacity4health.org
Contact us for individualized
capacity-building assistance:
Sonya Dublin, Capacity for Health
415-568-3335, [email protected]
Webinar: Pre/Post Tests
This webinar is
part of a 6 part
series…
Webinar: Pre/Post Tests