Download Report

Evaluation Toolkit
contents
Evaluation Toolkit
CONNECTED LEARNING : CONNECTED COMMUNITIES
EVALUATION FRAMEWORK
3Introduction
4 Logic model: CLCC
5 Evaluation questions
6 Data collection and management
8 Data analysis
8 Reporting and disseminating findings
RESOURCES
9 Logic model template
10 Evaluation measures and data collection template
11 Evaluation data return form
12 Evaluation protocol: family field trips
13 Survey development
13 Key concepts in survey development
13 Survey construction
14 Survey sampling and administration
15 Sample survey: educators, post-only
Focus groups
19 Steps for development and implementation
21 Focus group questions for parents
22 Focus group questions for partners 23 Focus group questions for educators
25 Focus group facilitator protocol
26 Focus group recorder protocol
27 Focus group tally sheet
28 Focus group sign in sheet
©2013 The Children’s Museum of Atlanta, Inc. All rights reserved.
Evaluation Framework for a Museum
Community Outreach Program
Based on Children’s Museum of Atlanta, Connected Learning: Connected Communities
Evaluation is a process that starts with the beginning of a program concept. Evaluation shouldn’t be a process you do at the
end; evaluation should be part of your on-going program of work. Participatory evaluation actively involves stakeholders
at various steps along the way – program planners, funders, evaluators, participants, and partners. Using a participatory
process helps build capacity for all stakeholders. Having the end in mind (what you are trying to achieve) should drive your
evaluation process. Start with goals and objectives you want to achieve.
A logic model is an easy way to organize your program and visualize the flow of activities. There is no one way to do a
logic model, they can be as simple or as complex as you want them to be. A logic model describes your theory of change,
what you want to measure and the outcomes you hope to achieve.
There are many examples you can follow. Use one that is easy for you to understand and is easy to share with staff and
partners. You can always modify your logic model and add complexity. A logic model can be a general overview of your
program or you can have a multi-page logic model with very specific details. Design one that fits your needs.
PAGE 3
FIGURE 1: LOGIC MODEL FOR CONNECTED LEARNING…CONNECTED COMMUNITIES EXAMPLE
Context
Inputs
Activities
Intermediate Outcomes
Long-term Outcomes
COMMUNITY
•CMA building with family-
friendly exhibits designed to spark discovery and creative learning
• CMA educational program and staff
• AECF financial investment in
Atlanta Civic Site and
neighborhoods
• Neighborhood residents engaged in creating positive change
• Neighborhood programs providing family support and skill building (parenting classes, GED classes)
• Early learning centers and schools involved in quality improvement initiatives
• Leveraging of CMA funds with investment from Blank Foundation
•Create Neighborhood Alliance for Learning with involvement of key
neighborhood leaders
•Create CLCC Advisory Committee for oversight
and guidance
•Neighborhood-based
programming and events
•Target Tuesdays, second Tuesdays free admission
for families
•Imaginators extend resources of museum with regular visits to early learning centers and schools
•Imaginators extend resources of museum with regular visits
to on-going parenting classes and to GED classes
•Museum-in-a-Box provided to early learning centers and schools with thematic links to special exhibits at museum
•Field trips to museum for
neighborhood families
•Child care training for early learning providers
•Neighborhood art events
•Artist in residence
•Increased understanding by families and early learning providers of the power of play in learning
•Increased awareness of the
museum as an educational
resource by families, early learning providers, and schools
•Improved access to activities, materials, and supplies that enhance children’s learning
•Increased neighborhood and civic engagement
•Increased engagement of neighborhood adults in the education of their neighbor
hood’s children.
•Museum repositioned not just as a facility for learning but as a valuable community resource.
Historic urban neighborhood
in decline; community associations focused on engagement
and change; high vacancy
rates and blight; significant
drug and criminal activity;
limited businesses and support
services for residents
RESIDENTS
Primarily African-American;
high poverty rates; low
educational attainment; high
unemployment rate
PAGE 4
Evaluation Questions and Indicators
Evaluation questions provide a framework for the evaluation. There are two types of questions that can help guide your
evaluation: process or implementation questions and results or impact questions. The word formative is often associated
with process questions and summative is often used in place of implementation or impact.
Developing and answering questions helps you determine what was actually done, to whom and by whom,
and what was achieved.
Implementation questions address: Did the project do what it said it would do?
Impact questions address: What were the outcomes for participants?
IMPLEMENTATION QUESTIONS
Sample Implementation Questions
Sample Indicators
How many children and families
were reached?
Number and description of children and families reached
(age, gender, race/ethnicity, grade, socioeconomic status)
How frequently were children and
families involved?
Contact frequency / frequency of involvement
(e.g. once a month, every week for six weeks)
What was the duration of their
involvement?
Contact duration (e.g. 45 minutes once a week; two hours
weekly for six weeks)
Who was involved in providing the
programs or activities?
Number and type of providers
(e.g. 6 art instructors, 10 hours per week each) ;
Number and type of partners
(e.g. GED classes led by local college)
What kind of educational events
were provided?
Number of hours of programming provided;
number and type of workshops provided
How many and what kind of schools
and child care centers were reached?
Number of schools/centers reached,
characteristics of those schools/centers
(e.g. Title I, grades or ages served at school or center)
What were key challenges in
implementing the program?
Steps taken to overcome challenges
PAGE 5
IMPACT QUESTIONS
Sample Impact Questions
Sample Indicators
What did participants think of
programming provided?
Satisfaction with performers
What did participants learn from
programming provided?
Shifts in attitudes and knowledge pre and post-visits to the
Museum or pre and post-participation in workshops
Did the programming provided align
with curriculum standards?
Satisfaction with curriculum alignment
Did perceptions toward the Museum
change?
Perception of the Museum by parents, by teachers
Did perception of play as learning
change?
Perception of play as learning by parents, by teachers
Did educators think curriculum
materials were helpful?
Satisfaction with curriculum materials
Did children learn new concepts?
Educator perception of children learning new concepts as a
result of curriculum materials or visits by Museum staff
Data Collection and Management
A detailed data collection plan, with roles, instruments, and dates clearly defined, helps to ensure on-time, accurate,
complete collection. A data collection plan should be established at the beginning of the program and monitored for
compliance and completeness. Have clear roles of responsibility established. Your evaluator should provide data
collection protocols for the program and provide surveys or instruments for data collection.
Counts or process data should be the most basic information collected. Counts can be as simple as doing a head count
or having people sign in on a form. Having more specific information about people enriches your reporting and findings.
What kind of people attended?
Outcome data helps you determine the ‘so what’ of your program. Did participants learn something new? Learn a new
skill? Change their attitude or behavior? Have new aspirations? Outcome data involves measuring changes in attitudes,
knowledge, skills or aspirations. Surveys, focus groups, knowledge assessments, demonstration of skills acquired, and personal
interviews are some of the methods for collecting outcome data. You can use surveys developed by an evaluator or assessments
that are developed by the provider of a curriculum you may be using. It takes time and skill to develop outcome measures;
you need to reliably measure results so you can determine if your program had an impact.
PAGE 6
Program records, such as donated funds, volunteer time or staff development, help you develop a complete picture of
program outcomes. What kind of funding or staff development is necessary to achieve the desired outcomes?
Data management includes collecting, entering, analyzing, and storing data. Who does what at each step should be
established between the program and the evaluator. Written evaluation protocols are a good idea if there are several
different people involved in data collection. If you are collecting data from children, make sure you are following any FERPA
and HIPAA regulations or any local regulations set forth by a school system or service provider. The evaluator will generally
be responsible for data entry but you can save evaluation costs by having staff enter data from surveys or assessments.
Work with your evaluator to make sure data entry formats are appropriate and the right coding formats have been used.
Evaluation Measures and Data Collection Schedule: Example
Measure
Periodicity
Administration / Instrument/
How Collected
Person Responsible
for Data Collection
Outreach performances at
school and child care centers
Pre-event and
at event
Excel spreadsheet with detailed
information on location including number
of children, age range of children,
date of performance, name of performer
Outreach coordinator
Teacher satisfaction at outreach
performances
At event
Satisfaction survey
Outreach performer
Teacher satisfaction with Museum
in a Box
With each
box distributed
Satisfaction survey
Outreach coordinator
Teacher assessment of the program
Annually
Focus Group
Evaluator
Family attendance at program events
At event
Sign-in sheet on bus and at Museum
Outreach coordinator
Family satisfaction pre- and post-visit
At visit to Museum
Pre-visit satisfaction survey; post-visit
satisfaction survey (match surveys)
Outreach coordinator
PAGE 7
Evaluation Measures and Data Collection Schedule: continued
Measure
Periodicity
Administration / Instrument/
How Collected
Person Responsible for
Data Collection
Family assessment of program
Annually
Focus Group
Evaluator
Educator knowledge assessment
from child care training classes
At class
Knowledge assessment
Quality Care for Children instructor
Educator satisfaction with child care
training classes
At class
Satisfaction survey
Quality Care for Children instructor
Artist-in-Residence assessment
of program
At end of
program
Satisfaction survey
Outreach coordinator
Scientist-in-Residence
assessment of program
At end of
program
Satisfaction survey
Outreach coordinator
Staff training/development
End of year
Hours, topics per staff member
Project Director maintains records;
Evaluator reviews
Budget/financial management
Quarterly/annually
Records review
Project Director maintains records
Partner engagement
Monthly meetings
Records review of
meeting attendance
Project Director maintains records,
Evaluator reviews
Partner assessment of program
Annually
Focus Group
Evaluator
Data Analysis
Data analysis can be as simple as basic descriptive analysis (counts, means) or use more sophisticated analyses such as
tests of significance or causation. The design of your program, the data collection process, the indicators and the data
collected will determine what type of analysis can be done for your program. Excel, SPSS, and SAS are commonly used
to analyze data.
Reporting and Disseminating Findings
Determine how frequently you will report findings. Interim findings, reported mid-year or mid-project, can help you determine
if the program is on-track and if any adjustments are needed. Develop different versions of your report for different audiences.
A funder will often have specific guidelines for reporting of findings. Develop succinct handouts for board of directors or
advisory boards. Remember to report findings back to all stakeholders including teachers, parents, and community members.
Use best practices in data visualization for reporting of results – a finding presented in an informative bar chart is often
more useful than pages of text. Use multiple methods of distributing findings. Put results on your web site, prepare a handout
for board members, write an article for a newsletter, put a link to your report on Facebook, Tweet an important finding –
make sure the results of your efforts are shared with as many stakeholders as possible.
PAGE 8
LOGIC MODEL TEMPLATE
Context
Inputs
Outputs or Activities
Short-Term and/or
Intermediate Outcomes
Long-Term Outcomes
What is the situation?
What will you invest?
What resources will be
needed to accomplish
the program?
What are you going to do?
Who are you going to do it
with?
Outcomes can be divided into several
columns (short-term and intermediate)
or you can combine in one column.
Some logic models have a column for
process (counts) data.
What conditions will change
as a result of the program?
What data do you have to
support the need for the program?
Describe the community.
Staff
Describe the target audience.
Volunteers
What assets are in place?
Partner assets
Funding
Materials, supplies
What are the specific activities
that will take place in order to
achieve the outcomes?
You can divide the outputs/
activities by program activity
or by targeted age/characteristics of the target audience.
For example, children age 0-5,
students in grades 6-8, parents,
first time visitors to the museum.
Space
Equipment
Technology
PAGE 9
Short-term impacts can include process data – how many participated?
What are the characteristics of those
who participated? Did you change
awareness? Increase knowledge?
Increase skills? Improve attitudes?
Intermediate outcomes are more
complex and take longer to measure.
Did you change decision-making?
Change behavior?
What is the ultimate impact?
Evaluation Measures and Data Collection Schedule
Measure
Periodicity
Administration / Instrument/
How Collected
Person Responsible for
Data Collection
List each evaluation measure:
How often are you going
to collect the information?
What instrument are you using?
Who is responsible for
making sure the data are
collected? This is the person
who hands out the survey,
makes sure the sign in sheet
is filled in, etc.
process measures
outcome measures
budget or staffing measures
Sign-in sheets?
At an event?
At end of program?
Once a year?
Surveys?
Who is developing the survey?
Are you distributing them
at a program?
On-line surveys?
Focus groups?
Observation?
PAGE 10
Evaluation Data Return Form
Data Collected
Number of Surveys Returned
How Delivered
Pre-test parent satisfaction survey
mailed to Evaluator
hand-delivered to Evaluator
hand-delivered to Project Director
other (explain
)
Post-test parent satisfaction survey
mailed to Evaluator
hand-delivered to Evaluator
hand-delivered to Project Director
other (explain
)
Satisfaction survey, outreach visits
mailed to Evaluator
hand-delivered to Evaluator
hand-delivered to Project Director
other (explain
)
Satisfaction survey, Museum in the Box
mailed to Evaluator
hand-delivered to Evaluator
hand-delivered to Project Director
other (explain
)
Knowledge assessment, child care
educator training
mailed to Evaluator
hand-delivered to Evaluator
hand-delivered to Project Director
other (explain
)
Sign-in rosters for Family or Community
Events (after each event)
mailed to Evaluator
hand-delivered to Evaluator
hand-delivered to Project Director
other (explain
)
Date Mailed or Delivered:
Date Received:
Signature:Signature:
PAGE 11
Evaluation Protocol: Family Field Trips to the Museum
1. Please take a sign-in sheet to each event and have it completed on the bus. Have another sign-in form at the Museum
for parents who drive themselves to the Museum.
2. To each event take sign-in sheet, surveys, pencils, a large manila envelope, and a survey return form.
3. Take more than enough copies of surveys you think you will need for the field trip. Ask parents to complete the pre-visit survey on the way to the Museum.
4. To prepare parents for survey:
a. Tell them you are going to give them a survey that will ask some questions about how they feel. Please tell them that there is no right or wrong answer AND that the surveys/answers will not be seen by the Outreach Coordinator.
b. Please explain that the purpose of the survey is to help us improve the program. We welcome their suggestions
and feedback.
c. Make sure each parent has a pencil.
d. Ask the parents to please print their names – that we would like their opinions before they go into the museum and
after they visit the museum. Tell them they will receive another survey when they board the bus after the conclusion of the field trip. Ask them if they have any questions. If not, ask them to fill out survey and then place it in the envelope before getting off the bus. (if you have parents who drove to the Museum, ask them to complete a survey and put it in the envelope).
5.After the visit, ask parents to complete the post-visit survey. Pass the manila envelope around and ask parents to put their survey in the envelope.
6.Fill out survey return form indicating whether you are mailing the completed surveys or hand delivering the surveys to the evaluator. Sign and date the form. Then hand-deliver or mail the surveys. The evaluator will send a confirmation email to you when she receives the surveys.
7. If you have any questions, please call the evaluator at 999/888-7777 or 333-1111 (cell)
or email at [email protected]
8.This is not optional, please make surveys are administered at each event. Thank you!
PAGE 12
Surveys
Survey Development Key Concepts
. Surveys are a relatively inexpensive way to gather information from a large group of people.
. Surveys can be anonymous and thus can provide information that people are unlikely to share
(e.g. do your parenting skills need improving?)
. Matched pre/post surveys help you measure changes in attitudes, behavior, or knowledge.
. It is easier to analyze survey responses than data from large number of focus groups or interviews.
Survey Construction
Wording
. Sequence of wording needs to be clear and concise
. Avoid jargon and do not assume everyone knows the jargon
. Be mindful of literacy levels
. Have others check wording
. Make sure each item/question is only asking one concept or thing
. Include open-ended questions. Ask what is the best thing? what needs improving?
What additional comments do they have?
Introduction to survey
. Tell respondents why you are doing the survey
. Make sure to note it is confidential
. Tell respondents where or when results will be available
. The introduction can be done as a cover page or as paragraph
Language versions
. You may need to produce versions in various languages depending on the audience
Length
. Not too long, not too short
. Consider the environment in which the survey will be completed when determining how many question to ask
. Do a pilot and see how long it takes to complete
. If you have multiple pages, put visual clues so respondents know to turn the page over
Types of questions
. Multiple choice
. Yes/no are not very informative when tabulating data
. Ranking
. Scaled (Likert scale, five point works well)
PAGE 13
Layout
. Needs to be easy to read
. Font should be easy to read (Arial, Tahoma, Calibri)
. Have plenty of white space
. Shaded lines help visually separate questions
. Use circles to bubble in or squares to check
Survey Sampling and Administration
How many is a representative sample? Statistical charts are available on the web to help you
determine an appropriate sample size.
Types of surveying: paper survey, telephone, on-line
. Possible types of administration include paper survey, Scantron-type surveys, telephone, and on-line
. Internet surveys are low-cost but are limited to people who have access to the web and have a comfort level with using computers
. Need to have email addresses if using a service like Survey Monkey. Not everyone has an email address so if you are surveying families, you may not get a representative sample.
. Can do both methods to increase sample size. Convert paper survey to Survey Monkey or similar service. Use same survey. You may need the more advanced version of Survey Monkey or Zoomerang (about $100) so you can tabulate and download results.
PAGE 14
2012-2013
survey
IMAGINATOR VISIT
Please tell us your title or position in the early learning,
pre-kindergarten or kindergarten program:
V Director of center/program
V Teacher
V Paraprofessional
V Other
Excellent
Good
Fair
Adequate
Poor
1 The Imaginator’s performance
V
V
V
V
V
2 The Imaginator’s rapport with the children
V
V
V
V
V
3 The Imaginator’s classroom management skills
V
V
V
V
V
4 The Imaginator’s time management skills
V
V
V
V
V
Strongly
Agree
Agree
Not Sure
Disagree
Strongly
Disagree
5 The activities were age appropriate.
V
V
V
V
V
6 The content of the program kept the interest
of the children.
V
V
V
V
V
7 The content of the program related to Georgia V
V
V
V
V
V
V
V
V
V
Please rate the following aspects of
the Imaginator visit:
Please answer the following questions about
today’s Imaginator visit.
Early Learning Standards or the Georgia Pre-K standards.
8 The program is useful in helping reinforce what I am teaching the children.
PLEASE TURN OVER FOR MORE QUESTIONS
PAGE 15
2012-2013
survey
IMAGINATOR VISIT
Strongly
Agree
Agree
Not Sure
Disagree
Strongly
Disagree
V
V
V
V
V
10 The children learned something new today.
V
V
V
V
V
11 There was enough variety in the activities.
V
V
V
V
V
12 The amount of time for each activity
V
V
V
V
V
V
V
V
V
V
V
V
V
V
V
V
V
V
V
V
Please rate the following aspects of
the Imaginator visit:
9 The children enjoyed participating in
the activities.
was appropriate.
13 The activities gave me ideas I can later use
with the children.
14 The program helps positively impact
student achievement.
15 As a result of the visit and the activities,
I am more interested in using museum
educational resources.
16 What needs to be improved or changed about the program?
17 What is the BEST thing about the Imaginator Program?
18 What additional comments do you have?
THANK YOU!
YOUR SURVEYS ARE CRITICAL FOR US SO WE CAN IMPROVE OUR PROGRAMS AND FOR REPORTING
TO OUR FUNDERS. WE MUST HAVE THIS DATA TO BE ABLE TO CONTINUE TO OFFER THE IMAGINATOR
PROGRAM. THANK YOU FOR COMPLETING AND RETURNING SURVEYS.
PAGE 16
Focus Groups
The feelings, impressions, and opinions of residents, stakeholders, public officials, youth, and others provide answers and
information that cannot be gathered from other sources. Much of this information is undocumented but provides a valuable
perspective about a community.
Information gathering strategies include:
.
.
.
.
Focus Groups
Stakeholder Interviews
Surveys
Record Reviews
Focus group data collection helps us answer the following questions:
What are the positive and negative characteristics that most accurately define the community?
What are the strengths that bond families together in the community?
What are the most pressing challenges and concerns confronting residents in the community?
What services, activities, and resources are needed to address these issues?
What barriers prevent residents from obtaining the services they need to address their challenges and concerns?
What strengths and resources to support families are currently in the community?
How can these strengths and resources be fortified, supported, replicated, or expanded?
What community services and programs do residents use most?
What types of contributions can residents make in addressing the programs that most affect and impact the community?
PAGE 17
Information-Gathering Strategies
Strategy
Definition
Advantages
Focus Group
Series of small meetings to solicit opinions,
anecdotes, experiences, and impressions
from small groups of individuals
. Is relatively easy to arrange
. Can be more efficient than other community
assessment methods
. Can be more in-depth and focused on
particular topics
. Engage groups of individuals that may be
otherwise inaccessible
. Collects depth of opinion, can get closer to
what people are really thinking and feeling
. Can yield a lot of information in short period
of time
. Builds community identity by initiating
discussions in particular neighborhoods
Key Informant
Interview
Interview with an individual with an important
viewpoint or role in the community (e.g. public
official, administrator, elected official, health care
provider, neighborhood leader)
. Allows for in-depth, one on one, exploration
of topics
. Requires minimal expenditures of resources
. Gives focus to specific issues related to the
community assessment
. Allows for immediate clarification of questions and answers
. Helps establish relationships
. Provides opportunity to discover previously
unknown issues, concerns or assets
Survey
Formal, systematic survey of defined populations
in specific geographic areas or survey of specific
age groups to gather information on health, social
well-being, patterns of service utilization, and
opinions about assets and needs of community
. Provides respondents anonymity
. Provides up-to-date data
. Can provide data on individuals with unmet needs and barriers preventing their access
to services
. Allows collection of common data elements across diverse audiences
. Comparatively easy to collect data on a wide range of topics
Adapted from “Know Your Community, Family Resource Coalition of America, 1995
PAGE 18
Steps for Development and Implementation of Focus Groups
DEFINITION
of a focus
group
A focus group is a formal group meeting guided by a trained leader with a specified group of individuals
designed to solicit opinions, anecdotes, experiences and impressions on designated topic(s).
1. Determine what specific data you want to collect. For example:
. Perceptions of quality of service or program
. Perceptions of community strengths and needs
. Experiences with services or program
2. Determine why you need the focus group data. For example:
. To guide survey development
. To explore and explain survey findings
. To gather information from small groups otherwise inaccessible
. To obtain detailed data regarding experiences in programs
. To obtain detailed data regarding specific life experiences
. To observe or better understand specific group dynamics
3. Identify the groups from whom you want to collect data.
. Parents
. Community partners
. Students
. Program providers
4. Recruit focus group participants.
Use multiple methods to recruit focus group participants
. Work with partners like schools to recruit participants. If recruiting parents, a school system parent involvement
coordinator or Title I coordinator will be very helpful
. Incentives may be appropriate for some groups. Gift cards are helpful when recruiting low income families.
. Try to have a diverse group of participants – gender, race/ethnicity.
. Use flyers, emails, text messaging and personal contact. Personal or targeted recruitment is better than
general recruitment.
. Be clear in your recruitment what the purpose of the group is, if there will be food or child care provided.
. Pay attention to transportation details (e.g. offer bus tokens, bus transportation, parking vouchers).
5. Develop focus group protocol and questions.
. Have your questions written down.
. Use questions that are clear and not offensive. Don’t use yes/no questions.
. Have a reasonable number of questions; have probe or follow-up questions in case you need them.
PAGE 19
. If you are taping the conversation, be sure and inform participants.
. It’s always great to provide food for participants; if food is provided, build in time for people to eat.
. If you are doing a focus group with parents, provide child care or child focused activities in an adjacent space.
6. Develop methods for storing and analyzing data collected.
. Have written notes and tape record the conversation.
. Don’t associate a person’s name with comments in a transcript.
. Have a back-up plan in case computer batteries fail, the recorder goes out, etc.
7. Conduct the focus group.
. It’s best to have two people – one to facilitate and one to record.
. Seek the participation of all in the group; watch out for dominant personalities.
. A good focus group size is 12 to 20 people. If group is larger, do two focus groups.
. Use a sign-in sheet, especially if you are distributing incentives or gift cards.
. Make sure your introduction encourages open dialogue and informs the group as to
how the focus group findings will be used.
. End on time!
8. Prepare summary of findings.
. Analyze conversation for common themes and topics.
. Don’t associate a person’s name with comments in a transcript.
. Do a word cloud (www.wordle.net) as a fun way to look at common themes.
PAGE 20
?
Example of Focus Group Questions for Use with Parents
1. Are you aware of the Museum and the Imaginator program?
2. How many times have you been to the Museum?
3. How many of you have ridden the Imagine It bus to the Museum?
4. If you have not, why haven’t you? (probe for time of day, day of week, publicity)
5. What do your children tell you about the program?
6. How have the Imaginator visits made a difference in your children? Can you list some concrete examples?
7. What value does the Imaginator program bring to your child?
8. In what ways could the Imaginator program help increase parent/child bonding and communication?
9. Are there ways the Museum could more effectively reach children and parents?
10. If the Museum could provide parent workshops for you – what topics would interest you? When should they be
offered? (time of day, time of year) How long should the workshops be? Would you come to the Museum for them
or would you prefer them in the community?
11. If you were trained by the Museum would you be willing to lead workshops and train other parents?
12. What do you think of take home activity packs for you and your children? What would be helpful to have in
an activity pack?
13. What incentives can the Museum use to increase parent participation in museum programs?
14. What are other ways that the Museum could help foster parent participation in their children’s learning?
15. Are there community activities that the Museum should be involved in to help publicize the programs of
the Museum or to bring Museum programming to the community?
16. Are there school activities that the Museum should be involved in?
17. How can the Museum increase community, school, and parent awareness of the programs that it offers?
18. What kind of Museum resources should be available to you at schools or child care centers?
19. What does a long term presence by the Museum look like in your community? What kind of programs, activities,
or events would that be?
20. What other input, advice, or comments do you have for the Museum staff?
PAGE 21
?
Examples of Focus Group Questions for Use with Partners
What are some of the positive changes you have observed in the neighborhood over the last xxx years?
. What impact have partner organizations had in encouraging, fostering, or creating those changes?
. Can any of those changes be attributed to the work of the Museum in the community and schools?
What strengths do you see in the neighborhood now as compared to xxx years ago? In schools?
. Parent groups: Are they stronger than before? More involved?
. Neighborhood leaders: Stronger? More involved?
. School leadership: have the outside resources made a difference in school climate, willingness to partner?
Impact on student achievement?
What impact do you think the Museum has had in the neighborhood?
. with parents?
. with children?
. with school teachers and leaders?
What challenges do you think still exist in the neighborhood?
What are your short range (one to two years) plans for your own work in the neighborhood? Do you have longer range plans
(three to five years)?
How can the Museum partner with you in the short term (one to two years) and longer term (three to five years) to keep a
presence in the neighborhood?
. Are there community activities that the Museum could be involved in to help keep a presence in the neighborhood?
. Are there school activities that the Museum could be involved in to keep a presence in the schools?
What are some lessons from your own work in the neighborhood that the Museum should take into account as we move
into a new neighborhood?
. If you were expanding your own work into a new neighborhood what would you do differently from your
current approach?
When we talk about building capacity in neighborhoods – what do you think that looks like for the Museum in terms of the
current neighborhood?
. What specific skills, programs, or activities would demonstrate that the Museum has built local capacity?
What has been the BEST aspect of your relationship with the Museum?
What do you think could be improved about how the Museum works with partners? With schools? With parents?
What other input, advice, or comments do you have for the Museum staff?
PAGE 22
?
Examples of Focus Group Questions for Use with Educators
(school system teachers and child care providers)
How has the outreach program made a difference in your students? Can you share some specific examples?
What have you liked most about the outreach program?
Think about your classroom and students before the outreach program began visiting your school.
. Do you see a difference in your students from before the outreach program to the students you have now who have
participated in the outreach program?
. Do you have students who have participated in the outreach program for the past five, four, or three years?
How has it made a difference for them?
What value does the outreach program add to your classroom?
What value does the outreach program add to your students?
Are there specific attributes of the outreach program that are of special value to you as a teacher?
What are some ways you and your students use the Museum in a Box?
What value has the Museum in a Box been to you as teacher?
How could the Museum in a Box be improved?
The Museum has been working with your schools for several years and now must focus their efforts with another set of schools.
. What are some ways that the Museum can stay connected with you – without having the outreach program come to the school every month during the school year?
. What are some ways that you and your students can stay connected to the Museum?
How can you continue to infuse the resources of the Museum (exhibits, field trips) into your classroom?
What are ways you can continue to use the materials you have received from the Museum
(the curriculum materials in the Museum in a Box)?
What community activities could the Museum be involved in to help keep a presence in the neighborhood?
. What would be the best way for the Museum to stay informed about community activities and opportunities?
What school activities could the Museum be involved in to help keep a presence in the schools?
. What would be the best way for the Museum to stay informed about community activities and opportunities?
How can the Museum increase community, school, and parent awareness of the programs and resources offered by the Museum?
PAGE 23
What do you think it means for the Museum to build capacity in neighborhoods?
.What specific skills, programs or activities would demonstrate that the Museum has built local capacity?
What has been the BEST aspect of your relationship with the Museum?
What do you think could be improved about how the Museum works with schools? With parents?
What other input, advice, or comments do you have for the Museum staff?
PAGE 24
Focus Group Facilitator Protocols
Set-up:
1. When you arrive, arrange seating in a circle or semi-circle so everyone can see each other.
2. Set up snacks and water if available.
3. Set up a stand and flip chart for writing down ground rules and it may be necessary to capture specific information or to post important questions for participants to consider that they might want to see visually.
4. Make sure there are no “observers” or outsiders in the group—it is for participants only.
5. Set out sign in sheet and ask people to sign in when they arrive
6. Set up voice recorder if you plan to use one and make sure your focus group Recorder person has the appropriate set of questions pulled up on his/her laptop.
7. Set up a watch or timer next to you so you can keep time.
8. You can stand or sit as you facilitate, just be comfortable. Make sure everyone can hear you. Keep looking around
the group, making eye contact often, keep an eye out for someone who wants to say something.
Welcome:
1. After people have arrived and signed in, welcome them and introduce yourself and your Recorder.
2. Thank them for coming and tell them that you will finish on time.
3. Briefly explain the planning process. We are developing a strategic plan for a community outreach program to expand museum services to the larger community. We need their help identifying both barriers to success and
strategies and solutions.
4. Explain that a focus group is a method of collecting data. It allows us to better understand quantitative data –
it illustrates the story behind the numbers/data. Quantitative data is important for understanding trends or finding
differences among groups of people but their opinions and thoughts, as shared in a focus group, help set priorities and help us better understand what programs and activities are effective or not effective.
5. If you are recording the conversation, tell them that although it is being recorded, that is just for transcription
purposes and it will be destroyed after one week. Ask if anyone has any objections to the recording of the
conversation. Emphasize that all of their comments are anonymous—the Recorder doesn’t attribute comments to specific people. The sign in sheet is used for us to keep a tally of how many people participated and what different organizations and agencies are represented. The names of participants will not be published.
6. Ask them to establish simple ground rules for the group—for example, raise your hand if you want to speak, one voice at a time, please do not interrupt other speakers, etc. Depending on the group, you can write the ground rules on flip chart paper. This may be helpful when doing a group with children or young adults.
7. Remind them that all opinions are valid – urge them to express their opinion even if it contradicts another person’s.
8. Ask if there are any questions before you begin and then Recorder will start the voice recorder if one is being used.
During Focus Group:
1. Have the written set of question with you. As the conversation progresses, you may need to add questions or ask a follow-up probe but try to stay to the questions drafted. If you need to change the order of questions due to flow of conversation, that is fine.
2. Adjust your language “up or down” for your group—if it is a group of students, more informal language should
be used.
3. Humor always helps put a group at ease.
4. Stay neutral. Try not to give feedback or your own opinions of what they are saying, such as “Great idea!”
PAGE 25
5. Watch the time so you don’t spend too long on one question.
6. Make sure that by the end everyone has gotten a chance to speak at least once.
Closing Out Focus Group:
1. Last question: “Is there something you want to share that I haven’t asked you about?”
2. Make sure you finish on time.
3. Thank them for their time and tell them that their input was extremely valuable to the museum process. Explain that a summary describing findings from the focus groups will be part of the strategic plan (or evaluation findings, depending on your purpose for focus group).
Focus Group Recorder Protocols
Set-up:
1. Set up voice recorder if you plan to use one and open up the appropriate set of questions on your laptop.
2. Set up a watch or timer next to you so you can help the facilitator keep time.
Welcome:
1. As the facilitator is explaining the process and establishing ground rules, complete the demographic questions at the top of your Focus Group questions (# of males/females, race/ethnic background).
2. After the facilitator asks if there are any questions before they begin, turn on the voice recorder if using one.
During Focus Group:
1. Open document with focus group questions; if possible, try to type responses under the question. If different
questions or follow-up probes are asked, try to capture as much of the question (as well as the answer) as possible. Add it wherever you can.
2. Type using “shorthand” (don’t worry about spelling, grammar) so you can capture as much information as possible and if something is not clear, ask that it be repeated.
3. Comments are anonymous and should not be attributed to any particular person.
4. Make sure that your computer is SAVING the document periodically.
Closing Out Focus Group:
1. Turn off the voice recorder and SAVE the transcript.
Follow-up:
1. Review your transcript and make edits within a couple of days of the Focus Group so you don’t forget key information.
2. Email your edited transcript to the Facilitator for their review.
3. If you used a tape recorder, please delete the taped conversation after you and the Facilitator are finished
with transcripts.
PAGE 26
Focus Group Tally Sheet
Date:
Location:
Please record attendance in
addition to sign in sheet
No. in
Attendance
White female
Black female
Hispanic female
Asian female
Other female
Total Female
No. in
Attendance
White male
Black male
Hispanic male
Asian male
Other male
Total Male
Name of Facilitator:
Name of Recorder:
PAGE 27
SIGN IN SHEET: Focus Group, Month, DAY, Year
Please print first and last name
Position/Title
Signature
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
PAGE 28
THANK
YOU
The Children’s Museum of Atlanta would like to thank the Institute for Museum and Library Services for support
of this program and the National Tool Kit and the Arthur M. Blank Family Foundation for their support of this
program from the beginning and their deep commitment to early childhood education.
CONTACT US
Karen M. Kelly, Director of Exhibits & Education
[email protected]
Tawana Francisco, Manager of School and Outreach Programs
[email protected]
Julie Sharpe, CEO, Sharpe Solutions LLC
[email protected]
PAGE 29