1. No category

Evaluation of the Sanford Inspire Program
Teacher Preparation Analytics LLC
March 2015
This is the eighth in a series of evaluation reports with notable findings and project accomplishments for the Sanford Inspire
Program. Independent evaluation of the Sanford program is made possible by cooperation from Sanford Inspire Program leaders
and team members, from teacher candidates, school and district administrators, and because of the leadership and faculty of the
Mary Lou Fulton Teachers College (MLF TC) at Arizona State University.
This evaluation report analyzes and reports on these components of the Sanford Inspire Program: the work to develop,
test, and disseminate on-demand modules for teacher candidates and in-service teachers; Sanford’s role in working
with college faculty and school partners to pilot surveys of K-12 students; its marketing and dissemination strategies;
and reports from K-12 partners about the impact of the Sanford Inspire Program on their work. The evaluation
evidence behind this report comes from data collected by Sanford and shared with the external evaluator for
independent analysis, in-depth review of six Sanford on-demand modules, and from interviews with Sanford team
members, ASU site coordinators, local school leaders, and the CEO of a partner organization.
On-Demand Modules for Candidates and In-Service Teachers
TPA reported last fall on the initial stages of work by the Sanford Inspire Program team to create and test on-line
professional development resources. These on-demand learning modules for teacher candidates and novice in-service
teachers are being created to meet specific needs for their audiences, based on research by Sanford. As we reported
earlier, the Sanford Inspire Program team gathered data from in-service teachers, graduates of Teachers College, and
ASU teacher preparation instructors. Results were discussed with preparation program and College leaders. Sanford
also used teacher candidate focus groups to identify teaching skills topics where the candidates wanted additional
information, and the Sanford team tapped performance assessment scores on the TAP rubric to identify areas where
2013-14 ASU teacher candidate performance indicated the need for reinforcement or refinement.
Six on-demand modules are profiled and reviewed in this evaluation report, including a fuller discussion of the
“Causes of Misbehavior” resource that was profiled briefly in October 2014. In each case the module was reviewed
from beginning to end through the ASU Professional Learning Library (PLL) by Sanford’s external evaluator. This
included viewing and interacting with all module components, completing the built-in assessment, reviewing planning
and implementation aids are included with the modules, and looking at the background research literature included
with each module. A brief description of module purposes and goals adapted from the modules themselves is presented
for each of the six:
Causes of Misbehavior: In order to respond to classroom misbehavior, teachers have to understand its “root
causes”. The module addresses some of these causes and provides resources through which teacher candidates
and classroom teachers can identify them and develop effective interventions. The module also presents material
on why teachers sometimes do not respond effectively to misbehavior.
Designing Effective Procedures: A safe and productive learning environment requires effective classroom
procedures. The module explains steps for creating successful procedures, helps users to identify what makes for
effective classroom procedures, and describes three steps for creating them.
2
Giving Clear Directions for a Task: This skill-building module aims to help users give clear directions that
minimize off-task behavior in the classroom “ and set your students up for success”. Among other things,
candidates and teachers utilizing the module will learn about the link between clear directions and fairness or
equity in the classroom, identify criteria that make for good directions, and review how to give directions
successfully.
Identifying Necessary Procedures: The main purpose of this module is to help teachers select procedures that
will enable them to manage the classroom efficiently. As the module overview notes, “ This module will
introduce you to a thought process that will help you to identify procedures to fill needs, fix problems, and solve
inefficiencies in your classroom.”
Physical Environment of Your Classroom: Its purpose is to help teachers arrange or modify their classroom
environment to meet the learning needs of students. The module explains how classroom environment affects
behavior, motivation, and achievement; it offers guidance and practice tools for improving the physical set-up of
a classroom.
Teaching Procedures: The module prompts users to think about the value of classroom procedures, with
examples of how to teach, practice, and reinforce key procedures with students. Users are given the opportunity
to create a plan for teaching a procedure to students, with templates and on-line guidance.
The modules can be accessed through Arizona State University’s Professional Learning Library (https://pll.asu.edu/p/)
or by linking to the Sanford Inspire Program’s Curriculum Resource Catalog (https://pll.asu.edu/p/node/192622). The
work discussed in this document was accessed through the January 21, 2015 edition of the Resource Catalog. Each
module has similar components—an introductory “foundation” section that defines an issue or teaching challenge; a
resource section that offers practice tools for learning about and implementing module materials; the assessment
component where users test and refine their knowledge; the application section of the module where users actually
apply their new knowledge to their own classroom setting; and the conclusion which points to further reading for those
interested in learning more about a module topic.
Some readers of this report may wonder about the need for these on-line learning resources or about the meticulous
attention to small details in the module content, implementation tools, and assessment instruments. Do teachers and
teacher candidates really need to give this much focus to procedural issues in their classrooms? Doug Lemov, in Teach
Like A Champion, provides one answer. Lemov refers to “specific, concrete, actionable techniques” as the basis for
successful teaching. He goes on to tell readers that “many of the techniques you will read about in this book may at
first seem mundane, unremarkable, even disappointing” (page 5), but “carefully built and practiced routines” can have
a powerful impact on student achievement (page 7).
As the Coalition for Psychology in Schools and Education of the American Psychological Association recently stated
in its 2015 publication, Top Twenty Principles from Psychology for PreK-12 Teaching and Learning: “Improved
social and classroom behavior, like academic skill, can be shaped and taught. In the most effective classrooms,
classroom rules and expectations represent a social curriculum that is taught and retaught throughout the academic
year” (page 29). Tapping an extensive research literature, the APA Coalition went on to note that:
Classroom rules and expectations can be taught and re-taught using the same principles as those used in
academic instruction including: clear presentation of a goal, task or behavior; opportunities for practice with
timely and specific feedback; reinforcement for desired behavior; and behavioral correction as needed (page
29).
3
Further empirical evidence for the value of these modules comes from the Measures of Effective Teaching Study
(MET). Employing videotaped observations of 3,000 teachers in six school districts, researchers found significant need
for improvement in areas tackled by the Sanford on-demand modules. For example, although 75% of observed
teachers got the highest positive score for classroom management on the Danielson Framework For Teaching (FFT),
only 5% of the same teachers received the highest score for managing classroom procedures. Similarly, a majority of
observed teachers obtained the top behavior management rating on the CLASS observation instrument, but only 15%
scored at that level for what CLASS calls “productivity” (defined as how well instructional time and routines are
handled). And MET study ratings on the UTeach observation instrument—known as UTOP—indicated that 40% of
observed teachers received the highest score for “students on task” and for classroom management, but 20% scored
equally well on lesson organization, and 15% for allocation of time during instruction.1
The Sanford Inspire Program’s curriculum team relied on evidence about candidate and teacher needs to identify
module topics, and they reached into the literature about effective teaching to shape the content of these modules. As
Lemov and the APA group indicate, the needs are real from the perspective of teachers and their students. The payoffs
for successful practice are equally real—well-organized classrooms, maximized opportunities for students to learn, and
the likely outcome of enhanced student achievement.
Developing and Testing the On-Line Modules
All six modules were developed, pilot tested, refined, and completed through a formal development process. The key
steps included deciding on pre-service and in-service teacher needs from the data analysis strategies described in the
first paragraph of this section; reviewing research literature to ensure that module content provides a research-based
learning resource; consulting with university faculty and/or subject-area experts; embedding learning assessment tools
in the module to gauge how well users are mastering its content; pilot tests with teachers, teacher candidates, and
teacher educators; refining the module from this actual use and feedback; and final production steps to ensure
professional polish as well as ease of use by those hoping to improve their practice.
All modules are pilot tested and feedback is reviewed carefully to ensure that products and processes are strong and
meet the needs of users. Sanford team members described for the evaluator the pilot testing process as well as specific
changes made as a result of feedback. The premise of pilot testing is that modules “are not perfect out of the gate” and
can be improved through this extensive review and feedback processes. Changes that resulted from feedback have
included additional research studies as sources for module content, improved navigation, additional grade- or contentarea examples in the module, branching within a module to give users more choices to fit their specific needs, closed
captioning, and editing to reduce the size of sections within modules so that users can return to a specific section more
quickly as needed. Sometimes a module goes through a second round of pilot testing after modifications based on first
round feedback.
Feedback from actual users was based on their experiences with pilot versions of each module. There were 157
participants in these pilot tests, including groups of teacher candidates (N=75), practicing teachers (N=35), ASU
teacher educators (N=18), and school administrators (N=20). Participants were asked to evaluate module content, ease
of use, relevance to their own teaching, the extent to which the module use experience was engaging, whether they
would use its content in their own classroom, and how likely they were to recommend it to others.
As a general summary of this feedback for the six modules, nearly everyone found the on-line module easy to use
(except for Identifying Necessary Procedures, where about 28% in the pilot test disagreed with the survey statement
that the module was “easy to navigate throughout”). With regard to the feedback about Identifying Necessary
Procedures, the Sanford team responded by improving the on-line navigation process, reducing the amount of typing
users have to do, and made some content optional. Nearly every user was able to complete a module in 60 minutes or
less (about 95% of all pilot users), an important finding for the Sanford marketing strategy since potential users have
many claims on their time and energy. Pilot test groups reported that module content in all six examples was relevant
to their own practice and/or to teaching. Over 80% of respondents found modules to be “engaging”, an important trait
for a self-directed online resource that most users will be interacting with in the context of limited time and perhaps
after a long day or week of teaching In the case of the Teaching Procedures module, however, only 59% of those in
pilot group found it engaging. According to a communication to the evaluator from the Sanford team, the module is
under review to address this finding. Even so, all of them would use its content in their own work, a finding echoed by
almost every participant in the six pilots. And finally, between 86-100% of those providing feedback said they would
4
recommend the module to a colleague. This suggests that users of the modules will become a big source of
dissemination to their colleagues. It can also be seen from the table below that nearly every pilot user of the module—
from four different groups—reported that the module they tested was “high quality”.
Slight variations in these findings owe more to small sample sizes of individual module pilot test groups. Even so,
however, it would be worthwhile for the Sanford team to take a look at the feedback for Identifying Necessary
Procedures and Teaching Procedures, where feedback was somewhat less positive than for the other modules.
Selected Survey Results from Pilot Tests of On-Demand Modules2
Module
Survey Questions
Percent Agree/
Strongly Agree
Causes of Misbehavior
Easy to Navigate
100%
Engaging
92%
High Quality
100%
Would Use in Teaching
100%
Recommend to Others
100%
Designing Effective Procedures
Easy to Navigate
86%
Engaging
86%
High Quality
91%
Would Use in Teaching
88%
Recommend to Others
90%
Giving Clear Directions for a
Task
Easy to Navigate
88%
Engaging
84%
High Quality
95%
Would Use in Teaching
100%
Recommend to Others
96%
Identifying Necessary
Procedures
Easy to Navigate
71%
Engaging
86%
High Quality
100%
Would Use in Teaching
94%
Recommend to Others
95%
Physical Environment of
Classroom
Easy to Navigate
92%
Engaging
81%
High Quality
92%
Would Use in Teaching
100%
Recommend to Others
93%
Teaching Procedures
Easy to Navigate
82%
Engaging
59%
High Quality
89%
Would Use in Teaching
100%
Recommend to Others
86%
Observations3
5
Apart from the comments above, it is also noteworthy what individual pilot test participants had to say about the
quality and value of the on-demand modules—for themselves and for other candidates or classroom teachers.
Reviewers of Causes of Misbehavior had these comments about the module and its value:
• “I thought it was concise, coherent and left you with tangible take away to use immediately in your classroom.”
• “My initial thoughts regarding the module on misbehavior was that it was good review from the content we have
studied in previous classes. I also thought that it would help a beginning teacher think about misbehavior and how
tools can be used to assess students.”
• Another reviewer said: "This module would push novice teachers to consider behavior from a different perspective
and hold themselves accountable for not making assumptions."
A sample quote from reviewers of Giving Clear Directions also indicates its value for novice teachers:
"Yes, I think novice teachers would find this very helpful. Giving instructions efficiently and effectively
helps eliminate many of the problems that novice teachers are most likely to struggle with, including
difficulty meeting objectives and classroom management issues. The information was also presented in a
very practical and applicable way, which I appreciated and which I'm sure new teachers will appreciate as
well. In the teaching program, we receive a lot of theoretical information with little practical application, and
this was very refreshing in its practicality."
One reviewer for Designing Effective Procedures gave positive feedback as well as this suggestion: "Good, but some
of the examples were a bit too generic. However, for a first year teacher I could see why it would be beneficial. I think
the module should have gone over being consistent with procedures because I think that's even harder to do."4 Another
person said: "This module provides valuable tips and sequential steps for the development of procedures."
Pilot use of Identifying Necessary Procedures prompted insightful thoughts from reviewers. One comment had
advice for which users would be most likely to benefit: "I think procedural issues often occur b/c teachers are unaware
of the disorganization/chaos. The checklist is helpful but I imagine the module would be better if assigned to someone
who has been identified to struggle in this area. I don't imagine it to be a self-selected module." Another
reviewer identified an interesting issue in marketing and using teaching related modules, commenting that novice
teachers would not be likely to find it helpful because "I think a lot of novice teachers feel they don't need
improvement.” As noted in the table above, most reviewers were quite positive in their assessments of this module’s
quality and relevance to their classroom—including, presumably, classroom teachers who were not all veterans. The
survey findings indicate that most reviewers have been in their current role for four or fewer years, and Sanford reports
that only a handful of reviewers had more than five years of experience. Survey answers suggest that “93% of inservice teachers who piloted these modules were in our 5 or less years target audience range.” This means that pilot
test reviewers overwhelmingly come from the same teaching experience pool as teachers for whom the modules have
been designed.
The Teaching Procedures on-demand module also received interesting comments from reviewers. One responded, "I
do think that novice teachers would find this helpful. I think that the clarify, explain, rehearse, reinforce is a perfect
model for novice teachers to follow, “ while another reviewer agreed with this but went on to say: "However, I would
include videos to supplement each part of the module. I think this would exemplify the concepts more clearly and
provide a clear illustration of practical classroom strategies for teaching procedures. The students can then better
understand the steps in the module and what it looks like in an actual classroom."
Recommendations
These on-demand modules have obvious value to teacher candidates, current practitioners, and teacher educators at
Arizona State University and elsewhere. Comments and ideas cited earlier from Doug Lemov, the APA Coalition, and
the Measures of Effective Teaching project indicate the extent to which the Sanford Inspire Program is responding to
real needs in preparation programs and K-12 classrooms. Aside from relatively minor comments about small-scale user
issues in the modules that will be conveyed separately to Sanford, there are two significant recommendations that grow
out of this external evaluation of the on-demand module initiative:
6
•
•
•
•
•
It will be important to track and share impact data from implementation of the final version of each module,
including frequency of use and by whom (e.g., candidates, teachers, program faculty associated with ASU and
externally). Sanford plans to do this, and a future evaluation report will tap into this information to explore how the
modules are being used.
Sanford should expand the research base that supports development and implementation of each module.5 It is
worth going more deeply into the research behind future modules to establish the rigor of the knowledge base
employed by the Sanford team. This step would strengthen the value of on-demand modules to university faculty,
making it more likely that courses and course experiences would be linked to or influenced by the modules.
Designing Effective Procedures is an example of a module where a fuller set of references would strengthen its
appeal to some potential users. Unless there really is no other relevant literature, this very short list is likely to get
in the way of faculty adopting and recommending the work. And it may raise questions about the extent to which
the developers really know the literature related to the topics that are being addressed through the modules.
If developers have relied on studies in addition to those cited at the conclusion of each module, consider adding
these to the list or creating a link between each module and a full set of references organized by module title.
Finally, Sanford might consider identifying a tenured or tenure-track faculty member with expertise in the subject
to be an adviser on each module development project. The idea behind this is not to turn each module development
project into a graduate thesis or a research paper but to ensure that the module is grounded in—and is seen to be
grounded in—a reasonably extensive research base. The Sanford Design Team made it clear in interviews with the
external evaluator that feedback from College faculty is sought after and highly valued, so this recommendation
should not be taken to imply that external faculty feedback is not a component of the module development process.
K-12 Student Surveys
The Sanford Inspire Program’s K-12 student survey initiative has continued and expanded over time, reaching more
schools, classrooms, and students. Through partnerships with individual schools and with the Teacher Retention
Project, the Sanford survey initiative is providing new and valued feedback across the ASU service area. Beyond these
activities, the use of student surveys is being embedded in teacher preparation programs at the Mary Lou Fulton
College of Education as a source of feedback for teacher candidates and as evidence of performance for programs.
By the end of 2014, K-12 student surveys had been administered in 76 classrooms at twelve different schools. Almost
two thousands students participated by completing surveys on paper or on-line. This external evaluator observed
survey administration in a second grade classroom where just over two dozen students answered the Sanford survey
questions with pencil and paper. Survey administration in this second grade classroom had multiple purposes, as it did
in other schools and classrooms. Surveys have generated findings of interest to school leaders and their teachers,
produced useful feedback for ASU teacher candidates and their instructors, and given College program leaders insights
about the value of student surveys as a source of useful performance data. And beyond all these benefits, lessons from
each survey experience have been used to improve the overall process of administration, analysis, and reporting.
According to Sanford, each student survey cycle has yielded insights about how to generate this feedback from
students more efficiently and with less intrusion on the time and energy of school administrators. Survey instruments
have been tweaked to reduce the number of questions and even lower the average number of words per item, resulting
in significantly fewer items on the Sanford elementary and secondary survey instruments than are found on typical
commercial products such as Tripod or MyVoice. And as TPA reported last summer, in 2015-16, the College is
planning to pilot surveys with all teacher candidates. In the 2016-17 academic year, will require surveys of their
students by all teacher candidates in the program by including the survey as an assignment in the student teaching
course.
It is also important to note that one of Sanford’s partners in the student survey initiative—the Teacher Retention
Project (TRP) utilized the survey without direct assistance from the Sanford Inspire Program. TRP coaches who work
with novice or relatively inexperienced teachers in Phoenix-area school districts administered the survey themselves.
Results were tallied by Sanford and returned to TRP.6 Results are also being shared with TRP teachers. A separate
interview with Dr. Teressa Ribbens, chief executive officer of the Teacher Retention Project, provided other details of
this experience as well as examples of the benefits from collaborating with Sanford. Sanford also partners with the
Teacher Retention Project through access to the on-demand modules discussed in the previous section of this report.
7
TRP coaches are assigned to teachers in schools across Arizona, working to help them improve their practice and
thereby build successful careers in the teaching profession. According to Dr. Ribbens, TRP coaches were trained to
administer the student survey by Valerie Simmons, Sanford’s internal evaluator, and 34 teachers supported by TRP
have used the Sanford survey with their own students (primarily completed by students in school computer labs), all
but one school served by the TRP. The resulting survey findings gave “clear and accurate results” helpful to teachers,
coaches, and schools—and the findings have “driven next quarter goals for our teachers.” Dr. Ribbens also praised the
readability of survey reports generated by Sanford; she said they stand alone as a coaching and teacher devleopment
resource but also are valuable because their content links back to the Danielson Framework For Teaching used as an
observation instrument in local schools. She also commented about the student survey marketing opportunity that
exists through her TRP coaches who can talk up the survey’s value with principals and others in the schools where
they work with individual teachers.
In the way of recommendations about the student survey experience, comments from Dr. Ribbens echoed the Sanford
experience described in a previous evaluation report about the surveys. As Val Simmons said in discussing the survey
initiative with the external evaluator, Sanford is “being careful not to overask or overreach” in working with schools
and districts. This outlook has facilitated the early successes of the student survey initiative. Arranging permissions
with schools to administer surveys has been challenging for Sanford in part because of University institutional research
board (IRB) deadlines and policies and because Sanford has to navigate school district-level policies as well. These
issues are being worked out as more experience is gained in the survey work, supporting the longer-term focus on
student surveys that will ensure smoother administration in the future. Dr. Ribbens pointed to survey results where
TRP teachers can use additional development in areas such as the general academic domain, academic engagement of
students, questioning strategies and skills, and differentiated instruction. These areas also stand out in the MET report
as weaknesses for most of the 3,000 teachers who participated in that study.
As Valerie Simmons and Ryen Borden wrote in “The Use of Student-Facing Surveys in the Classrooms of Teacher
Candidates to Advance Teacher Preparation Practice,” a paper submitted to the 2015 annual conference of the
American Association of Colleges for Teacher Education (AACTE), “We see the use of student surveys with
preservice teachers as an innovative practice with great potential value to the teacher candidates and ultimately, the
students they serve.” They also commented that Sanford’s survey administration and reporting processes were
developed so that they are “free to low cost... sustainable in its administration and distribution protocols...[and]
replicable in other institutions with little background training and/or knowledge”.7 In addition to providing context for
Sanford’s goals and experiences, the papers cited here provide solid evidence about the dissemination strategy the
Sanford Inspire Program is using to help others learn from its work.
Earlier evaluation reports commented on the growing national interest in K-12 student surveys as a source of feedback
for teachers, teacher candidates, and preparation programs. Perhaps stimulated by the highly visible Measures of
Effective Teaching project, states, schools, and preparation programs across the country have been developing student
surveys. As MET noted in “Asking Students About Teaching,” good quality student surveys can complement
classroom teaching observations and measures of student achievement in providing insights about teachers and their
impact on students. “Teaching is a complex interaction among students, teachers, and content that no one tool can
measure. The search for different-but-aligned instruments has led many to use student surveys,” according to MET
researchers. Moreover, they have enhanced value when standardized test data are not available:
“Analysis by the Measures of Effective Teaching (MET) project finds that teachers’ student survey results
are predictive of student achievement gains. Students know an effective classroom when they experience
one. That survey results predict student learning also suggests surveys may provide outcome-related results
in grades and subjects for which no standardized assessments of student learning are available”.8
A task force of education leaders and measurement experts convened by the American Psychological Association also
cited the benefits of asking students about the performance and attitudes of their teachers. In “Assessing and
Evaluating Teacher Preparation Programs,” the APA Task Force said that “States should work with teacher preparation
program providers to design systems of data collection that include information collected at the stages of selection,
progression, program completion, and postcompletion”.9 The APA group went on to make these observations about the
value and quality of student surveys as a measure of teaching performance:
8
“[S]tudent surveys of teacher effectiveness have considerable support in the empirical literature. Scores of
constructs based on observable behaviors are internally consistent and stable, are related to achievement
outcomes in both college and K–12 students, are more highly correlated with student achievement than are
teacher self-ratings and ratings by principals, and distinguish between more- and less-effective teachers
identified using other metrics”.10
And finally, in work produced by Teacher Preparation Analytics (TPA) for the Council on Accreditation of Educator
Preparation (CAEP), the authors of “Building an Evidence-Based System for Teacher Preparation” included K-12
student surveys of teaching practice as a key indicator of preparation program quality.11 While most states and
preparation programs are not yet deploying student surveys for program improvement or for accountability, the
Arizona State University experience offers valuable insights about ways to move in this direction.
Marketing Strategies to Support Phase 2 of the Sanford Inspire Program
Since the early 2014 transition from Phase 1 to Phase 2 of the Sanford Inspire Program, marketing has focused on
supporting the transition in a variety of ways. This includes marketing team work on reviewing and revising Phase 1
curriculum products to ensure that these products will still be available and attractive as pre-service and in-service
curriculum or professional development resources. Phase 2 marketing has seen the team play an important role in the
work of pilot testing on-demand modules, engaging with schools and districts to support the student survey initiative,
and disseminating to other teacher preparation programs a Recruitment Toolkit built from the Sanford Inspire
Program’s early Phase 1 recruitment activities.
While marketing involves “driving demand to the modules from potential users”, the Marketing Team participates
significantly in several design and production phases. Working with the design and production teams, marketing vets
the clarity and content of individual modules (e.g., module titles to convey content as clearly as possible) from the
perspective of prospective users. Team members look at a module’s ease of use and provide other feedback to
maximize its value to that user. They also work to set up and manage pilot reviews of the modules to generate the
kinds of feedback discussed above in this report. This ensures that marketing is fully in sync with product development
and that product development is a marketing asset.
The Sanford Inspire Program’s student survey initiative also depends on marketing expertise. Team members traveled
to schools and district with Valerie Simmons for meetings and to observe early implementation of the surveys, with the
idea that the team will have direct responsibility as the student survey initiative widens its scope in terms of schools
and districts. In late fall 2014 evaluation interviews, the team felt that the student surveys had been the most successful
Phase 2 Sanford product marketed to schools because of its benefits to individual schools and their leaders.12 One
team member explained this in terms of the pitch to school leaders: “we’d like to come in and survey your students to
help you better understand your teachers’ effectiveness through the eyes of your students.”
At the same time, marketing work aimed towards schools and districts that have used the student surveys will
emphasize the on-demand modules as professional development resources and their linkages to survey findings.
Sanford is also underscoring connections that surveys and modules have with teacher evaluation rubrics used in
surrounding districts, schools, and charters. An internal Sanford document developed in 2014 is a kind of crosswalk or
Venn diagram showing overlaps among local evaluation systems—the TAP rubrics in districts employing that system,
Danielson’s Framework For Teaching, the Marzano system, and Teaching as Leadership (TAL).
Just listing these multiple ways through which schools try to assess teaching quality points up the complexity of
developing resources to help large numbers of schools, their teachers and pupils. But intentional Sanford efforts to
align—and to make alignment connections explicit—these approaches with the student survey and the on-demand
module initiative are evidence for a carefully-designed and thoughtful set of strategies to advance Phase 2 work across
all of its many fronts.
In broad strokes, marketing activities have two big audiences. Outside Teachers College, these include the schools and
districts discussed above, as well as initiatives like the Teacher Retention Project whose leader and coaches report
great value in their work with Sanford. Communications materials are being created to share with this audience, and as
time goes on, marketing will have sole responsibility for promoting the use of surveys and managing the transition
9
through which schools and districts will take up the survey work on their own. In this sense, marketing is also about
sustainability.
Within Teachers College, marketing includes recruiting faculty reviewers of the pilot modules but it also about efforts
to inform as many internal constituencies as possible. This is no easy task in a large and complicated College where
faculty and staff work at multiple locations, where they staff very large programs, and when they are subject to
relatively frequent large scale upheavals. Sanford outreach seeks to inform and engage course instructors, tenured and
tenure-track faculty, program coordinators, and program leaders—especially about Phase 1 resources they may want to
use.
One marketing team member summed up the challenge of this work at a very large institution by noting, “different
faculty have different perceptions of what we [Sanford] do”. Some know about Phase 1, others are familiar with the
prior recruitment focus. “They have different snippets of what we do... most people have heard about us but they’re not
clear about what we do.” The most productive outreach seems to be with those involved in courses for which Sanford
has a resource—instructors, course and program coordinators, and those who lead programs where Sanford resources
can and do fit in. All this is not say that Sanford is unsuccessful in the work of informing, engaging, and involving
faculty across the College. Participation in module reviews, College support for scaling up the student surveys across
all pre-service teacher education, partnering in research and dissemination activities are concrete examples of
awareness and support.
Recommendations
Sanford Inspire Program team members made the effort to find commonalities among a number of teacher observation
and evaluation systems used by the College and by school or district partners where Sanford resources are used or
marketed. This is a clever way to ensure that school and district leaders see the student surveys and on-demand
modules not as added work or peripheral to their goals , but complementary to their efforts to improve instruction and
raise student achievement.
In that same spirit, Sanford should consider obtaining and analyzing local district teacher evaluation instruments other
than those mentioned already, looking for relevance and alignment with the student surveys and on-demand modules.
Teressa Ribbens from the Teacher Retention Project stressed the value of this alignment for her coaches and their
efforts to improve instruction and persistence in teaching. Making specific connections between evaluation rubric
components and surveys or modules can open the door to conversations with principals or district leaders. The next
step might be sharing actual evaluation results—not for individuals but maybe for grade levels or subject areas—to
explore ways in which the surveys and current or future on-demand modules can be used to solve specific problems.
Dissemination Activities
An important aspect of the Sanford Inspire Program from the beginning has been dissemination information and
resources across the ASU community, and to local schools, districts, and other partners. Sanford’s work in Phase 1
and Phase 2 has big implications for teacher preparation and teacher professional development, and so dissemination
also has aimed to reach and engage those communities as well. As this report is being completed in February 2015,
Sanford team members are preparing to make presentations at two national conferences of teacher educators—for the
Association of Teacher Educators (ATE) and the American Association for Colleges of Teacher Education (AACTE).
At ATE, papers are being presented on data-driven decision-making and on the role of K12 student surveys in
fostering reflective practice by teaching candidates. A few weeks later, AACTE conference attendees will learn about
Sanford’s work on the student survey initiative and about research findings on “the outcomes of a clinically-based
student investment project.” AACTE’s meeting agenda has at least one other session about student surveys—presented
by the teacher education accreditor body because well-done surveys will be accepted as a form of evidence about
program performance. Thanks to Sanford, Arizona State University is a national pioneer in developing survey-based
evidence about teacher candidates and practicing teachers. Another example of dissemination to education
professionals is a paper developed by Valerie Simmons accepted to the 2015 conference of the Society for Information
Technology and Teacher Education (SITE). The Sanford team is also sharing its work widely through a quarterly
newsletter and by providing the on-demand modules to teacher candidates through their course and field experience
instructors.
10
Other dissemination work took place through the Learning to Teach (LTT) conference in late fall 2014, involving
innovators, funders and policymakers from across the country. Media coverage has highlighted Sanford initiatives such
as Quest2Teach and mentor teacher training. A redesigned website for the Sanford Inspire Program is generating
significant traffic; Sanford resources are much easier to identify and link with through this polished website.
Teachers College Dean Mari Koerner was one of a small number of education deans announced as members of “Deans
for Impact” in January 2015. This is a change-focused group that has received funding support from the Schusterman
Family Foundation because members “are working together to improve student learning and other outcomes” of
teacher preparation programs.13 Teachers College at ASU has seen massive changes to policies and practices under
Dean Koerner’s leadership; her support for the Sanford Inspire Program (as well as growing awareness of Sanford’s
work) has increased the visibility of the teacher preparation reform agenda being implemented at ASU and may well
have an impact on the work of Deans for Impact.
It should also be clear that Sanford’s engagement with the Teacher Retention Project, the Paideia Academies and
Academia del Pueblo Elementary Charter School is about dissemination through close partnerships. Feedback from
these organizations about the student surveys and on-demand modules shows the Sanford Inspire Program is making a
difference to their work in very specific ways. Through this outreach and support, Sanford is also laying the
groundwork to sustain Phase 2 of the Sanford Inspire Program.
Final Thoughts
Interviews and other evidence collected and analyzed for this report indicates that the Sanford Inspire Program has
made a very rapid transition into Phase 2 of its work. At the same time, the Sanford team has been taking steps to
improve and continue its support for the Phase 1 curriculum resources that have been assessed in previous evaluation
reports. As the teams work on multiple fronts, extensive interviews and internal documents from team meets indicate
the existence of clearly articulated strategies being pursued by each of the teams interviewed for this report. Team
members have a clear understanding of their goals, well-developed and detailed strategies for getting to them, and
strong commitment to the overall effort to serve the needs of schools, teachers, teacher candidates and others touched
by the Sanford work.
As in previous interactions with them, Sanford Inspire Program team members continue to demonstrate a remarkable
openness to feedback and constructive ideas from their partners, from resource reviewers, and from this external
evaluator. With the work moving forward on new fronts at a rapid pace (as always), assessment of this complex project
is coming closer than ever to having impact data as the College continues to build its own capacity to tap data on
program outcomes.
1
Findings from the Measures of Effective Teaching study are contained in several reports available through
http://www.metproject.org/.
2
The surveys were developed and administered by Valerie Simmons, internal evaluator for the Sanford Inspire
Program, who generously shared findings and analysis with TPA for this report.
3
Notes taken while reviewing the six modules discussed here include other observations about smaller-scale issues
that cropped up—positive assessments about how a module is addressing a specific topic (e.g. in Causes of
Misbehavior, “The emphasis on specific easily understood steps that can be used to diagnose and respond to particular
events/incidents is a strength—users can draw parallels to their own classroom, they can also ground the module’s
ideas in practice, something that is too often missing in teacher prep coursework.”) as well as notes about typos or
duplicated choices in some assessment questions—are provided to the program separately from this evaluation report
(such as examples from Designing Effective Procedures).
4
Sanford’s response to this feedback: “Because we wanted to keep the modules short (<60 min) and focused, we
address this in another module… This is addressed in the Teaching Procedures module, which explains the
importance of reinforcing/re-teaching procedures as necessary over time.”
11
5
The team working on creating modules told this external evaluator “On average, each of the last half-dozen modules
that have been sent to production has had a minimum of fifteen references (consisting almost wholly of books and
scholarly articles).”
6
As with other data in this report such as survey feedback about the on-line modules, information about student survey
administration has been provided to TPA by Valerie Simmons, Sanford’s internal evaluator.
7
For more see http://sanfordinspireprogram.org/wp-content/uploads/2015/01/AACTE-Student-Survey.pdf and
http://sanfordinspireprogram.org/wp-content/uploads/2015/01/Abstract-of-Presentation-Student-Surveys-ATElive.pdf.
8
For more on the Measures of Effective Teaching Project’s findings and recommendations regarding student surveys,
see http://www.metproject.org/downloads/Asking_Students_Summary_Doc.pdf.
9
American Psychological Association, 2014. Assessing and Evaluating Teacher Preparation Programs. Washington,
DC: APA Center on Psychology in Schools and Education. Retrieved from
http://www.apa.org/ed/schools/cpse/teacher-preparation-programs.pdf.
10
APA Task Force, page 24. Citations embedded in the quoted text have been omitted but can be found in the original
reference.
11
A summary of this report can be found at https://caepnet.files.wordpress.com/2014/12/tpa_report_highlights.pdf.
12
When evaluation interviews were conducted only one on-demand module was publicly available; the strength of the
survey initiative as a marketing opportunity should not be taken to imply weak demand for or interest in the modules.
13
For more see http://www.deansforimpact.org/.