Videocase-based, Analysis-of-Practice for Teacher and
Videocase-based, Analysis-of-Practice for Teacher and Student Learning: How To’s from a 10‐year Line of Research Jody Bintz, BSCS Connie Hvidsten, BSCS Paul Numedahl, BSCS Chris Wilson, BSCS Kathy Roth, California State Polytechnic University-Pomona Goals • Increase understanding of framework‐ centered, videocase‐based analysis of practice as a focus for transformative professional development • Increase awareness of the power of developing a line of research © 2015 BSCS BSCS Mission To transform science teaching and learning through research and development that strengthens learning environments and inspires a global community of scientifically literate citizens. © 2015 BSCS Opening • Introductions – Name – Role – What attracted you to this workshop? © 2015 BSCS STeLLA Line of Research Research on Student Ideas in Science STeLLA I ViSTA STeLLA II EMAT RESPeCT ViSTA Plus © 2015 BSCS Science Teachers Learning from Lesson Analysis (STeLLA) STeLLA is … • A year‐long, videocase‐based, analysis‐of‐ practice professional development program designed for upper elementary teachers • An RCT, scale‐up study with exciting results • A line of research © 2015 BSCS STeLLA is a PD Program • • • • STeLLA theory of teacher learning STeLLA program substance STeLLA program form STeLLA program resources © 2015 BSCS STeLLA Theory of Teacher Learning • Situated cognition theory of teacher learning • Cognitive apprenticeship instructional model Program elements that embody this theory: 1) Learning progression starting with highly scaffolded experience moving towards greater independence 2) Experiences that create a “need to know,” dissonance 3) Use of experts who plan for and guide teacher learning © 2015 BSCS STeLLA Program Substance: STeLLA Conceptual Framework © 2015 BSCS © 2015 BSCS STeLLA Program Substance: Teacher Learning Goals • Science content knowledge (CK) • Pedagogical content knowledge (PCK) related to the Student Thinking and Science Content Storyline Lenses • Ability to use CK and PCK to analyze teaching and learning • Ability to use CK and PCK in teaching science © 2015 BSCS STeLLA Program Form Summer Institute 2 weeks (9 days) 58.5 hours face‐to‐face PD Half‐day content deepening, led by university science faculty Half‐day videocase‐based, lesson analysis sessions in grade‐level study groups of 5‐10 teachers, each led by a PD leader © 2015 BSCS STeLLA Program Form Fall • Grade‐level study groups led by STeLLA PD leaders • Teachers analyze lesson video and student work from their teaching of Academic‐year: the STeLLA lesson plans Monthly 3.5‐hour study Winter/Spring group meetings • Grade‐level study groups led by STeLLA PD leaders 30 hrs face‐to‐face PD • In a second content area, teachers o analyze lesson video and student work from other teachers o plan lessons collaboratively © 2015 BSCS STeLLA Program Form End of program Teachers teach lessons they designed in the second content area Total hours of face‐to‐face PD 88.5 © 2015 BSCS STeLLA Program Resources • • • • • • • Videocases Lesson Analysis Process, Norms, and Protocol Analysis Guides STeLLA lesson plans and pre‐post tests Features Analysis Charts Planning Tools Readings: – STeLLA Strategies booklet – Content/PCK Background for each content area – Common Student Ideas in each content area © 2015 BSCS So STeLLA is … A one‐year professional development program for elementary teachers STeLLA Professional Development Program Teacher Science Content Knowledge Teaching Practice Teacher Pedagogical Content Knowledge Student Science Content Knowledge In the context of videocase‐based analysis of practice © 2015 BSCS STeLLA is also: A scale‐up study © 2015 BSCS Population 77 schools, 144 4th and 5th grade teachers from ~2,800 students Sample 77 Schools STeLLA Comp 42 Schools 35 Schools © 2015 BSCS Treatments STeLLA COMPARISON Lesson Analysis + Content Deepening Content Deepening 88.5 hours (2‐week summer institute & monthly meetings) 88.5 hours (2‐week summer institute & monthly meetings) Same science content learning goals © 2015 BSCS Overview of STeLLA PD Content Deepening Program Summer Institute, 2 weeks Full‐day science content sessions with university science faculty • • STeLLA PD Program Half‐day science content sessions with university science faculty 4th grade: food webs, Earth’s changing surface 5th grade: water cycle, Sun’s effect on 5th grade: water cycle, Sun’s effect climate and seasons on climate and seasons 4th grade: food webs, Earth’s changing surface Half‐day lesson analysis study groups with BSCS PD leaders 4th grade: Earth’s changing surface 5th grade: Sun’s effect on climate and seasons Academic year, 30 hours Content sessions with university science faculty Lesson analysis study groups Fall: Content Area 1 Winter/Spring: Content Area 2 © 2015 BSCS Theory of Change and Focus Today STeLLA Professional Development Program Teacher Science Content Knowledge Teaching Practice Teacher Pedagogical Content Knowledge Student Science Content Knowledge © 2015 BSCS An Issue: Instructional Sensitivity PD Program Student Outcomes Teacher Learning Trickle Down ` Pretest Posttest Pretest Posttest Treatment Comparison Pretest Posttest Pretest Posttest Treatment Comparison Teacher Level Student Level © 2015 BSCS An Issue: Instructional Sensitivity “Education reform usually arrives with fanfare, great expectations, and overconfidence. Truth be known, typical education‐reform effects tend to be small. Evaluations, if done at all, burst the reform balloon, having difficulty finding effects.” Ruiz‐Primo et al., 2002 © 2015 BSCS Due to the scale of many efficacy studies, we are often limited to crude measures. (e.g. 150 teachers, pre and post, 4 topics, 30 student per class = ~36,000 student tests) © 2015 BSCS Use Validated Instruments © 2015 BSCS Instructional Sensitivity • • • • Learning goal specification and monitoring Developing construct models Item selection and development Pilot testing and item/instrument analyses Learning Goals Item Difficulty Difficulty Number Pre NOVICE 7 .32 8 .12 9 .21 Post .65 .53 .29 Curriculum Item Materials Item‐Total 1 Discrimination EXPERT Correlation .77 .33 .21 Backward Design (Wiggins and McTighe, 1998) .51 .52 .19 Pre‐Post 2 Difference .33 .41 .08 Learning Goals Instructional Sensitivity .49 .47 .10 Flag X Curriculum Mapping (Ruiz‐Primo et al., 2010) © 2015 BSCS Increase in Sensitivity Example © 2015 BSCS Findings STeLLA Professional Development Program Teacher Science Content Knowledge Teaching Practice Teacher Pedagogical Content Knowledge Student Science Content Knowledge © 2015 BSCS Group Similarity School‐level demographics % Female % Limited English proficiency % Free/reduced price lunch % Asian % Black % Hispanic % White Urbanicity Urban Suburban Rural Overall % for all treatment schools (n = 42) Overall % for all comparison schools (n = 35) 48 12 44 2 5 28 59 % of schools in treatment group (n = 42) 48 13 41 3 3 28 59 % of schools in comparison group (n = 35) 43 40 17 63 23 14 © 2015 BSCS Outcome Measures • Four separate outcome measures – Food Webs – Earth’s Changing Surface – Water Cycle – Sun’s Effect on Climate and Seasons • Twenty‐four multiple choice items on each test • Equated to one overall test to preserve randomized sample using common person equating and the Rasch Measurement Model © 2015 BSCS Impact Findings Level Two Parameter Estimates (Fixed Effects) Level 2 fixed effects Unstandardized coefficient Standard t‐ratio error DoF p‐value INTERCEPT 53.22 0.42 127.77 74 p < .001 TREAT 6.11 0.84 7.27 74 p < .001 PRETEST 0.39 0.15 2.66 74 p = .01 © 2015 BSCS Practical Significance • Effect size (Hedge’s g) = 0.68 • Larger‐than‐average effect for elementary school interventions with this type of outcome (avg = 0.33) • Nearly a two‐year advantage • 23 percentile points • Advantage in reasoning © 2015 BSCS Nationally Generalizable Result State (N) B‐index Generalizability USA 57,763 0.86 High CO 1,150 0.95 Very high MA 1,177 0.87 High IL 2,673 0.85 High NY 2,992 0.80 High NV 429 0.80 High © 2015 BSCS . | WSPO23 Harder Items Group 1: Items with high difficulty in .##### | both treatment groups . | FSPO3 SSPO1 .###### S| WSPO24 .### |T 60 .###### + ESPO7 WSPO14 Treatment group Comparison group ## | (pretest adjusted) (pretest adjusted) ####### | mean = 56.03 mean = 50.57 .###### | ESPO16 SSPO23 Group 2: Items the average .####### | ESPO9 FSPO5 treatment group student had a .###### |S ESPO19 FSPO14 SSPO12 SSPO7 WSPO4 greater than 50% chance of .######### | ESPO21 FSPO6 SSPO11 answering correctly, and the .####### M| ESPO11 ESPO12 ESPO15 FSPO16 SSPO15 average comparison group .########### | ESPO24 FSPO15 SSPO2 SSPO24 SSPO6 student had a less than 50% .#### | ESPO22 ESPO6 ESPO8 FSPO4 SSPO22 WSPO2 WSPO5 chance of answering correctly. 50 .############ + SSPO17 SSPO9 WSPO21 WSPO7 .#### | ESPO2 ESPO20 FSPO20 SSPO3 SSPO5 .###### |M ESPO1 ESPO14 ESPO23 SSPO13 SSPO4 WSPO13 WSPO17 .########## | ESPO3 FSPO1 FSPO21 SSPO19 SSPO21 WSPO15 WSPO16 WSPO19 WSPO20 .### | FSPO8 SSPO16 SSPO18 WSPO18 .####### | FSPO10 SSPO14 WSPO12 .#### S| ESPO10 ESPO13 ESPO4 SSPO20 .## | FSPO23 FSPO24 SSPO8 WSPO9 .###### | ESPO5 FSPO12 FSPO13 .## |S ESPO18 FSPO11 FSPO18 SSPO10 WSPO10 40 .## + ESPO17 FSPO2 FSPO22 WSPO22 .## | .## | FSPO19 WSPO6 WSPO8 .# | FSPO9 WSPO1 .# | WSPO3 . T|T FSPO17 . | FSPO7 Group 3: Items with low difficulty in Low Ability . | both treatment groups Easier Items . | Students . | 30 + WSPO11 High Ability Students © 2015 BSCS . | WSPO23 Harder Items .##### | . | FSPO3 SSPO1 WSPO23: Which of the following statements about condensation is true? .###### S| WSPO24 A. During condensation, water turns from a liquid to a gas. .### |T B. During condensation, energy is lost. 60 .###### + ESPO7 WSPO14 C. During condensation, water gets heavier. ## | ####### | D. During condensation, water is created. .###### | ESPO16 SSPO23 .####### | ESPO9 FSPO5 .###### |S ESPO19 FSPO14 SSPO12 SSPO7 WSPO4 .######### | ESPO21 FSPO6 SSPO11 .####### M| ESPO11 ESPO12 ESPO15 FSPO16 SSPO15 .########### | ESPO24 FSPO15 SSPO2 SSPO24 SSPO6 .#### | ESPO22 ESPO6 ESPO8 FSPO4 SSPO22 WSPO2 WSPO5 50 .############ + SSPO17 SSPO9 WSPO21 WSPO7 .#### | ESPO2 ESPO20 FSPO20 SSPO3 SSPO5 .###### |M ESPO1 ESPO14 ESPO23 SSPO13 SSPO4 WSPO13 WSPO17 .########## | ESPO3 FSPO1 FSPO21 SSPO19 SSPO21 WSPO15 WSPO16 WSPO19 WSPO20 .### | FSPO8 SSPO16 SSPO18 WSPO18 .####### | FSPO10 SSPO14 WSPO12 .#### S| ESPO10 ESPO13 ESPO4 SSPO20 .## | FSPO23 FSPO24 SSPO8 WSPO9 .###### | ESPO5 FSPO12 FSPO13 .## |S ESPO18 FSPO11 FSPO18 SSPO10 WSPO10 40 .## + ESPO17 FSPO2 FSPO22 WSPO22 FSPO7: In what order do a hawk, grass, and rabbit form a food chain in a .## | meadow? .## | FSPO19 WSPO6 WSPO8 .# | FSPO9 WSPO1A. Hawk Grass Rabbit .# | WSPO3 B. Grass Hawk Rabbit . T|T FSPO17 C. Rabbit Grass Hawk . | FSPO7 D. Grass Rabbit Hawk Low Ability . | Easier Items . | Students . | 30 + WSPO11 High Ability Students © 2015 BSCS . | WSPO23 Harder Items .##### | . | FSPO3 SSPO1 SSPO11: Which position shows summer in Earth’s Southern Hemisphere? .###### S| WSPO24 A. A .### |T B. B 60 .###### + ESPO7 WSPO14 C. C ## | ####### | D. D .###### | ESPO16 SSPO23 .####### | ESPO9 FSPO5 .###### |S ESPO19 FSPO14 SSPO12 SSPO7 WSPO4 .######### | ESPO21 FSPO6 SSPO11 .####### M| ESPO11 ESPO12 ESPO15 FSPO16 SSPO15 .########### | ESPO24 FSPO15 SSPO2 SSPO24 SSPO6 .#### | ESPO22 ESPO6 ESPO8 FSPO4 SSPO22 WSPO2 WSPO5 50 .############ + SSPO17 SSPO9 WSPO21 WSPO7 FSPO6: The diagrams below show three food chains. Each food chain uses .#### | ESPO2 ESPO20 FSPO20 SSPO3 SSPO5 100 pounds of green plants as a source of food. In which of the three food .###### |M ESPO1 ESPO14 ESPO23 SSPO13 SSPO4 WSPO13 WSPO17 .########## | ESPO3 FSPO1 FSPO21 SSPO19 SSPO21 WSPO15 WSPO16 WSPO19 WSPO20 chains is the most energy available to people? .### | FSPO8 SSPO16 SSPO18 WSPO18 .####### | FSPO10 SSPO14 WSPO12 .#### S| ESPO10 ESPO13 ESPO4 SSPO20 .## | FSPO23 FSPO24 SSPO8 WSPO9 .###### | ESPO5 FSPO12 FSPO13 .## |S ESPO18 FSPO11 FSPO18 SSPO10 WSPO10 40 .## + ESPO17 FSPO2 FSPO22 WSPO22 .## | ESPO24: Which of the following figures best shows the location of the .## | FSPO19 WSPO6 WSPO8 earth's tectonic plates? .# | FSPO9 WSPO1 .# | WSPO3 . T|T FSPO17 . | FSPO7 Low Ability . | Easier Items . | Students . | 30 + WSPO11 High Ability Students © 2015 BSCS Findings STeLLA Professional Development Program Teacher Science Content Knowledge Teaching Practice Teacher Pedagogical Content Knowledge Student Science Content Knowledge © 2015 BSCS Teacher Effects PD Program Student Outcomes Teacher Learning effect size = 0.66 (teacher effect: B = 4.79, SE = 0.72, p < .001) pretest posttest pretest posttest treatment comparison effect size = 0.68 pretest posttest pretest posttest treatment comparison Teacher Level Student Level © 2015 BSCS Next Steps • Dissemination • Mediation analysis: folding in teacher PCK and teacher practice measures to examine how teacher learning and practice predicts student learning • Moderation analysis with student demographics • Examining which STeLLA strategies were most predictive of student learning • Studies in different contexts • Qualitative analyses • PD leader studies © 2015 BSCS STeLLA is also a line of research Research on Student Ideas in Science STeLLA I ViSTA STeLLA II EMAT RESPeCT ViSTA Plus © 2015 BSCS What is the PD research terrain for the STeLLA line of research? © 2015 BSCS Mapping the PD Research Terrain (Borko, 2004) Phase 1. Existence proof Phase 2. Well‐specified PD program Phase 3. Multiple programs Single program Single program Multiple programs Single site Multiple sites Multiple sites Developers are PD leaders New PD leaders New PD leaders Document teacher learning Document teacher learning Document student and teacher learning © 2015 BSCS STeLLA Line of Research Research on Student Ideas in Science STeLLA I ViSTA STeLLA II EMAT RESPeCT ViSTA Plus © 2015 BSCS Mapping the PD Research Terrain: Existence Proof Phase 1, Borko STeLLA I Existence proof Existence proof Single program Lesson analysis program compared to existing content deepening program of shorter duration Single site Single site (San Gabriel Valley, CA), four study groups Developers are PD leaders Developers are PD leaders Outcomes focus: Teacher learning Outcomes focus: Teacher learning, teaching practice, student learning Quasi‐experimental design © 2015 BSCS STeLLA Line of Research Research on Student Ideas in Science STeLLA I ViSTA STeLLA II EMAT RESPeCT ViSTA Plus © 2015 BSCS Mapping the PD Research Terrain: Testing a well‐specified program Phase 2, Borko Phase 3, Borko STeLLA II Well‐specified program Well‐specified programs Well‐specified program Single program Multiple programs Two programs Multiple sites Multiple sites New sites (Front Range, CO), multiple lesson analysis study groups and content deepening groups New PD leaders New PD leaders New PD leaders (BSCS) Outcomes focus: Teacher learning Outcomes focus: Teacher learning, student learning Outcomes focus: Teacher learning, teacher practice, student learning Experimental design: RCT © 2015 BSCS STeLLA Line of Research Research on Student Ideas in Science STeLLA I ViSTA STeLLA II EMAT RESPeCT ViSTA Plus © 2015 BSCS Mapping the PD Research Terrain: Adapting STeLLA for scalability and sustainability Phase 3, Borko ViSTA Plus, EMAT, RESPeCT Well‐specified programs Adapting a well‐specified program for new audiences and for sustainability Multiple programs Multiple variations of the STeLLA program – variations in both substance and form Multiple sites Multiple sites, multiple audiences New PD leaders New PD leaders (including teacher leaders) Outcomes focus: Teacher learning, student learning Outcomes focus: Teacher learning, teacher practice, student learning Multiple quasi‐experimental studies © 2015 BSCS Mapping the PD Research Terrain: Adapting STeLLA for scalability and sustainability ViSTA Plus EMAT RESPeCT Adapted STeLLA program for work with preservice – first‐year teachers Adapted STeLLA program for online work with HS teachers Adapted STeLLA program for K‐6 teachers and students in a high‐needs urban district Single 2‐year program compared to BaU Single semester course, quasi‐experimental design Single year‐long program compared to BaU Two university sites Online site Pomona Unified School District, CA New and experienced STeLLA PD leaders New PD leaders Teacher leaders as PD leaders Outcomes focus: Teacher learning, teacher practice, student learning Outcomes focus: Teacher Outcomes focus: Teacher learning, teaching practice, learning, student learning student learning © 2015 BSCS STeLLA Line of Research and the Consensus Model of Effective PD How does this line of research build on and contribute to research‐based knowledge about effective professional development? © 2015 BSCS Consensus Model of Effective PD (Yoon et al., 2007; Desimone, 2009; Wilson, 2013) Effective PD … • Focuses on specific subject matter content • Engages teachers in active learning • Is coherent (aligned with teachers’ prior knowledge and beliefs; aligned with reform documents and school policy and practice) • Is of sufficient duration • Involves the collective participation of teachers (all teachers in a school, grade level, or dept.) © 2015 BSCS Research supporting the consensus model • Relies heavily on teacher self‐report • Rarely looks at impact on student learning • And when the research does look at student learning, the results are mixed © 2015 BSCS STeLLA contributes to knowledge about effective PD: Beyond the Consensus Model • The professional development program is guided by a theory of teacher learning. • The program substance is organized around a conceptual framework. • Science content learning is intertwined with analysis of practice. • Analytical tools and videocases support collaborative, deep analysis of science teaching, student learning, and science content. • There is an internal coherence of program form and substance. • Learning is directed and scaffolded by knowledgeable PD leaders. © 2015 BSCS Our Assertions • The consensus model is not enough to guide the design of professional development opportunities that will support NGSS. • The STeLLA line of research contributes to building a research‐based model of effective PD—beyond the consensus model. • More strong lines of research on effective PD are needed. © 2015 BSCS Research BSCS Strategies for Effective Science Teaching: Using the Student Thinking and Science Content Storyline Lenses Lesson Analysis Conceptual Framework STUDENT THINKING Learning to analyze science teaching through two lenses SCIENCE CONTENT STORYLINE © 2015 BSCS The Lenses • What is the Student Thinking Lens and why is it important? • What is the Science Content Storyline Lens and why is it important? © 2015 BSCS © 2015 BSCS Strategies Read the strategy document and complete the graphic organizer about asking questions to elicit, probe, and challenge student thinking. Be prepared to share your chart with a partner. – What is the purpose of each strategy? – What are the key features of each strategy? In your table group, create a chart for your assigned strategy. © 2015 BSCS What are questions that elicit and probe student thinking? • What is the difference between a question that elicits student thinking and a question that probes student thinking? © 2015 BSCS What are questions that probe and challenge student thinking? • What is the difference between a question that probes student thinking and a question that challenges student thinking? • What is a leading question? How is it different from an effective challenge question? © 2015 BSCS Lesson Analysis: The Basics • Viewing Basic #1: Look past the trivial, the little things that “bug” you. • Analysis Basic #1: Focus on student thinking and the science content storyline. • Viewing Basic #2: Avoid the “this doesn’t look like my classroom” trap. • Analysis Basic #2: Look for evidence to support any claims. • Viewing Basic #3: Avoid making snap judgments about the teaching or learning in the classroom you are viewing. • Analysis Basic #3: Look more than once. • Analysis Basic #4: Consider alternative explanations and teaching strategies. © 2015 BSCS Lesson Analysis: Classroom • Identify • Analyze • Reflect © 2015 BSCS STeLLA PD Program and Results • Program substance – Conceptual Framework • Program form – Teacher study group (focused on videocase‐based analysis of practice) • Program results – 0.68 effect size on student learning and 0.66 effect size on teacher learning Given what you’ve seen so far, how might you explain the impact on student and teacher learning? © 2015 BSCS Lesson Analysis: PD Leader • Identify • Analyze: What does the PD leader do (or not do) to make teacher thinking visible? • Reflect © 2015 BSCS Analysis of Practice • What are your key insights about this line of research so far? • What are your questions? © 2015 BSCS Panel Discussion • Theoretical Frameworks and Lines of Research: Kathy Roth • Partnership Development: Paul Numedahl • PD Program and Variations: Connie Hvidsten • Research Methods and Results: Chris Wilson • PD Leader Development: Jody Bintz © 2015 BSCS Closing • What is one take‐away from this session? © 2015 BSCS References • Borko, H. (2004). Professional development and teacher learning: Mapping the terrain. Educational Researcher, 33(8), 3‐15. • Desimone, L. M. (2009). Improving impact studies of teachers’ professional development: Toward better conceptualizations and measures. Educational Researcher, 38(3), 181‐199. • Ruiz‐Primo, M. A., Shavelson, R. J., Hamilton, L., & Klein, S. (2002). Journal of Research in Science Education, 39(5): 369 – 392. • Wilson, S. M. (2013). Professional development for science teachers. Science, 340: 310‐313. • Yoon, K. S., Duncan, T., Lee, S. W. Y., Scarloss, B., & Shapley, K. L. (2007). Reviewing the evidence on how teacher professional development affects student achievement. Washington, DC: National Center for Educational Evaluation and Regional Assistance, Institute of Education Sciences, US Department of Education. © 2015 BSCS Contact Kathy Roth [email protected] Connie Hvidsten [email protected] Chris Wilson [email protected] Paul Numedahl [email protected] Jody Bintz [email protected] Betty Stennett [email protected] Download presentations at bscs.org/sessions. [email protected] This material is based upon work supported, in part, by the National Science Foundation under Grants DRL‐0918277, DRL‐220635, DRL‐1321242, DRL‐1118643. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. © 2015 BSCS
© Copyright 2024