Learning Analytics – Future possibilities and current limitations. Examples from Norway, Denmark and Singapore. Conexus - Release knowledge! Founded in 2001 100 employees' Software and content • • • • • We work in: Norway Denmark Singapore Sweden from 2014 • 250 municipality costumers • The Norwegian ministry of education • The Norwegian Directorate for Education and Training • Norwegian research consul • Innovation Norway Funded by: Concept partners Norway • IMTEC • Gyldendal • Kommuneforlaget (KS) • mYouTime • Microsoft International • KMD Education (Denmark) • Conexus Orient (Singapore) • Orient Software (Vietnam) • Knewton (New York) • IST (Sweden) In Norway • The national government, all students and teachers. • 80% marked • All big publishers Evidence Informed Quality Work (EIQW) The main purpose of the project is to develop and test processes and technology that supports quality work based on updated evidence. Over 30 school and preschool owners participate, together with 100 organizations (preschools, primary schools, upper secondary schools). Content analyzed in the examples, form: • • • • • • • • • • Norwegian government All SAS providers in Norway SSB Gyldendal (publisher) Cappelen (publisher) Aschehoug (publisher) KIKORA National Center for Mathematics National Center for Reading 20 Other content providers Academic feedback from: • • • • • • • Erling Lars Dale Knut Roald Peter Mortimore Louise Stoll Kjell B. Hjertø Andy Hargreaves Viviane Robinson Learning analytics has to develop: • • • • • • • • Learning for all students Professional Capital Assessment for learning Professional learning communities 21 CENTURY capacities Evidence informed processes Student Centered leadership Open To Learn Communication The strategy to implement Learning analytics must: 1. Not only focus on the easy counted information 2. Build capacity and evidence informed processes 3. Improve students and school performance by assessment for learning 4. Be based on trust in the relation between government, school leadership, teachers, students and the community. Learning Analytics – possibilities Learning Analytics – ecosystem “A tool is only as good as the mindset using it” (Fullan 2013) 1. Vision, values and direction 1. Establishing goals and expectations 1.1 Uplifting leadership 1.2 Ethical leadership 1.3 Establishing goals 2. Fostering relationships 2. Evidence-informed practise 2.1 Reflective enquiry 2.2 Research-informed practice 2.3 Data-informed practice 3. Professional learning and development 3.1 Learning environment 3.2 Feedback culture 3.3 Collaborative learning 3.4 Collective responsibility 4 Innovation, improvement and change 4.1 Change processes 4.2 Improvement 4.3 Innovation 4.4 Continuous learning and development 5 Local and global responsibility 5.1 Collaboration with the municipality 5.2 Collaboration between school and home 5.3 Collaboration with other schools 5.4 Global awareness 6 Management 6.1 Information Literacy 6.2 Using School Administration Systems (SAS) 6.3 Communicating and sharing information (LMS) 6.4 Evaluating/monitoring (PULS) 6.5 Tracking and follow up of students (VOKAL) 6.6 Capacity assessment and appraisal systems (KAN) 1.1 Clear Purpose 1.2 Raising expectations 2.1Relationships with students 2.2Relationships between students 2.3Relationships with parents 2.4Relationships with community? 3. Engaging students 3.1 Promote passion for the subject 3.2 Recognising different starting points 3.3 Designing challenging learning programs 3.4 Developing students’ responsibility for learning 3.5 Encouraging well-organized cooperative learning 3.6 Enhancing learning through technology 4. Assessing learning 4.1 Creating success criteria 4.2 Closely monitoring students’ understanding 4.3 Giving and receiving feedback 4.4 Promoting self-assessment 4.5 Promoting peer assessment 4.6 Tracking progress over time 5. Enquiring into your practice 5.1 Reflective enquiry 5.2 Research-informed practice 5.3 Data-informed practice 6. Committing to collective responsibility 6.1 Collaborative learning 6.2 Collective responsibility 7. Stepping up to leadership 8. Innovating and changing 8.1 Experiencing change 8.2 Learning about your learning School leaders need to know • • • • • • • Professional Capital Assessment for learning 21 CENTURY capacities Evidence informed processes Professional learning communities Student Centered leadership OTL Teachers need to know 1. Establishing goals and expectations 1.1 Clear Purpose 1.2 Raising expectations 2. Fostering relationships 2.1Relationships with students 2.2Relationships between students 2.3Relationships with parents 2.4Relationships with community? 3. Engaging students 3.1 Promote passion for the subject 3.2 Recognising different starting points 3.3 Designing challenging learning programs 3.4 Developing students’ responsibility for learning 3.5 Encouraging well-organized cooperative learning 3.6 Enhancing learning through technology 4. Assessing learning 4.1 Creating success criteria 4.2 Closely monitoring students’ understanding 4.3 Giving and receiving feedback 4.4 Promoting self-assessment 4.5 Promoting peer assessment 4.6 Tracking progress over time 5. Enquiring into your practice 5.1 Reflective enquiry 5.2 Research-informed practice 5.3 Data-informed practice 6. Committing to collective responsibility 6.1 Collaborative learning 6.2 Collective responsibility 8. Innovating and changing 8.1 Experiencing change 8.2 Learning about your learning Modell 1 Relation between data MAP Results MAV Support from home Class climat Teacher Self-eff Motivation Learning analytics in Norway Final estimation of fixed effects Fixed Effect For INTRCPT1, β INTRCPT2, γ TRINN_ME, γ K_MILJ4, γ LAERER17, γ For KJONN slope, β INTRCPT2, γ For KARAKT6 slope, β INTRCPT2, γ For VEIL3_11 slope, β INTRCPT2, γ For HJEM3_09 slope, β INTRCPT2, γ Coefficient t -ratio d.f. p -value 0 3,78 169,44 0,03 0,65 0,13 2,53 0,33 4,21 00 01 02 03 115 <0.001 115 0,52 115 0,01 115 <0.001 1 10 0,09 2,58 1757 0,01 0,08 4,33 1757 <0.001 0,19 11,97 1757 <0.001 0,14 8,59 1757 <0.001 2 20 • Individnivå (N=1757/55) – Nr.2. Den nest viktigste faktoren for motivasjonen er vellykkede veiledningssamtaler med rådgiver (t=11.97, β=.19***). Jo bedre veiledning, desto bedre motivasjon. Rådgivningen er den mest signifikante faktoren, men ikke den sterkeste. • 3 30 4 40 Final estimation of fixed effects Fixed Effect Coefficient t -ratio For INTRCPT1, β INTRCPT2, γ 3,78 166,74 TRINN_ME, γ -0,02 -0,39 K_MILJ4, γ 0,20 3,80 LAERER17, γ 0,42 5,84 For KJONN slope, β INTRCPT2, γ 0,10 2,76 For KARAKT6 slope, β INTRCPT2, γ 0,08 4,78 For VEIL3_11 slope, β INTRCPT2, γ 0,20 12,63 TRINN_ME, γ 0,02 0,56 LAERER17, γ 0,10 1,98 For HJEM3_09 slope, β INTRCPT2, γ 0,14 8,84 d.f. p -value 0 00 01 02 03 115 115 115 115 <0.001 0,70 <0.001 <0.001 1755 0,01 1755 <0.001 1755 1755 1755 <0.001 0,58 0,05 1755 <0.001 1 10 2 20 3 30 31 32 4 40 Hvis man tar med interaksjonen mellom trinn og rådgiver, er det en signifikant interaksjonseffekt mellom rådgivers veiledning og tilfredshet med lærerne på motivasjonen (t=1.97; β=.10; p<.05). Jo bedre lærere , desto bedre effekt av individuell veiledning, ut over den direkte effekten mellom veiledning og motivasjon. – Nr.3: Støtte fra hjemmet. Jo bedre hjemmestøtte, desto høyere motivasjon (t=8.59; β=.14***). – Nr.5: Kjønn (t=2.58; β=.09**). Kontrollfaktor. – Nr.6: Karakterer (t=4.43; β=.08***). Kontrollfaktor. • Gruppenivå (N=115) – Nr.1: Viktigste faktor: Lærerne (t=4.21; β=.33***). Lærerne er de som påvirker elevenes motivasjon mest. • NB! Elevene ble ikke spurt om lærernes faglige dyktighet (naturlig nok?). – Nr.4: Klassemiljø (t=2.53; β=.13**). – Nr.7: Klassetrinn betyr ingen ting for motivasjonen (kontrollert for de andre faktorene i modellen). • Konklusjon: Learning analytics in Norway Final estimation of fixed effects Fixed Effect For INTRCPT1, β INTRCPT2, γ Coefficient t -ratio d.f. p -value 0 00 TRINN_ME, γ K_MILJ4, γ LAERER17, γ For KJONN slope, β INTRCPT2, γ For SE8_01 slope, β INTRCPT2, γ For MAV6_03 slope, β INTRCPT2, γ For MAP3_12 slope, β INTRCPT2, γ 01 02 03 3,77 218,71 115 <0.001 0,01 0,17 0,23 0,29 4,51 3,92 115 0,77 115 <0.001 115 <0.001 0,10 3,19 1757 0,00 2 20 0,31 11,96 1757 <0.001 0,11 4,05 1757 <0.001 0,22 10,42 1757 <0.001 3 30 4 40 Final estimation of fixed effects Coefficient t -ratio d.f. p -value 3,78 167,03 -0,02 -0,46 0,20 3,78 0,42 5,98 115 115 115 115 <0.001 0,65 <0.001 <0.001 0,13 3,65 1755 <0.001 0,31 0,10 -0,03 11,28 1755 1,68 1755 -0,43 1755 <0.001 0,09 0,67 0,11 3,99 1755 <0.001 0,22 10,03 1755 <0.001 0 00 01 02 03 1 10 2 20 21 22 3 30 40 – Nr.1. Mestringstro (self-efficacy). Den viktigste faktoren for motivasjonen er mestringstro (t=11.96, β=.31***). Mestringstro er både den sterkest faktoren og den mest signifikante faktoren. • 1 10 Fixed Effect For INTRCPT1, β INTRCPT2, γ TRINN_ME, γ K_MILJ4, γ LAERER17, γ For KJONN slope, β INTRCPT2, γ For SE8_01 slope, β INTRCPT2, γ K_MILJ4, γ LAERER17, γ For MAV6_03 slope, β INTRCPT2, γ For MAP3_12 slope, β INTRCPT2, γ • Individnivå (N=1757/55) 4 Hvis man tar med interaksjonen mellom lærer og mestringstro, er det en marginalt signifikant interaksjonseffekt mellom mestringstro og klassemiljø på motivasjonen (t=1.68; β=.10; p<.10). Jo bedre klasseklima, desto sterkere effekt har mestringstroen på motivasjonen, ut over den direkte effekten av på motivasjon. – Nr.3: Aktiv mestring mot mål (MAP) (t=10.42; β=.22***). – Nr.5: Perfeksjonistisk mestring (MAV) (t=4.05; β=.11***). – Nr.6: Kjønn (t=3.19; β=.10**). Kontrollfaktor. • Gruppenivå (N=115) – Nr.2: Lærerne (t=3.92; β=.23***). Lærerne er de som påvirker elevenes motivasjon mest etter mestringstro. – Nr.4: Klassemiljø (t=4.51; β=.17**). – Nr.6: Klassetrinn betyr ingen ting for motivasjonen (kontrollert for de andre faktorene i modellen). • Konklusjon: Results and progression Learning analytics is about: Selection and focus 1 Input 3 Proses Profil Actions Multi 3rd grade Thegraph shows the amount of money four children spent in a week. Use the graph to answer the questions. How much did Ben spend? Dollar $ Who spent over $ 500? John Eve Alan Kate What is the difference between the one who spent the most and the one that spent the least? 40$ 60$ 80$ 100$ John Previous Next Eve Kate Alan Math <header> Numbers Algebra Data Stats % Meassurement Geometry <header> <name> <updated> <header> The result of a test is measured as a percentage of the norm (national average) for each function or each subtest. A score of 100% means, that the result is identical to the national average. A score of 130% would mean that the student has 30% more right than the national average <header> <level> <esultatet av en test i Kartleggeren måles i prosent av norm (landsgjennomsnittet) for hvert funksjonsområde eller hver deltest. Et resultat på 100 % betyr altså at resultatet er identisk med landsgjennomsnittet. Et resultat på 130 % vil bety at eleven har 30 % mer rett> <header> <level> <esultatet av en test i Kartleggeren måles i prosent av norm (landsgjennomsnittet) for hvert funksjonsområde eller hver deltest. Et resultat på 100 % betyr altså at resultatet er identisk med landsgjennomsnittet. Et resultat på 130 % vil bety at eleven har 30 % mer rett> <header> <level> <esultatet av en test i Kartleggeren måles i prosent av norm (landsgjennomsnittet) for hvert funksjonsområde eller hver deltest. Et resultat på 100 % betyr altså at resultatet er identisk med landsgjennomsnittet. Et resultat på 130 % vil bety at eleven har 30 % mer rett enn landsgjennomsnittet, mens et resultat på 70 % vil tilsvarende bety at eleven har svart rett på 70 % av det en gjennomsnittselev har gjort. Kategori 1: > 80 % av norm. Resultatet indikerer at eleven trolig ikke har behov for tilrettelagt opplæringstilbud på dette> <header> <header> <text> <text> <header> <header> <header> <page> <page> <page> <chapter link> <text> <text> Math New data in cooperation with Knewton Work load Task level Goal % Progression Test correlation Students development tool • • Assessment for learning 21 CENTURY capacities 21 CENTURY capacities What it means The term 21st century capacities refers to a broad set of knowledge, capacities , work habits, and character traits that are believed to be critically important to success in today’s world, particularly in collegiate programs and modern careers. Generally speaking, 21st century capacities can be applied in all academic subject areas, and in all educational, career, and civic settings throughout a student’s life. Do it again Creative thinking 80% Creative thinking is the process we use when we come up with new ideas. creative thinking process can be accidental or deliberate Students overview Students overview Numbers Algebra Math Data ENG Stats % Reading Meassurement Geometry Science Sport Test-correlation Progression Task level Self-efficacy Self-assessment Logged in Start Page Supervision report Management areas School owners report Display Teaching evaluation Selection School report Selection Year My reports NYC 2013 Total report Information Management targets Quality description Export Dotted Updated 06.02.14 Total report Key information School 1 Indicators Parents’ education level Class room management Motivation and coping National tests, 5th grade National tests, 8th grade National tests, 9th grade Primary points Deviation achievement- and examination mark Proportion who have successfylly completed first year of college Grade development from middle school to first year of college School 2 School 3 School 4 Level Quality characteristics - Student Assessment Level Students find that feedback from the teachers is adapted to their prior knowledge. Students know what they have achieved and what 4 Indicators Student survey - Motivation to strive for. They utilize the feedback so the learning environment is - learning and characterized by motivation and student participation. The teachers assessment have a common understanding of what competence the various - professional guidance characters express. - criteria for self Level Students and teachers have a common understanding of the assessment strategies that should be used to achieve good learning. Teachers 3 engage students in assessing their own work. Final assessment is based on the total competence goals. Level The school has a clear focus on assessment, but the feedback is often general and self-assessment is not set in system. Final assessment is 2 to some extent based on the overall competence goals. Level The teachers' practice of continuous assessment is unsystematic. Final assessment is just to a small extent based on the overall 1 competence goals. Level Level 4 Level 3 Level 2 Level 1 Quality characteristics - Organizational Learning The school is involved in learning at all levels. There is a culture of sharing knowledge and exchanging good practice. Colleagues support each other and make each other better. The discussions are characterized by confidence and openness. The school's development is characterized by collaboration and reflection on own practice and analysis of their results. The school is in the forefront of academic and educational development, uses new research and learn from good examples from other schools. The school's educational development follow central planning and control documents. The school's goals and priorities are known to the staff. The school analyzes its own performance and set targets. The school tries to systematize the educational development in relation to central planning and control documents. Individual results are presented for the staff. The school's educational development is characterized by randomness, and the effort of individual initiative. Indicators Personnel survey - Collegial support - Common goals - Learning Press - Extra role behavior • • • • • • • • • • Learning Analytics – current limitations Not all information in the same system (the ecosystem is incomplete) Not enough standards in the ed-tec industry The ed-tec industry is not cooperating The target is not the students learning but the government control need, or researchers joy in finding correlations Researchers focus on algorithms or teaching, not the processes in a ecosystem The governments focus on content and not structures Professional capital and capacity building is not the target for the tools and reports To much data driven control or personalized tasks, not enough evidence based processes and learning to learn. Not enough focus on the markers of future performance and formative assessment (Improve school performance by assessment for learning) Not enough trust in the relation between government, school leadership, teachers, students and the community. Learning Analytics – ecosystem Yngve Lindvig Chief Research and Development Officer Conexus [email protected] www.conexus.no + 47 90785069 Linkedin: http://no.linkedin.com/pub/yngve-lindvig/30/907/47b/
© Copyright 2024