Engineering Education Research
in the AWAKEN Project
Mitchell J. Nathan
University of Wisconsin - Madison
The AWAKEN Project
Funded by the National Science Foundation
Acknowledgements
Co-Investigators
Graduate Students
• L. Allen Phelps
• Sandra Courter
•
•
•
•
•
Amy Prevost
Amy Atwood
Natalie Tran (CSUB)
Kyle Oliver
Benjamin Stein
Funding from NSF Division of Engineering
Education & Centers
AWAKEN
1. Curriculum Response
– In 2006, $1.3B for career & technical education (CTE)
programs in every state in the US.
• National Research Council’s (2007) Rising
Above the Gathering Stormcalls for educational
leaders to optimize knowledge-based resources
and energize the STEM career pipeline.
Project Lead the Way
Competing Hypotheses
• Control for student prior achievement & other
student & teacher characteristics.
• Enriched Integration Hypothesis: higher
standardized test scores in mathematics and
science than the students who are not taking
any PLTW courses
• Insufficient Integration: No advantage
• Adverse integration: Lower for PLTW students
– Curriculum for middle and high school.
– Over 17% of US high schools, in 50 states
– 7 HS courses qualify for credit at accredited colleges.
• Do PLTW students achieve both academic
& occupational competencies?
• Is enrollment in PLTW associated with
enhanced math and science achievement?
M J Nathan
Study Sample #1
– Treatment N = 70 Ss with one or more PLTW class
– Control N = 70 Ss with no PLTW, matched on DV
measures
• Perkins Vocational Education Act mandates that
technical and academic education be integrated
“so that students achieve both academic and
occupational competencies."
AWAKEN
• NRC: PLTW as “a model curriculum.”
• Project Lead the Way (PLTW):
• US urban district: African American 57%,
Hispanics 22%, White 12%, Asian 4%, other 4%
• Free/Reduced Lunch Program 72%
• All high schools certified in PLTW (N = 5)
• Level 1: Students (N = 140)
Clarion Call for Education Reform
AWAKEN
Data & Design
• Multi-level analysis
• Dependent Variable:
– Grade 10 achievement in Math & Science
• Independent Variables:
– Level 1 Student: Race, Sex, F/R Lunch, Special Ed
– Covariate: Grade 8 Math & Science achievement
– Level 2 Classroom: Teacher Experience
• Level 2: Classroom (N = 27)
• Correlational study: No random assignment, so
causal inferences are not supported
AWAKEN
AWAKEN
– Selection bias reduced with propensity score matching
* Working with District Data *
• Confidentiality issues
– “Cooperative agreement” with district
– School board approval
• Technical issues
– Size of data set
– Merging across databases
– Missing data & quality control: BIG problems
– Massive time commitment from technician
• Giving back to the district
Selection Bias in Control Group
• Propensity score matching (PSM) technique
• PSM: conditional probability of assignment to
treatment given a set of observable covariates.
• Propensity scores from the group of students with
complete data (N = 772).
• Matched prior achievement
PLTW
Control-PSM
Math
508.7
509.19
Science
368.8
372.5
• Match on free/reduced lunch program, gender.
Data Analysis: Level 1
• Achievement = β0 +
β1Prior Achievement +
β2Female +
β3Free/Reduced Lunch +
β4PLTW +
R
• Predictors were grand-mean centered
• β0: Grand mean of student achievement
Data Analysis: Level 2
•
•
•
•
•
β0 = γ00 + γ01Years_Experience + U0
β1 = γ10
β2 = γ20
β3 = γ30
β4 = γ40
• Classroom mean achievement regressed on
teacher years of experience and the classroom
residual variance (U0).
• Slopes for Level 1 factors (β1- β4) treated as fixed
Results
Results
• For group (N = 140) significant achievement
gains from 8th grade to 10th grade in math
(p< 0.01) and science (p< 0.01) (Paired t).
• Moderately high correlations between 8th
grade and 10th grade achievement (0.73
math, 0.77 for science)
• Variation at the classroom (teacher) level.
• 21% (math) to 31% (sci) of σ2 in student
achievement at classroom level.
• However, after controlling for student prior
achievement and student characteristics,
decreases to about 9% to 8%.
• Gains in math overall (8th to 10th), p< .05,
• But lower for PLTW, p< .05.
AWAKEN
AWAKEN
AWAKEN
Results: Math Achievement
Results: Math Achievement
Results: Science Achievement
Results: Achievement
• Controlling for student and teacher DVs,
PLTW showed a (relative) mean
decreaseof 10.76 points in 10th grade
scores, compared to non-PLTW students.
• No difference between students enrolled in
one or more than one PLTW class (p = 1).
• Gains in science overall.
• Lower for PLTW students, but n.s.
• Smaller gains in Math achievement for PLTW
students contradicts the Enriched Integration
Hypothesis that PLTW enrollment contributes to
higher math achievement.
• Math: Most direct support for the model that
follows from the Adverse Integration Hypothesis.
• Science: Most direct support for the model
derived from Insufficient Integration Hypothesis.
AWAKEN
AWAKEN
AWAKEN
Study #2: Replication
• Sample: 70% White, 24% F/R Lunch, 2%
ELL; city rated highest rate of PhDs in US
• Multiple regression analysis (N = 176), due
to lack of variability across classrooms
• Factors: prior achievement, free/reduced
lunch eligibility, special ed& gender.
• PLTW (n=57) and non-PLTW (n=119)
matched using PSM; only gender differed.
Results: Math
• Gains in math overall, p< .01
• PLTW is sig. predictor, p = .05.
• Along with F/R L, p< .01
Results: Science
• Gains in science overall, p< .01
• PLTW not a sig. predictor of achievement
AWAKEN
Conclusions: Achievement
• For math, enriched integration hypothesis
– Opposite of findings from low-SES sample!
– Different analytical models were applied
• In science, insufficient integration fits best
• Need to explore relation of PLTW to math
across a broader sample.
• Data do not support policies to use PLTW
and other pre-engineering courses to
satisfy science graduation requirements.
•
•
•
•
Correlational – no random assignment
Small-ish sample sizes (missing data)
Studies limited to Midwest region (tho’ vary SES)
Curriculum is more than which books you use
– Teacher training: pedagogy and content knowledge
– Classroom learning and instruction
– Assessment design and match to curriculum
AWAKEN
Match to State Assessment
10th
• Reviewed all released
grade items on
state achievement assessments.
• Standardized achievement tests align
most closely with academic course
material and content standards.
• These items may fail to capture important
aspects of engineering preparation.
• Could explain null result, but not the
significantly smaller gains in study #1.
Insufficient and Adverse
Integration
Limitations
• Why do PLTW students show mixed, low, or no
achievement gains?
1.Match of the course to achievement tests.
– Want a test that control Ss took pre / post.
2.Lack of math in the engineering courses.
3.Lack of explicit connections between the
math and the engineering content.
AWAKEN
Math Content
“Explicit Integration” of Math
• How much math is present in engineering
curriculum?
• What is the level of math that is used?
• Is the math explicitly connected for the
student to the engineering activities?
• Places in the curriculum where materials
and instruction specifically point to a
mathematics principle, law, or formula,
and depict how it is used to carry out or
understand an engineering concept, task
or skill.
AWAKEN
AWAKEN
Transfer
The Manifold Curriculum
• When assessment items share both
surface features and deep structure with
training tasks, “near transfer” is supported.
• When surface features & context change
(“far transfer”), students fail to notice
matches with deep structure (CTGV, 1997).
• To develop transfer, novices require
explicit instruction, frequent practice and
timely feedback (Pellegrino et al. 2001).
• Curriculum analyses (Porter, 2004) can be
divided into the study of 4 different aspects:
– Intended curriculum
– Assessed curriculum
– Enacted curriculum
– Learned curriculum
Samplefrom IED Unit 6, for Algebra standard
Area
No. Scored
Items
(X)
Opps. for
Explicit
Integration
(N)
%
Integration
Teacher
Training
0
26
0
Planning
3
25
12%
Activities
2
5
40%
Assessment
1
4
25%
– Planning materials:
• Anticipatory set;
• Concepts and performance objectives;
• Daily lesson plans and planned presentations.
– Classroom activities:
• Hands-on projects;
• Worksheets.
– Teacher training materials:
• Training documents; Activities; Projects;
• Self-Assessment and Self-Reflection Items.
AWAKEN
Intended Curriculum: Method
Intended Curriculum
AWAKEN
Intended Curriculum: Results
Enacted Curriculum
Student Materials
• Year 1 Math content is light, limited to a few standards
(geometry, number & operations, meas), & poorly integrated
• Years 2 & 3, more math, more advanced math, and much
better explicit integration with engineering
Teacher Training Materials
• Years 1 & 2 content standards: 31% or less integtation.
– Content: Geometry, Number, Measurement
• Year 3 (digital el.) 83% explicit integration.
– Content: Number and Algebra content; prop. calculus
– Process: Representation and Connections
AWAKEN
Enacted Curriculum: Results
• Analyzed 4 lessons from Intro course (4 more!)
• Overall, 34 math concepts (content standards)
were identified
– Geometry (17), Measurement (7), Number &
Operations (10).
• Less than 33% of math concepts integrated.
– Often, very basic ideas (parts of geometric objects, or
number operations).
– Advanced ideas (projective geometry, set theory)
were most often implicitly embedded within
procedures and software operations.
Assessed Curriculum
• Data: Formative, summative assessments:
– Projects;
– Presentations given by students; and
– Written examinations.
Q1. Do assessments allow students to
demonstrate connections of math (and
science) concepts to engineering?
Q2. Is there alignment between the intended
curriculum and PLTWs own assessments?
Assessed Curriculum: Results
•
–
–
–
IED: Better than the other foundations courses!
Content: Geometry (29%) & Measure (17%)
Process: 21% for Repn and Commun.
Disconnect of class (5%) & assessments (29%)
• POE: Integrated activities but not assessed
– Disconnect with assessment the other way!
• DE: Not much overlap with math standards
– Assessments down from 8% (Algebra) to 0%
(Measurement, and Data/Probability).
Insufficient and Adverse
Curriculum Integration
Conclusions: Curriculum
• Students show little benefit in math achievement
from limited & embedded experiences.
• Assessments tend to favor academic focus.
• Even PLTW’s own assessments not well aligned.
• Math is present but limited content and basic
demands, especially in Year 1.
• More math content in later years, but smaller
enrollment in advanced engineering courses.
• Math is usually implicitly embedded in software,
instrumentation, instruction & printed materials.
• Lack of explicit connections between academic
and technical courses may reinforce the different
skills and knowledge valued in Tech Ed vs.
college preparation (high-stakes assessments).
• May feed attitudes counter to math and science
learning, and expect that the technology will or
should do the math for them.
• Especially problematic for students already
exhibiting low math achievement.
Next Steps: Curriculum
• Consider assessments that better align with
expectations of the math (and science) in
engineering and technical education.
• Many math content standards go untapped in
engineering.
• Examine impact of explicit integration in
curriculum design and teacher training (Stone 2008).
– Professional development teams
– Curriculum redesign teams (with kibitzing)
AWAKEN
End of Part 1
www.engr.wisc.edu/services/elc/hple
ngr.htm
2. PedagogicalResponse
• What are teachers thinking, anyway?
Aim #2: Teacher Beliefs
Student factors influence teacher decisions
How beliefs change with PLTW instruction
• Instructional practice and teacher decision
making are influenced by teachers’ beliefs
about learning and instruction.
• For EEd reform, necessary to study and
incorporate teachers' attitudes and beliefs
about engineering instruction and learning.
• JEE editorial: Need to understand the
“engineering teaching culture.”
Aim #2: Teacher Beliefs
Method
Method: Likert Scale
1. Develop an instrument that identifies a
reliable set of constructs
2. Document teachers’ beliefs & expectations
3. * Identify differences among teachers
• Engineering Education Beliefs and
Expectations Instrument – EEBEI
• On-line survey, 84 forced-choice items
• Spring 2008; Participants offered $10
• N = 144 high school teachers from Midwest
• White (93%), Male (58%)
• Science (65%), Math (41%), Tech Ed (36%)
• What is the nature of high school
teachers’ beliefs and expectations about
preparation for engineering study and
career success?
Mitchell J. Nathan
University of Wisconsin - Madison
The AWAKEN Project
Funded by the National Science Foundation
– Technical education v. traditional college
preparation in math and science (MS)
4. * Document changes that occur with PLTW
training and instruction
AWAKEN
• Frequency items (5-point scale)
– The math content taught in my courses is explicitly
connected to engineering.
– 1 (Never) 2 (Almost Never) 3 (Sometimes) 4
(Often) 5 (Almost Always)
• Agreement items (7-point scale)
– To be an engineer a student must have high overall
academic achievement.
– 1 (Strongly disagree) 2 (Disagree) 3 (Somewhat
disagree) 4 (Neutral)
– 5 (Somewhat agree) 6 (Agree) 7 (Strongly agree)
Results: Likert Scale
Conclusions: Likert Scale
M
α
+
.70
B. Student background, interests influence my instruction.
0
.83
C. Student science/math/technical learning takes place in
out-of-school contexts.
++
.78
D. Students need high academic achievement in math
and science courses to be an engineer.
++
.83
E. Student social background affects pursuit of
engineering.
+
.80
F. Science and math content taught in my courses is
explicitly connected to engineering.
+
.92
G. Schools provide adequate environmental and
structural support for engineering.
-
.78
A. Student academic abilities influence my instruction.
Results: Recommend Enroll
Teachers believe that -• Students’ interests, culture & family, and academic
performance guide their instruction.
• Engineering prep. takes place in multiple academic
courses and community settings.
• Eng. students need high academic achievement.
• Being male, white or Asian, child of an engineer
increases likelihood of entering engineering.
• A minority of teachers report they adequately
integrate math & science with engineering
activities.
Results: Recommend Enroll
Method: Vignettes
Academic
V1
Gender: Male
Grade: 10th
Background: low SES
GPA: 3.85
Interests: To enroll in
Principles of Engineering
course; attend college.
Social
V2
Gender: Female
Grade: 11th
Background: high SES
GPA: 3.45
Interests: To enroll in Digital
Electronics course; thinks
father’s work as an engineer
is “cool.”
V3
Gender: Male
Grade: 10th
Background: low SES
GPA: 1.35
Interests: Assembling body kits on
foreign cars; attend college. V4
V4
Gender: Female
Grade: 11th
Background: low SES
GPA: 3.45
Interests: To enroll in Digital
Electronics course; uninterested in
her parents’ blue-collar jobs.
Results: Decision Influences
• Compare academic: low-SES males (V1 v. V3)
Significantly more teachers recommend
enrollment for V1 (hi GPA) than V3
x2McNemar(1) = 31.24, p = .000.
• Compare social: Hi GPA females (V2 v. V4)
Teachers were more likely to recommend
enrollment for V2 (high SES) than V4
x2McNemar(1) = 6.667, p = .001
M J Nathan
M J Nathan
Results: Decision Influences
• Varied use of academic (GPA) & social (SES)
factors
• Academic factors, p = .000
– High GPA (V1), 75% report using academic.
– Low GPA (V3), 20-30% report using academic.
• Social factors, p = .000
– 0% report using SES to recommend enrollment
– For the more privileged student (V2), 50-73% reported
using academic factors
– For low SES (V4), 25-42% teachers reported using
academic factors.
Conclusions: Vignettes
Teacher Differences
• More situated, tacit assessment of teacher views
• When explicitly asked, (SES) was never a factor
in teachers’ decisions, family background was
somewhat, and academic performance was
frequently cited.
• But, when comparing vignettes --
• Compare MS (n = 97) v. PLTW (n = 47).
• MS less likely to identify sources of support
for engineering in their schools (G), p<.01.
• MS agreed more strongly an engineer
needs high achievement in math, science
and technology (D), p<.0001.
• PLTW more likely to claim that science and
math content was integrated with the
engineering content (F) , p<.05.
– Academic factors were applied unevenly across SES
– Social factors influenced teacher decisions, even for
students with comparable (high) academic records.
AWAKEN
Teacher Change
• Before/after intensive summer institute &
initial semester (fall) of PLTW instruction.
• National sample (N=84): 87% white 60% male
• Positive changes in 3 constructs (n=47):
1.More explicitly connect science & math to
engineering (F), p<.01
2.Greater support from their schools (G), p<.01
3.Instruction is more responsive to students’
academic abilities (A), p<.05
Conclusions: Teacher Beliefs
• Developed a reliable instrument to
document teachers’ views, identify group
differences, and track changes in beliefs.
• Influences of student SES and academics
• Teachers may be over-confident in their
integration of math & science with
engineering
– And PLTW training may amplify this
AWAKEN
Limitations
• Teachers’ notions of “engineering”
– Use with another instrument developed by
Yasar, Baker et al. (2006)
• Teachers’ influences on student coursetaking and career pursuits
– Guidance counselors’ views
• Order effects of vignettes
• Expand to a national, longitudinal sample
AWAKEN
Study Sample
www.engr.wisc.edu/services/elc/hple
ngr.htm
Assessed Curriculum: Results
•
–
–
–
IED: Better than any of the other areas of analysis
Content: Geometry (29%) & Measure (17%)
Process: 21% for Repn and Commun.
Disconnect of class (5%) & assessments (29%)
• POE: Integrated activities but not assessed
– Disconnect with assessment the other way!
• DE
Mitchell J. Nathan
– Assessments down from 8% (Algebra) to 0%
(Measurement, and Data/Probability).
University of Wisconsin - Madison
The AWAKEN Project
Funded by the National Science Foundation
AWAKEN
Enacted Curriculum: Method
• Videos segmented into clips.
• Coding system applied to each clip:
– Instruction time codes
• Lecture, tutorial, class management, non-interact
– Concepts (“big ideas”); whether math concepts
are explicitly integrated during instruction.
• Engineering or Math
– Skills address process-oriented activity.
• Engineering or Math
AWAKEN
Enacted Curriculum: Results
Instructor’s time mostly on class management
AWAKEN
Enacted Curriculum: Results
• About three times more instructor contact
time was devoted to skills (73 mins, 93
clips) than concepts (26 mins, 48 clips).
• 64% clips coded as skills but not concepts.
• 31% clips coded as concepts but not skills.
• Thus, concepts generally grounded in
skills, but skills were more likely to be
taught with no conceptual basis.
AWAKEN
Enacted Curriculum: Results
• Topically, when instruction on engineering,
skills (101 clips) were emphasized over
concepts (22 clips), and occupied more
time.
• When focus was on math, concept codes
(34 clips) were more frequent than skills
codes (4 clips), and occupied more time.
AWAKEN
Enacted Curriculum: Results
(1) Instructor’s time mostly on class
management (non-instructional) tasks.
(2) Overall, most observed instruction time
was devoted to skills than to concepts.
(3) Math concepts vs. engineering skills.
(4) Only a small fraction of instruction that
linked math concepts to engineering
coursework made those links explicit,
usually limited to basic topics in math.