1. No category

How to Define, Develop and Deliver an
Ability-Based Metric System for
Student Development
Continuous Curriculum Improvement
Larry F. Hanneman, Steven K. Mickelson,
Thomas J. Brumm and Kristen P. Constant
Iowa State University
Best Assessment Processes IX
Workshop 82
Rose-Hulman Institute of Technology
April 12-14, 2007
Outcome for Today:
“Understanding the System”
¾Ability-Based
Outcomes
Assessment
¾Student
Development
¾Continuous
Curriculum
Improvement
¾Accreditation
Outcome for Today:
“Understanding the Value of System
Thinking”
¾ Seeing the
“Big Picture”
¾ How to Visualize
Meaning
¾ How to get to
Understanding and
Action
¾ Adding Value Well
Beyond
Accreditation
Outcome for Today:
“ System Thinking and Connecting Themes”
¾ Language,
Language,
Language!
¾ “Say it with in
context!”
¾ Reducing the
Faculty Burden
¾ Involving
Constituents
Agenda
¾ Introduction to Ability-Based Outcomes
Assessment: “The Design Concepts”
Concepts
¾ Development of Quantitative Ability-Based
Metrics - Engaging Constituents
¾ Validation and “What we learned along the
way!”
¾ Collecting and Analyzing Ability-Based
Assessment Data
¾ Creating Understanding:
¾ “What to improve,”
¾ “How to improve,” and
¾ “Where to do it.”
ISU College of Engineering
Ability-Based Outcomes
Assessment Project
¾ Design Concepts
¾ Development
Process
¾ Product
¾ Validation
¾ Implementation
¾ Analysis & Continuous Improvement
Design Concepts
ABET Criteria 2000
[“Say it With in Context” 2005 ABET National Meeting]
“To be considered for accreditation,
engineering programs must prepare
graduates for the practice of
engineering at the professional
level.”
Design Concepts
ABET Criteria 2000
“…requires that each program
measure what it wants its
graduates to know and be able to
do - measurements much more
relevant to actual learning.”
“How Do You Measure Success?” ASEE Professional Books,1998
Design Concepts
ABET Criterion 3 (a-k)
“…the assessment process should include
direct and indirect measures…”
“Student self-assessment, opinion surveys,
and course grades are not, by themselves
or collectively, acceptable methods of
documenting achievement of outcomes.”
Design Concepts
ABET Criterion 3 (a-k)
“Generally not acceptable:
¾
¾
¾
¾
¾
¾
Using course mappings as an indicator of student learning.
No direct measures,
Inefficient process,
Inordinate faculty program assessment load,
Faculty not involved in the decision making process,
“traditional” course evaluations and student satisfaction
surveys...”
“Course grades do not constitute measurement of
outcomes.”
Design Concepts
ABET Criterion 3
“Engineering programs must
demonstrate that their graduates
have:”
ABET Criteria 2000
Criterion 3. Program Outcomes and Assessment
Engineering programs must
have:
(a)
(b)
(c)
(d)
(e)
demonstrate that their graduates
an ability to apply knowledge of mathematics, science and engineering
an ability to design, conduct experiments,as well as analyze and interpret data
an ability to design a system,component or process to meet desired needs
an ability to function on multidisciplinary teams
an ability to identify, formulate and solve engineering problems
(f) an understanding of professional and ethical responsibility
(g) an ability to communicate effectively
(h) the broad education necessary to understand the impact of engineering solutions in a
global and social context
(i) a recognition of the need for, and an
(j) a knowledge of contemporary issues
(k)
an ability
ability
to engage in lifelong learning
to use the techniques, skills, and modern engineering tools necessary for
engineering practice
ABET Criteria 2000
How to Measure?
George D. Peterson:
Executive Director, ABET
“Establishing measurable objectives
and evaluating their outcomes are
sophisticated activities with which
most engineering educators have had
little or no experience.”
“How Do You Measure Success?” ASEE Professional Books,1998
Engineering
Knowledge
Necessary
BUT
Not
Sufficient!
ABET Criteria 2000
What to Measure?
Language, Language, Language
ISU College of Engineering
Ability-Based Outcomes
Assessment Project
¾ Design Concepts
¾ Development
Process
¾ Product
¾ Validation
¾ Implementation
¾ Analysis & Continuous Improvement
“an ability to…”
¾ A complex Ability cannot be observed
directly; it must be inferred from performance.
¾ Abilities are complex combinations of
Competencies.
¾ Competencies are the application of behavior
and motivation to knowledge, understanding
and skill.
¾ Key Actions that demonstrate workplace
competencies can be observed and measured.
Linking ABET Criterion 3 Outcomes to Competencies
ISU-DDI Plan
ABET Outcome Criteria
“Critical Incident” Data Collection
Alumni and
Students
ISU and International
Partnering Engineering Faculty
Categorize/Link Behaviors
into Competencies
Link Competencies to Outcome
Criteria
Engineers
within Industry
Exercise:
A Critical Incident Story
Situation/
Task
Actions
Results
STAR
ISU College of Engineering
Ability-Based Outcomes
Assessment Project
¾ Design Concepts
¾ Development
Process
¾ Product
¾ Validation
¾ Implementation
¾ Analysis & Continuous Improvement
ISU Competencies
¾ Engineering Knowledge
¾ General Knowledge
¾ Continuous Learning
¾ Quality Orientation
¾ Initiative
¾ Innovation
¾ Cultural Adaptability
¾ Analysis & Judgement
¾ Planning
¾ Communication
¾ Teamwork
¾ Integrity
¾ Professional Impact
¾ Customer Focus
Yoda: “There is yet another!”
ISU Competencies
¾Competency Definition
¾Clear
¾Independent
¾Specific (Vision/Mission)
¾Key Actions
¾Observable
¾Measurable!
Communication
Clearly conveying information and ideas through a variety of media to
individuals or groups in a manner that engages the audience and helps
them understand and retain the message.
Key Actions
¾ Organizes the communication—Clarifies purpose and importance;
stresses major points; follows a logical sequence.
¾ Maintains audience attention—Keeps the audience engaged through use
of techniques such as analogies, illustrations, body languages, and voice
inflection.
¾ Adjusts to the audience—Frames message in line with audience
experience, background, and expectations; uses terms, examples, and
analogies that are meaningful to the audience.
¾ Ensures understanding—Seeks input from audience; checks
understanding; presents message in different ways to enhance
understanding.
¾ Adheres to accepted conventions—Uses syntax, pace, volume, diction,
and mechanics appropriate to the media being used.
¾ Comprehends communication from others—Attends to messages from
others; correctly interprets messages and responds appropriately.
Exercise
Review, Discuss Report
¾ Take out the “ABET Outcomes vs. ISU
Competencies Matrix”
¾ Sum the number of ISU Competencies
for each ABET Outcome (Horizontally)
¾ Sum the number of ABET Outcomes for
each ISU Competency (Vertically)
What Do You See?
¾Horizontally?
¾Vertically?
¾Most Complex
Outcome?
¾Most
Leveraged
Competency?
ISU College of Engineering
Ability-Based Outcomes
Assessment Project
¾ Design Concepts
¾ Development
Process
¾ Product
¾ Validation
¾ Implementation
¾ Analysis & Continuous Improvement
More Exercise
Assess, Report
How important is
each Competency
and its Key
Actions to the
demonstration of
the linked ABET
(a-k) Outcomes?
Probability of
developing and
demonstrating Key
Actions in various
settings?
4.8
3.8
4.4
3.6 4.3 3.7
4.0
4.5 4.1
3.4
3.4
4.4
3.8 4.1 3.9
4.3 3.0
4.5 4.2 4.0 3.8
4.2
3.5
4.3
4.0
4.7
3.8 3.9 4.1
3.8
3.6 3.3
3.8
©2001 Development Dimensions international, Inc.
4.4
3.7
3.6
3.7 3.6
4.7
4.9
3.7
4.6
4.3
4.2
3.7 3.5
3.4 3.9 3.9
3.7
4.3 3.6 3.8 4.7 4.9 4.3 3.9
4.2
4.0
4.1 3.5
4.1
3.8
3.8 3.1
4.2 3.6 3.7
2.6 4.0
08/05/2005
Opportunity to Develop and Demonstrate:
Communication
90%
80%
70%
10%
0%
Activities-non Prof
20%
Activities-Prof
30%
Capstone Design
40%
Laboratory
50%
Classroom
60%
Coop/Intern
¾ Organizes the
communication …
¾ Maintains audience
attention …
¾ Adjusts to the audience …
¾ Ensures understanding …
¾ Adheres to accepted
conventions …
¾ Comprehends
communication from
others…
100%
Engr. Workplace
¾Key Actions:
Break Time
When we come back:
“Implementation”
“We should design the
curriculum as a set of
interrelated courses and
experiences that will help
students achieve the intended
outcomes.”
Mary E. Huba and Jann E. Freed, “Learner-Centered Assessment on College Campuses,” Allyn and
Bacon, 2000
40%
30%
20%
10%
General Knowledge
Engr Coop/Intern
Classroom Traditional
Customer Focus
Professional Impact
Integrity
Teamwork
Communication
Planning
Analysis & Judgement
Cultural Adaptability
Innovation
Initiative
Quality Orientation
Continuous Learning
0%
Engineering Knowledge
100%
Opportunity to Develop and Demonstrate:
Coop/Intern Workplace vs Classroom
90%
80%
70%
60%
50%
The Value of Engineering
Experiential Education
¾ Spring 1996 through
¾ In-Profession
Spring 2006:
Placement At
Graduation
¾ 8051 BS Graduates:
¾ 90%
¾ 572 Coops
¾ 82%
¾ 1026 Interns
¾ 76%
¾ 2817 Engr. Summer
¾ 53%
¾ 1648 No Engr. Work
¾ 1063 No Ex. Ed. Data
ISU College of Engineering
Ability-Based Outcomes
Assessment Project
¾ Design Concepts
¾ Development
Process
¾ Product
¾ Validation
¾ Implementation
¾ Analysis & Continuous Improvement
OPAL(TM) ABET-Aligned Survey
Implementation
ABET Criterion 3 (a-k)
“Evidence must be directly linked to the
specific outcome being assessed.”
“Data collection methods…can include such
things as: performance evaluation of
work/study, intern or co-ops; and/or
performance observations.”
Implementation and the
Constituents
Semester Data Collection
¾ Web-based delivery
¾ 150-250 Students
¾ All 10 Programs
¾ 150-250 Supervisors
¾ 80-150 Companies
Iowa State University
College of Engineering
Employer
EngagementAssessment:
¾ 250,000 Measurements
to-date
¾ 125,000 Supervisor’s
¾ 125,000 Student’s-Self
¾ 500+ Employer supervisors
assess students each year
¾ 200+ Employers/year
¾ Students from 10 ABETaccredited degree programs
O
R
E
Z
y
t
l
u
c
a
F
r
u
B
!
n
e
d
Standard Program Report
Analysis and the Constituents
¾ Semester Detailed
Reports
¾ Competency
¾ Key Action
¾ Constituents
¾ Program and College
¾ Curriculum Committees
¾ Advisory Boards
¾ Coop/Intern Task Group
¾ Coop/Intern Advisory Board
¾ Student Learning Task Force
ISU College of Engineering
Ability-Based Outcomes
Assessment Project
¾ Design Concepts
¾ Development
Process
¾ Product
¾ Validation
¾ Implementation
¾ Analysis & Continuous Improvement
Experiential Education Assessment
Data Analysis and Interpretation
¾Repeatability
¾Comparing Fall
and Spring Data
¾ What We Are
Learning
¾Communication
¾Teamwork
¾“How to
Improve”
3.80
3.60
3.40
3.20
Student
Supervisor
Teamwork
Quality Orientation
Professional Impact
Planning
Integrity
Innovation
Initiative
General Knowledge
Engineering Knowledge
Customer Focus
Cultural Adaptability
Continuous Learning
Communication
3.00
Analysis & Judgement
Fall 2001 Student & Supervisor
Assessment Comparisons
5.00
4.80
4.60
4.40
4.20
4.00
3.80
3.60
3.40
3.20
3.00
Student Fall
Student Spring
Teamwork
Quality Orientation
Professional Impact
Planning
Integrity
Innovation
Initiative
General Knowledge
Engineering Knowledge
Customer Focus
Cultural Adaptability
Continuous Learning
Communication
Analysis & Judgement
5.00
Spring 2002 Student & Supervisor
Assessment Comparisons
4.80
4.60
4.40
4.20
4.00
3.80
3.60
3.40
3.20
3.00
Student Fall
Student Spring
Teamwork
Quality Orientation
Professional Impact
Planning
Integrity
Innovation
Initiative
General Knowledge
Engineering Knowledge
Customer Focus
Cultural Adaptability
Continuous Learning
Communication
Analysis & Judgement
Fall vs Spring Student
Self-Assessment Comparisons
5.00
4.80
4.60
4.40
4.20
4.00
Design Concepts
ABET Criterion 3
“Each program must have an assessment
process with documented results. Evidence
must be given that the results are applied
to the further development and
improvement of the program.
The assessment process must demonstrate
that the outcomes of the program….are
being measured.”
3.60
3.40
3.20
3.00
Competency
Teamwork
Safety Awareness
Quality Orientation
Professional Impact
Planning
Integrity
Innovation
Initiative
General Knowledge
Supervisor
Engineering Knowledge
Customer Focus
Cultural Adaptability
Continuous Learning
Communication
Analysis & Judgment
Demonstration Level
5.00
College of Engineering Competency Demonstration
Experiential Education Workplace 2001-2005
Self
4.80
4.60
4.40
4.20
4.00
3.80
4.8
3.8
4.4
3.6 4.3 3.7 4.0
3.4
3.4
4.4
3.8 4.1 3.9 4.3 3.0 4.5 4.2 4.0 3.8
4.2
3.5
4.0
4.7
3.8 3.9 4.1 4.2
3.8 3.6 3.3
3.8
4.5 4.1
4.3 3.6 3.8 4.7 4.9 4.3 3.9 3.7
4.4
4.6
4.7
4.9
3.7
3.6
3.7 3.6
3.7 3.5
3.4 3.9 3.9
4.2 4.0
4.1 3.5
4.1
3.7 3.8
4.3
4.3
4.2 3.6 3.7
3.8 3.1
2.6 4.0
01/18/2002
ABET Criterion 3 (a-k) Program Outcomes
COE Attainment of Criterion 3 Program Outcomes
Experiential Education Workplace 2001-2005
(a)
Supervisor
Student
(b)
Supervisor
Student
(c)
Supervisor
Student
Supervisor
Student
(d)
Supervisor
Student
(e)
Supervisor
Student
(f)
Supervisor
Student
(g)
Supervisor
Student
(h)
(i)
Supervisor
Student
(j)
Supervisor
Student
Supervisor
Student
(k)
70%
1414 Supervisors
1624 Students
75%
80%
85%
90%
% Demonstated Acheivement of Program Outcomes
95%
100%
College Top 5/Bottom 5 Competencies
Experiential Education Workplace 2001-2005
Student Assessment
Top 5
Supervisor Assessment
Integrity
1
Integrity
Cultural Adaptability
2
Quality Orientation
Quality Orientation
3
Professional Impact
Teamwork
4
Cultural Adaptability
Professional Impact
5
Teamwork
Student Assessment
Bottom 5 Supervisor Assessment
General Knowledge
10
Analysis and Judgment
Communication
11
General Knowledge
Initiative
12
Customer Focus
Customer Focus
13
Communication
Innovation
14
Innovation
Exercise
Review, Discuss, Report
What do these data
say about Outcome (g):
“an ability to
communicate effectively”?
What do these data
say about Outcome
(d):
“an ability to function
on multidisciplinary
teams”?
an ability to communicate effectively
X
X
X
X
“Insanity is doing the same
things over and over and
expecting different results.”
A. Einstein
20%
10%
0%
100%
100%
100%
90%
90%
90%
80%
80%
80%
70%
70%
70%
60%
60%
60%
50%
50%
50%
40%
40%
40%
30%
30%
E x tra cu rr N o n - E n g r
E x tra cu rr E n g r
C la s s ro o m C a p s to n e
C la s s ro o m L a b
C la s s ro o m T r
F u ll-T im e E m p lo y
E n g r C o o p /In te rn
20%
10%
0%
E x tracu rr N o n -E n g r
E x tracu rr E n g r
C las s ro o m C ap s to n e
C las s ro o m L ab o rato ry
C las s ro o m T rad itio n al
F u ll-T im e E m p lo y
20%
10%
0%
30%
E x tr a c u r r N o n - E n g r
E x tr a c u r r E n g r
C la s s r o o m C a p s to n e
C la s s r o o m L a b o r a to r y
C la s s r o o m T r a d itio n a l
Communication
F u ll- T im e E m p lo y
E n g r C o o p /I n te r n
Customer Focus
E n g r C o o p /In tern
Opportunity to Develop
and Demonstrate
Initiative
X XX
an ability to function on a multidisciplinary team
X X XX X
“Futility is doing something
well that doesn’t need to
be done at all.”
Unknown
ABET Criterion 3 (a-k)
“First, programs must have in place a
curriculum, or other program
requirements, that provide students
opportunities to learn, practice and
demonstrate the elements listed in (a)
through (k).”
“The knowledge, skills and abilities
that students achieve at the end of
their programs are affected by how
well courses, and other
experiences fit together and build
on each other throughout the
undergraduate years.”
Mary E. Huba and Jann E. Freed, “Learner-Centered Assessment on College Campuses,” Allyn and
Bacon, 2000
Exercise (TTYP)
¾ Review
Communication Key
Action Results
¾ Record your
observations
¾ Discuss observations
¾ Report out
Communication Key Actions:
Rank Order Demonstration
College
Fall 2001
College
Spring/Summer 2002
Comprehends Communication from Others
Organizes the Communication
Adheres to Accepted Conventions
Ensures Understanding
Adjusts to the Audience
Maintains Audience Attention
Comprehends Communication from Others
Adheres to Accepted Conventions
Organizes the Communication
Ensures Understanding
Maintains Audience Attention
Adjusts to the Audience
College
Fall 2002
College
Spring/Summer 2003
Comprehends Communication from Others
Organizes the Communication
Adheres to Accepted Conventions
Ensures Understanding
Adjusts to the Audience
Maintains Audience Attention
Comprehends Communication from Others
Organizes the Communication
Adheres to Accepted Conventions
Ensures Understanding
Adjusts to the Audience
Maintains Audience Attention
COE/Program Attainment of Criterion 3 Program Outcomes
Experiential Education Workplace 2001-2005
COE Supervisor
(a) an ability to apply know ledge of
m athem atics, science, and engineering
Program Supervisor
COE Supervisor
ABET Criterion 3 (a-k) Program Outcomes
(b) an ability to design and conduct experim ents,
as w ell as to analyze and interpret data
Program Supervisor
COE Supervisor
(c) an ability to design a system , com ponent,
or process to m eet desired needs
Program Supervisor
COE Supervisor
(d) an ability to function on m ultidisciplinary team s
Program Supervisor
COE Supervisor
(e) an ability to identify, form ulate,
and solve engineering problem s
Program Supervisor
COE Supervisor
(f) an understanding of professional
and ethical responsibility
Program Supervisor
COE Supervisor
(g) an ability to com m unicate effectively
Program Supervisor
COE Supervisor
(h) the broad education necessary to understand the im pact
of engineering solutions in a global and societal context
Program Supervisor
Program Supervisor
COE Supervisor
(j) a know ledge of contem porary issues
Program Results:
119 Supervisors
Program Supervisor
COE Supervisor
(k) an ability to use the techniques, skills, and m odern
engineering tools for engineering practice
70%
COE Results:
1414 Supervisors
COE Supervisor
(i) a recognition of the need for, and ability
to engage in, life-long learning
Program Supervisor
75%
80%
85%
90%
95%
% Demonstated Acheivement of Program Outcomes
100%
COE/Program Attainment of Criterion 3 Program Outcomes
Experiential Education Workplace 2001-2005
COE Student
(a) an ability to apply know ledge of
m athem atics, science, and engineering
Program Student
COE Student
ABET Criterion 3 (a-k) Program Outcomes
(b) an ability to design and conduct experim ents,
as w ell as to analyze and interpret data
Program Student
COE Student
(c) an ability to design a system , com ponent,
or process to m eet desired needs
Program Student
COE Student
(d) an ability to function on m ultidisciplinary team s
Program Student
COE Student
(e) an ability to identify, form ulate,
and solve engineering problem s
Program Student
COE Student
(f) an understanding of professional
and ethical responsibility
Program Student
COE Student
(g) an ability to com m unicate effectively
Program Student
COE Student
(h) the broad education necessary to understand the im pact
of engineering solutions in a global and societal context
Program Student
COE Results:
1624 Students
COE Student
(i) a recognition of the need for, and ability
to engage in, life-long learning
Program Student
COE Student
(j) a know ledge of contem porary issues
Program Results:
133 Students
Program Student
COE Student
(k) an ability to use the techniques, skills, and m odern
engineering tools for engineering practice
70%
Program Student
75%
80%
85%
90%
95%
% Demonstated Acheivement of Program Outcomes
100%
System Value
¾ Quantify (a-k)
¾ Continuous
constituent feedback
¾ In-process
measurement
¾ Cycle time
¾ ZERO faculty burden
¾ Faculty analysis and
decision making
¾ What, How, and
Where to improve
Thank You!
What
Questions do
you have?
A Critical Incident Story
Situation/
Task
Actions
Results
STAR
BIG Picture