Utilization of an
Online Grammar
and Writing Program
to Improve Student
Work
Patricia Backer,
Professor, SJSU
SJSU Washington Square
Introduction
• Universities have begun to use online
technologies to increase instructor
efficiency and serve more students.
• We decided to pilot the ETS Criterion
Writing Evaluation System in the Tech
198: Technology & Civilization class,
which has recently been revamped into a
mega class that includes an optional
online portion.
• We will compare student
performance from the Criterion pilot with
existing assessment data from this
course.
SJSU Washington Square
Hypothesis
The hypothesis was that the use of ETS
Criterion would improve students’ writing
in the class, therefore reducing the
amount of time required to grade the
students’ research papers.
SJSU Criterion
Washington Square
ETS
• Criterion operates on a web-based
platform, is instructor driven, and can be
used a tool for students to plan, write and
revise their essays
• By providing quick, diagnostic
feedback and a holistic score students are
able to review, revise and resubmit
essays.
Categories
Trait Feedback Analysis
SJSU Washington Square
• Grammar score – based on errors
such as those in subject-verb agreement
among others
• Mechanics score – derived from
errors in spelling and other like errors
• Usage score – based on such errors as
article errors and confused words
• Style score – based on instances of
overly repeated words and the number of
very long or very short sentences as well
as other such features
• Organization/development score –
based on the identification of sentences
that correspond to the background,
thesis, main idea, supporting idea, and
conclusion
Criterion sample feedback
for Grammar
Criterion sample feedback for
Usage
Criterion sample feedback for
Mechanics
Criterion sample
feedback for Style
Criterion sample
feedback for
Organization &
Development
Assessment
of Criterion in Tech 198
SJSU Washington Square
• Study conducted over 3 semesters
• Students in Tech 198 were required to
submit four research papers to Criterion
for a grammar check:
• Research Exercise 1 Draft Paper
• Research Exercise 1 Final Paper
• Research Exercise 2 Draft Paper
• Research Exercise 2 Final Paper
SJSU Washington Square
Results
• 226 of the 375 students enrolled in
the class submitted all four papers to
Criterion
• For the rough and final drafts for both
Research Exercises, the instructor
reviewed the Criterion report for the last
essay submitted by each student.
SJSU Washington
Square
Results
by Ethnicity
AAPI
Grammar errors
"0" errors
"1" error
"2" errors
"3" errors
"4" errors
"5 or more" errors
Average errors
Ttest
RE 1 Draft
RE 1 Final
RE 2 Draft
RE 2 Final
AAPI All students AAPI All students AAPI All students AAPI All students
31
69
59
123
41
96
57
116
23
45
19
47
19
54
19
59
11
35
3
20
16
28
9
24
6
26
11
23
5
18
4
16
10
23
0
4
5
13
4
6
14
28
3
9
9
17
2
5
95
226
95
226
95
226
95
226
2.29
2.19
0.78
1
1.44
1.43
0.84
0.93
RE1 draft vs RE1 final:
RE 2 Draft vs RE2 final:
AAPI & non-AAPI 0.0003
AAPI & non-AAPI 0.0144 0.0372
0.0001
Average errors
RE 1 Draft
AAPI All students
24
42
8
27
15
31
12
31
10
31
26
64
95
226
3.89
3.84
Ttest
RE1 draft vs RE1 final:
AAPI & non-AAPI 0.0055
Usage errors
"0" errors
"1" error
"2" errors
"3" errors
"4" errors
"5 or more" errors
RE 1 Final
AAPI All students
42
88
11
39
12
32
5
13
7
14
18
40
95
226
2.17
2.21
Average errors
RE 1 Draft
AAPI All students
32
71
22
40
14
28
7
23
6
16
14
48
95
226
2.28
2.68
Ttest
RE1 draft vs RE1 final:
AAPI &non-AAPI 0.0015
Mechanics errors
"0" errors
"1" error
"2" errors
"3" errors
"4" errors
"5 or more" errors
0.0001
RE 1 Final
AAPI All students
58
129
22
51
8
22
4
12
0
2
3
10
95
226
0.88
1.05
0.0000
RE 2 Draft
AAPI All students
24
50
20
40
11
31
10
25
10
28
20
52
95
226
2.72
3
RE 2 Final
AAPI All students
40
97
15
42
14
30
13
22
7
16
6
19
95
226
1.62
1.65
RE 2 Draft vs RE2 final:
AAPI & non-AAPI 0.0042
RE 2 Draft
AAPI All students
43
88
26
61
12
33
8
14
0
9
6
21
95
226
1.22
1.65
0.0000
RE 2 Final
AAPI All students
63
129
18
49
8
26
3
9
1
6
2
7
95
226
0.61
0.84
RE 2 Draft vs RE2 final:
AAPI & non-AAPI 0.0081
0.0005
Overall, when comparing the draft and final papers for each research
exercise, students reduced the number of grammar, mechanics and usage errors.
For both AAPI and all students, the reduction of errors was significant.
There were enough students in the class to analyze the data by ethnicity for
the following ethnicities: Caucasian, AAPI, Latino/a, and African-American.
For each ethnic subgroup analyzed, the students significantly reduced the
number of “fixable” errors when comparing the draft and final papers for each
research exercise.
SJSU Washington
Square
Results
by Major
•
Of the 226 students who completed all four writing
assignments, 172 were students in the SJSU College of
Engineering and 54 students were students from other
colleges on campus.
•
An analysis of engineering versus non-engineering
students shows notable differences in the number of
errors made on each assignment.
•
In each of the three categories of errors (Grammar,
Mechanics, and Usage) non-engineering students made
more “fixable” errors.
•
The average number of grammar and usage errors
for non-engineering students was higher for all four
writing assignments than for engineering students.
•
Engineering students appeared to utilize Criterion
better and, overall, they improved their error rates more
than non-engineering students
American Indian
Asian
African-American
Caucasian
Latino/a
Pacific Islander
Multi-ethnic/Decline
to State
Total
Engineering
2
78
8
48
19
3
Non-Engr
1
13
4
22
5
1
14
172
8
54
SJSU Washington Square
Conclusion
• The use of ETS Criterion improved all
students’ writing in class
• The instructor spent more time
making content comments rather than
grammatical comments
• Engineering students appeared to
utilize Criterion to a greater extent than
non-engineering students.
• Students who lacked the motivation to
revise their papers, according to
Criterion’s suggested revisions, did not
benefit from the Criterion program.