Phase II Analysis of Chemistry Support Initiative
Prepared by Ahrash Bissell Ph.D.
Project background
Dr. Richard MacPhail (chemistry) and Dr. Ahrash Bissell (ASIP) met in Fall 2005 to
discuss the general chemistry (Chem 21 and 22) sequence, with particular focus on the
professor's desired learning outcomes, the constraints he faced, and his goals for the
future. Dr. MacPhail and the ARC staff exchanged information and ideas related to the
course goals, the ways in which to best support student learning, and measurements of
student success. Because the visitation rate of general chemistry students to the ARC was
so high, Dr. Bissell developed tools that could be used in ASIP one-on-one sessions to
more effectively convey the learning expectations of the general chemistry courses and to
give the students some concrete materials which could help them to practice the needed
academic skills. Based on our subjective assessments of student understanding and
performance, one of the best tools was something we called "problem manipulation".
Problem manipulation is a technique that is grounded in metacognitive theory. Briefly,
the method encourages students to generate and evaluate knowledge (appropriate to the
course) on their own. It is scaffolded, content-specific, and suitable for group study. It
also provides an appropriate means for predicting the types of questions that might
appear on an exam. Finally, the method integrates well with other academic skills that
students will often learn at the ARC, and is thus well suited to the "whole-course
approach" to supporting student learning and the notion that most students have a suite of
skills that should be attended to simultaneously.
Dr. MacPhail met with the ARC staff to discuss how we might both benefit from
embedding the method directly into his course recitation.
Project implementation in Spring, 2006
The format that we chose for integrating the method was as follows:
1. Pre-recitation exercise.
a. Prior to each week's recitation, the students would receive a single
problem to work on their own. The problem was designed to be
"advanced", but do-able, and the content was matched to that time-period
of the class.
b. Included in the pre-recitation problem were a few "canned" manipulations.
Hints were given, and the manipulations were designed mostly to illustrate
the concept, rather than challenge the students.
2. In-recitation group work.
a. In recitation, each class was divided up into groups of four (or so). Each
group worked on the same recitation exercise, which closely matched the
1
content of the pre-recitation exercise but was designed to be more
difficult. The expectation was that the group members would work
together to overcome the challenge of the problem.
b. The exercise also consisted of either "canned" manipulations (early in the
semester) or "open" manipulations (later in the semester). In either case,
the expectation was that the problem would require the group members to
work together in order to solve it in the allotted time. This teamwork
element was especially crucial when the groups were asked generate their
own manipulations later in the semester.
3. Post-recitation exercise.
a. After each recitation, the students picked up a post-recitation problem, due
the following week (along with the next pre-recitation problem). Again,
the content was matched to the other problems, and the difficulty level
was designed to be similar to that of the in-recitation problem.
b. These problems also contained manipulations, some canned, some open.
these problems were also graded, though the grade was mostly based on
effort, not performance.
There were a few recitations that were devoted to exam review, and one or two
recitations where a "fun" chemistry exercise was substituted, but the bulk of the
recitations were formatted as illustrated above.
At the end of the semester, all of the students in the course were asked to fill out a survey
in which they reflected on the recitation format, on their performance in recitation and the
course overall, and on the contrast of those recitation sections to other recitation formats
they might have had (in science). The return rate on the survey (administered
electronically) was very high, approximately 95%.
Alterations to this format for Spring, 2007.
Based on feedback from the TAs and the surveys from 2006, we made several
adjustments for Dr. MacPhail's Chem 22 class in 2007.
1. In addition to the recitation exercises (described above), we also administered
take-home quizzes that explicitly asked the students to use and reflect on the
problem-manipulation method. The quiz questions were very similar to at least
one of the questions on each midterm exam and the final. In addition to giving us
(and the students) more feedback on academic performance, this arrangement
makes it possible to evaluate the correlation between student improvement in
recitation (using the problem-manipulation method) and on the exams (using
whatever method they choose).
2. In order to more accurately make the comparison described above, course graders
separately recorded the scores for a subset of exam questions that were designed
to mimic the recitation problems and were most easily solved by applying the
problem-manipulation method.
2
3. Course T.A.s devoted more of the recitation period to metacognitive
development. They wanted the students to understand the rationale behind the
methods, rather than simply apply the method when asked to.
4. We also altered some of the survey questions to capture student perceptions that
were missed the first time and also to reflect the changes made above.
Results
Subjective assessment of the success of the project
In terms of all of the positive outcomes from the process described in the Introduction,
this project has been a great success. In addition to this specific project, we have engaged
meaningfully with faculty in assessing the characteristics and learning needs of "at-risk"
students in chemistry and math. The chemistry faculty involved in this assessment have
been enthusiastic and receptive throughout, and they have recognized the work and
perspective of the ARC as being key to the ongoing effort. There is continued interest in
the evaluation of the "off-cycle" chemistry courses, the development of additional peerled-team-learning support programs, and other collaborative programs as appropriate.
Given appropriate personnel and resources, we have every expectation that the ARC can
make a real difference in supporting freshman and sophomore students in these STEM
courses.
Feedback from Dr. MacPhail, Claire Siburt, and the course TAs was also very positive. In
general, the students seemed to be more engaged, especially in recitation, as well as more
aware of the learning expectations for the course. The TAs enjoyed leading their sections
more than they had in previous courses, and they felt invested in the teaching process.
Survey results
The student responses to the end-of-course survey were also overwhelmingly positive.
Survey responses were very similar for both years and are discussed together.
Between 77-87% of the students said that they preferred the recitation format they
experienced in Dr. MacPhail's class over prior college recitation experiences (in
chemistry or comparable classes). Similarly, 88-91% of students said that we should
continue to use this recitation format in future semesters.
We asked students to consider the impact of the recitation sections and problemmanipulations on their course performance and understanding of the material. 87% of
students said that the recitation sections and associated problem-manipulation
methodology helped to prepare them for the midterm exams. 90% of students found
every component of the recitation format to be helpful (pre-recitation, in-recitation, and
post-recitation exercises). In terms of student self-evaluation of the impact of the method
on their understanding of how they learn best, the results were more equivocal, with
approximately 50% of students agreeing that there was a positive impact. The results
3
were similar for student reports on whether they actually learned the chemistry concepts
better.
Nonetheless, students did not feel that the format was a burden. Indeed, students really
appreciated the group work (86-92% approval), and many students (approx. 60%)
indicated that they were already using the method in other courses (or could how it could
be useful). More than anything, it appeared that most students appreciated the thought
that went into the recitation format, as well as the notion that the professor was giving
them guidance for how to learn the material rather than just what to learn.
Student performance results
There were no obvious overall differences in course performance among the two
sections, or even as compared to previous years where this format was not used.
However, the course uses a curved grading scale, and with so many students it would in
fact be surprising if there was a substantial shift in the performance of every student in
the class. Instead, we are most interested in two things:
1. Does every student who truly desires to succeed have access to the knowledge
and skills that are necessary to do so?
2. Is there an impact on course performance based on performance on skill mastery
and performance in the recitation sections?
The first item is harder to quantify, but we have some information that leads us to believe
that we came closer to achieving that goal in these courses than in the other chemistry
courses. First, professor and TA feedback indicated that most students they met with
seemed to understand the demands of the course better than in previous semesters.
Moreover, more students were able to identify the specific areas of weakness and were
trying to get help on those areas accordingly. Second, the visitation rate to ASIP for
students in Dr. MacPhail's sections was quite a bit lower than the other chemistry
sections, despite the greater exposure to the ARC in his classes. This reduction is almost
certainly due to greater student confidence that they understand the learning expectations
and know which areas need improvement. Indeed, students in those classes that did visit
ASIP were clearly more aware of the possible reasons for their struggles than students in
previous semesters or other sections. Note, however, that there continues to be a concern
about student overconfidence in their metacognitive abilities and capacity to self-teach
better academic skills. This is an area that needs further attention.
The second item is amenable to statistical analysis by comparing student responses to the
end-of-semester survey to their actual performances, both in recitation and on course
exams.
4