The pleasures and perils of criterion referenced assessment: Putting students at the centre of learning in higher education Merrilyn Goos The University of Queensland Here’s the plan Student perspectives on assessment Changing assessment policy and practice: the UQ experience Why criterion referenced assessment? How can criterion referenced assessment be effectively implemented? Why assessment matters Students can, with difficulty, escape from the effects of poor teaching, they cannot (by definition, if they want to graduate) escape the effects of poor assessment. (Boud, 1995) 1 Student-centred assessment means … specifying desired learning outcomes specifying the assessment scheme with criteria and standards designing assessment tasks that have authentic purposes and encourage a deep approach to learning providing students with timely feedback that points out their strengths and weaknesses and explains how to make improvements for future learning How do students experience assessment? Changing assessment policy and practice: The UQ experience Review of policy and practice in 1995/96: no more 100% exams must use a variety of modes of assessment change from norm-referenced to criterionreferenced assessment 2 Planning for assessment Transparency is essential - all information about assessment requirements must be provided in course specification Assessment must match course objectives Implementing assessment Amount (Goldilocks principle) Form must be suitable to objectives must use more than one form Spread & timing spread assessment across semester rule of 2/3 Making judgments about student performance Use criteria and standards for each piece of assessment Aggregation of results for individual assessment pieces requires judgment, must be defensible, must be made explicit in course specification Final grade for course must reflect UQ grade descriptors 3 UQ grade descriptors 1 Fail. Fails to demonstrate most or all of the basic requirements of the course. 2 Fail. Demonstrates clear deficiencies in understanding and applying fundamental concepts; communicates information or ideas in ways that are frequently incomplete or confusing and give little attention to the conventions of the discipline. 3 Fail. Demonstrates superficial or partial or faulty understanding of the fundamental concepts of the field of study and limited ability to apply these concepts; presents undeveloped or inappropriate or unsupported arguments; communicates information or ideas with lack of clarity and inconsistent adherence to the conventions of the discipline. UQ grade descriptors 4 Pass. Demonstrates adequate understanding and application of the fundamental concepts of the field of study; develops routine arguments or decisions and provides acceptable justification; communicates information and ideas adequately in terms of the conventions of the discipline. 5 Credit. Demonstrates substantial understanding of fundamental concepts of the field of study and ability to apply these concepts in a variety of contexts; develops or adapts convincing arguments and provides coherent justification; communicates information and ideas clearly and fluently in terms of the conventions of the discipline. UQ grade descriptors 6 Distinction. As for 5, with frequent evidence of originality in defining and analysing issues or problems and in creating solutions; uses a level, style and means of communication appropriate to the discipline and the audience. 7 High Distinction. As for 6, with consistent evidence of substantial originality and insight in identifying, generating and communicating competing arguments, perspectives or problem solving approaches; critically evaluates problems, their solutions and implications. 4 Providing feedback Progressive assessment Feedback on every assessment piece Student performance provides feedback on teaching Benefits of criterion referenced assessment Students are graded on the basis of the quality of their work alone, not by reference to how other students in the course perform. Students have a clearer understanding of what is required in assessment. Teachers can more readily make and defend judgments about students’ work. Explicitness provides a language for describing learning. Criteria and standards defined Criteria are dimensions or properties or characteristics of performance. Example (essay task): Comprehensiveness Insight Coherence Communication 5 Criteria and standards defined Criteria are dimensions or properties or characteristics of performance. Standards are levels of attainment or quality that represent performance benchmarks for each criterion. A criteria and standards scheme Criteria (low) Standards (high) Comprehensiveness Insight Coherence Communication Challenges Coming to grips with the concept of a standard Working out how to set standards Devising ways to communicate standards to students and colleagues Becoming proficient in the use of standards (Sadler, 2005) 6 Conceptualising standards Experienced university teachers have developed personal “standards” from: accumulated discipline knowledge recollection of having own work graded as a student previous personal experience in grading student work Ways of setting standards Numerical cut offs Tacit “in the head” knowledge Exemplars Verbal descriptions A recommended process for setting standards Begin with a set of qualitative grading decisions (e.g., engage a group of teachers in discussing their judgments when grading student work). From these real judgments, identify the essential characteristics (criteria) and levels of quality (standards) in students’ work, selecting some works as exemplars. Develop an economical set of verbal descriptions and exemplars embedded within the context of the discipline that can be communicated to colleagues and students. 7 Ways of communicating standards 1. Decide on the number of standards o o o o What is the best possible standard that can be anticipated in this learning environment? What is the least standard that will be considered acceptable? What standards lie between these two? What standard can be anticipated as unacceptable? Ways of communicating standards 2. Decide on how to label the standards Grades 1-2 3-4 5 Letters D C B A 0-39% 40-59% 60-79% >80% % Bands 6-7 Marks 0-4 5-10 11-16 17-20 Labels Fail Competent Advanced Excellent Ways of communicating standards 3. Develop descriptions of each standard for each criterion o o o o Use brief, clear, specific language that is accessible to students Specify behaviour that can be demonstrated rather than that which must be inferred Avoid comparative terms (more, less) Avoid terms that indicate quantity, frequency, or amount (none, little, some; always, often, sometimes, rarely) 8 A criteria and standards scheme should … be sufficiently specific with identifiable standards be constructed so that each cell describes the behaviour/activities of a student performing at that standard specify qualitative, rather than quantitative, differences between standards for each criterion have an internal hierarchy so that if headings are removed, the scheme could be reassembled from the evidence of the constituent statements alone Criteria Comprehensiveness CC1 A Demonstrates sophisticated understanding of theoretical perspectives on mathematics education. Engages thoughtfully with a broad range of these ideas to interpret and critique past and/or current professional practices, and consider possibilities for change. B C Demonstrates clear understanding of theoretical perspectives Applies a broad range of on mathematics education, although these ideas to interpret and analyse professional some relevant material may be omitted or its practice. significance overlooked. Demonstrates thorough understanding of theoretical perspectives on mathematics education. D E Demonstrates some understanding of theoretical perspectives on mathematics education, possibly marred by omissions or misinterpretations. Identifies some examples of professional practice connected with these ideas. Demonstrates limited understanding of theoretical perspectives on mathematics education. Makes few connections between these ideas and professional practice. Applies some of these ideas to interpret relevant aspects of professional practice. Insight CC2 Identifies significant current issues and offers insightful analysis and critical evaluation of these issues in the light of theoretical perspectives. Identifies current issues and offers critical analysis in the light of theoretical perspectives. Identifies some current issues and explains their relevance in terms of theoretical perspectives. Identifies some issues but relevance remains unclear. Demonstrates limited ability to identify current issues and discuss relevance. Coherence CC3 Synthesises ideas and evidence to produce cogent arguments and conclusions. Argument is thoughtfully developed and elements are sequenced and connected to produce an integrated whole. Produces reasoned arguments and draws on evidence to support conclusions. Argument is logically developed in terms of content and structure. Produces well explained arguments that connect ideas and use evidence. Ideas are presented in a sensible sequence and move towards conclusions. Selection and sequencing of ideas is not always logical or connected. Conclusions may not be well supported by evidence or arguments. Difficult for reader to obtain meaning from explanations. Ideas are unconnected. No conclusions drawn. Communication Presentation observes all conventions of spelling, punctuation, grammar. Quoting and referencing are technically correct, consistent, and complete. Writing communicates concisely and effectively to intended audience within the specified word limit. Presentation observes almost all conventions of spelling, punctuation, grammar. Quoting and referencing are complete, with minimal technical errors or inconsistencies. Writing communicates clearly to intended audience within specified word limit. Presentation observes most conventions of spelling, punctuation, grammar. Quoting and referencing are complete, although there may be some technical errors or inconsistencies. Writing communicates adequately to intended audience within specified word limit. Presentation is marred by some errors in spelling, punctuation, grammar. There may be omissions or inconsistencies in quoting and referencing. Format and structure make it difficult for audience to follow the writer’s intentions. Presentation is compromised by many errors in spelling, punctuation, grammar. Quoting and referencing are incomplete or inconsistent. Little attention given to format and structure in communicating to audience. CC4 Criteria Comprehensiveness A Demonstrates sophisticated understanding of theoretical perspectives on mathematics education. Engages thoughtfully with a broad range of these ideas to interpret and critique past and/or current professional practices, and consider possibilities for change. Coherence Synthesises ideas and evidence to produce cogent arguments and conclusions. Argument is thoughtfully developed and elements are sequenced and connected to produce an integrated whole. C Demonstrates clear understanding of theoretical perspectives on mathematics education, although some relevant material may be omitted or its significance overlooked. Applies some of these ideas to interpret relevant aspects of professional practice. Produces well explained arguments that connect ideas and use evidence. Ideas are presented in a sensible sequence and move towards conclusions 9 Becoming proficient in using standards Develop in collaboration with colleagues in your discipline. Use in conjunction with exemplars of student work. Use social moderation processes to develop shared understanding of standards amongst teaching staff. Use criteria and standards schemes to provide feedback to students. Criterion referenced assessment: The pleasures Can increase student motivation and engagement in assessment. Can encourage use of greater variety of assessment tasks. Criteria & standards can communicate to students expectations regarding quality. Grading with criteria & standards can provide feedback to students about their learning. Development of criteria & standards can stimulate academic conversations around assessment. Criterion referenced assessment: The perils Developing explicit, usable criteria & standards takes time and effort. Providing criteria & standards to students does not guarantee they will understand or use them! Temptation to over-specify criteria & standards risks trivialising and fragmenting learning. Aggregating results from individual assessment tasks obscures overall pattern of achievement on different criteria. 10 Are we there yet? Assessment change is a long term process and people are at different stages of the journey. There are different but equally valuable ways to implement good assessment practices that align with university policies. Assessment policies need regular monitoring and review. Teachers benefit from regular review of their assessment practices by consulting with peers, responding to student feedback, maintaining currency with new policies. The pleasures and perils of criterion referenced assessment: Putting students at the centre of learning in higher education Merrilyn Goos The University of Queensland 11