Lecture 8
4/16/2015 Language Testing and Assessment Lecture 8 Brózik-Piniel, Katalin Construct: Interact successfully in a foreign language Comprehension and production CEFR- Overall Oral Production B2 Can give clear‚ detailed descriptions and presentations on a wide range of subjects related to his/her field of interest‚ expanding and supporting ideas with subsidiary points and relevant examples. 1 4/16/2015 What affects candidates’ performances which should be controlled for as much as possible in high-stakes tests: (construct irrelevant facets) Individual characteristics Task characteristics Other participants In order to ensure the reliability and validity of speaking examinations, as a means of standardizing the exam: design tasks carefully (select relevant+variety of activities) assessor and interlocutor (2 examiners) providing an Interlocutor Frame (all the instructions, suggested question prompts needed ) and guidelines for behavior train interlocutors and assessors employing analytic assessment scales that have been piloted and revised; monitoring interlocutors and assessors. Benchmark performances (Csépes & Együd, 2004, p. 25) 2 4/16/2015 Individual mode Questions of status Not life-like Interaction controlled by interviewer the examiner’s L1 background or acquaintanceship with the candidate can influence candidate performance the interlocutor’s acquaintanceship with candidates may also influence the communication strategies (help request) used by the participants Paired mode Testing candidates in pairs and small groups is motivated by: -dissatisfaction with the oral interview as the only test format -search for new tasks that can elicit different patterns of interaction -desire to generate positive washback on teaching by encouraging more interaction -to mirror good language teaching practice -time saving and cost effective (Csépes & Együd, 2004, p. 33) 3 4/16/2015 Characteristics of paired exams - Mismatch between proficiency/personality + WTC + high level involvement + large language sample + control, ease Ask ONLY from a set of pre-specified options indicated on the interlocutor’s sheet. The interlocutor is only allowed to make comments that are in compliance with the guidelines for conducting the exam. The interlocutor has to select and ask follow-up questions even if the candidate manages to talk about the pictures at length. It is not necessary for the interlocutor to use the question prompts in the order listed. Thank you and proceed to the next part of the examination or Thank you. That will do. (Csépes & Együd, 2004, p. 52) 4 4/16/2015 • interview • picture-based individual long turn • discussion • role-play Personal information Opinion Possible problems: Not life-like Variation in elicitation can lead to different judgments of language ability (can be counterbalanced by standardized interlocutor behavior) 5 4/16/2015 SAMPLE 1 SAMPLE2 Where are you from? Tell me about your town/village. Where do you live? How do you spend your free time? Do you have friends? What do you like most about television? Do you like watching TV? Tell me about your family. Do you have any brothers or sisters? How do you think American culture influences our life? Why do you think many countries see the American influence as a threat to their national identity? How have the technological inventions of the last century changed our life? Which scientific inventions of the 20th century do you think have changed people’s lives the most? p. 41 In interviews the interlocutor should follow specific guidelines: • Use global questions for elicitation. • Use wh-questions instead of yes/no questions whenever possible. • Never ask more than one question at a time. • Do not talk more than necessary: refrain from making unnecessary comments. • Do not interrupt or finish what the candidate wants to say. • Do not ask questions that require background knowledge. • Avoid ambiguous and embarrassing questions. • Use genuine questions and avoid display questions. • Maintain eye contact with the candidate when talking to him/her. -Task in accordance with proficiency level -Provide candidate with maximum scope of response (Csépes & Együd, 2004, p. 40 and p. 42) 6 4/16/2015 Using pictures +easy to find +economic and effective +freedom to show mastery +deficiencies in reading comprehension cannot prevent candidates from doing well (Csépes & Együd, 2004, p. 51) Inadequate use of pictures may stem from: - not challenging enough, not enough stimuli - is culture-dependent, needs special background knowledge - topic distressing, offensive, violent or taboo - surreal, abstract and symbolic pictures - bizarre, unrealistic situations - too many pictures to compare and contrast - only one picture, too simple, leads to description - pictures too similar for comparison and contrast - static pictures - over-crowded pictures (Csépes & Együd, 2004, p. 52) 7 4/16/2015 Paired and individual mode Two-way task (interaction) Negotiation of own opinion – come to an agreement (planning, listing, ranking, solving a problem etc.) – specific focus is needed Avoid eliciting two monologues GUIDELINES FOR WRITING PAIRED TASKS Item writers should bear in mind that • the reading input should be minimal • the task instructions should be read out by the Interlocutor • instructions should be clear • contributions to the interaction should be balanced • comparable tasks • the interaction should be task-based • the tasks have to be guided but not fully controlled • controversial topics seem to be more likely to generate a discussion than neutral topics; • distressing, offensive, violent or taboo topics should be avoided; • the topic should be something on which it is reasonable to expect candidates to have an opinion • define the context of the conversation very carefully • Candidates should be given a chance to voice their own opinion rather than argue for something that they cannot identify with. (Csépes & Együd, 2004, p. 90) 8 4/16/2015 Involve simulating roles we take on in our everyday lives Give candidates familiar roles. Give them a standardized, clear and concise role description. Instruct them about their expected contributions to the role-play situation. Assessment is based on the process and not the result. Select prompts/input text and design the task so that it generates the appropriate amount of varied language at the required level. Use prompts that are clear, and appropriate for the target age. The language level of verbal prompts should be below the tested level. Do not use distressing, offensive, violent or taboo topics. Treat sensitive topics with care. Do not use surreal, abstract, puzzling or symbolic pictures at lower levels. Do not require students to use their imagination or creativity as these are not to be tested. (Csépes & Együd, 2004, p.102-131) Narrative tasks Instruction tasks Description tasks Prepared monologue 9 4/16/2015 Aim for consistency (intra-rater reliability: self-check, go back to the first performance) Rating checklists can be adapted and used in learningrelated settings (Luoma, 2004) Rating checklists with useful and comprehensive descriptors should provide useful informaation for the learner Peer assessment: learners become more aware of the learning goals, use criteria that learners comprehend(should be task-related, good idea to develop the criteria together with the learners, listing features of task achievement) Feedback: concrete, descriptive, and relates the performance to the goals (using feedback reports and clearly formulated speaking goals) (Luoma, 2004) Share the learning objective(s) for the talk activity clearly. Encourage pupils to reflect on how they are learning through using talk. Discuss and agree criteria for success in advance of an oral activity or task. If possible, listen to/watch examples first, or model a successful outcome yourself. Different tasks will require different types of speaking and listening, and hence specific success criteria. Set up peer response partners to provide feedback and to help each other improve. Encourage pupils to use a talk log or journal and to note down successful contributions they have made, or to assess the contributions of others, and to reflect on what they have learned. Provide them with vocabulary/terminology to aid reflection and analysis. Use pupil observers to focus on specific aspects of talk and provide feedback to other groups. Use video on occasions to enable pupils to observe and comment on their own performance. Plan time at the end of a specific oral activity to discuss and debrief with pupils. Use quick pair discussion first or “thinking” time before inviting comment. Focus on what went well before looking at areas for improvement. 10 4/16/2015 Key skills for assessment Evidence for assessment Target for improvement (examples are given for guidance) Communicating clearly and imaginatively Structuring and sustaining talk Adapting talk to different situations and for different audiences Adopting different roles and using a range of techniques Present ideas clearly and confidently Listening carefully to a presentation, reading, programme, discussion etc. Understanding key points, ideas and views Listen carefully to identify key points and understand ideas Participating in discussions Listening carefully to others’ contributions Making helpful and sustained contributions to discussions Contribute effectively to group discussions Using standard English appropriately Use standard English in formal situations Speaking and Listening targets Examples of follow-up actions Listen carefully to identify key points and understand ideas Contribute effectively to group discussions Present ideas clearly and confidently Vary expression and vocabulary to suit the audience and purpose Vary expression and vocabulary to suit the audience and purpose Prepare helpful frameworks for the task or topic before you listen and use them to record key points as you listen Listen for and list key “naming” words for the topic and their synonyms, or topic sentences that mark key stages in the argument. Practise reconstructing the text using these lists. Provide a timed summary for someone else of the topic you’ve just listened to. Revise “golden rules” for effective group work, or devise your own list. Listen to another group’s discussion and make notes on what is helpful and not so helpful in getting the task done or reaching agreement. Decide what you can change about your own contributions in the light of this list. Ask a friend to monitor your progress. Give yourself a set number of counters to “spend” when you make a helpful contribution to discussion. You must use them all but cannot contribute once they are all used. Practise using helpful prompt phrases for taking turns in discussion e.g. I agree with …because…; I can see what you’re saying but….; a new point I want to make is…; to sum up, we seem to be saying that…. Practise making notes on a topic or task in different formats, and using them to structure your talk. Choose a subject or process that you know a lot about and write very brief notes about it, e.g. 6 key words in sequence. “Teach” someone else your topic, using only these notes. Ask them for feedback about clarity and confidence. Practise giving 15 second/30 second/60 second talks on a topic, building up the detail each time. Prepare notes on a topic with a friend, then get them to ask you questions about it. Use the notes to prompt your answers. Highlight key words in the task that give information about who the text is for, and what its purpose is. Plan key points under these headings and list examples of suitable words and phrases to use. Ask someone to take the role of the intended audience, listen to your talk and give you suggestions about what needs changing. Record yourself giving your talk and review its effectiveness. 11 4/16/2015 Construct: receptive skill, it varies from test to test (what do we wan to measure)- ways and reasons It involves following arguments, recognizing attitudes, obtaining the gist (Hughes, 2003) variety of purposes a range of texts different kinds of information No replay Chunks Redundancy Reduced forms Performance variables Colloquial expressions Accents Speed Intonation, rhythm, stress 12 4/16/2015 selective attention (level of attention) listening for understanding / filtering / hearing skills / subskills ability to understand gist, main points, details, infer speakers’ attitudes, opinions or feelings Understanding: local meaning full linguistic meaning inferred meaning Bottom-up, top-down Specific information vs. Gist (global) Intensive listening vs. Extensive listening Stress-intentions of the speaker Sound discrimination 13 4/16/2015 Type: monologues, dialogues , group conversations Form: description, narration, etc. Length Speed Dialect Accent Authentic (slightly edited) texts intended for listening Domain: personal, social, academic, vocational Genre: group discussion, debate, lecture, presentation, talks, report Topic: abstract/concrete, complex/simple (propositionally/linguistically), familiar/unfamiliar, technical/general Medium: live, broadcast, recorded 14 4/16/2015 Short answer questions Multiple-choice items Note taking Matching and sequencing Information transfer (e.g. charts) Gap filling Partial dictation Transcription (numbers, words spelled out) Based on texts heard (not written) !!! Authentic texts Can be written in L1 Trialing Items set sufficiently far apart Key words-warning signal 15 4/16/2015 All texts must be authentic. Texts must not be taken from printed material in other examinations or English language teaching materials. The topic of the texts must be accessible to the students’ age group. Texts must not be offensive, distressing or violent. Texts must be of a suitable length. Texts must be of a suitable level of difficulty. A text must have a title unless it is used as part of the item. (Fehérváryné & Pizorn, 2003) Have a clear idea what skill and sub-skill(s) you intend to test and select the task and texts/prompts accordingly. The task type must be one which is familiar to the students who are tested. Tasks must be accessible to the students’ age group. Tasks must not be offensive, distressing or violent. The time allocated for each task must be sufficient for somebody who has the ability being tested to complete the task comfortably within the time limit. Students must be able to see easily how the task relates to the text. No more than one testing technique should be used in each task. Do not switch from multiple-choice to single-word answers in a single task, for example. (Fehérváryné & Pizorn, 2003) 16 4/16/2015 There should be a minimum of five items in a task. Items must follow the text sequence. Items must be spread evenly through the text. It must not be possible to answer any item without reference to the text. This must be checked carefully. In non-sequencing tasks, items must not be interdependent – students should not need one answer in order to find another. Items must not overlap. Two items must not have similar answers. Items must have a complete answer key on a separate sheet. All possible good answers and predictable wrong answers should be provided. The items in each task must be numbered sequentially. One item in each task must have an answer provided as an example. The example item, written in italics, should be marked 0 (zero) and should precede the other items. The example answer must be provided in the form in which candidates are expected to write their answers and, where appropriate, it must be handwritten(italics). “Find the wrong answer” type items are not acceptable in multiple choice tasks. Each item must score one point. Clear and sufficient rubrics (Fehérváryné & Pizorn, 2003) Good quality If live speech – training is needed to be reliable Familiarization with the items 17 4/16/2015 Spelling and grammar mistakes are not taken into account Listening Log Task: Listening to Recorded Passages Name: __________________________ Here is your task: Keep this listening log in your binder/folder. Fill it out each time that you finish listening to a text. Date Type of Text Topic Main Idea Comment The passage that I found the easiest to understand was ___________ because _________. The hardest passage to understand was _________ because ________. Source: www.caslt.org 18
© Copyright 2024