Creating Tests That Assess Higher Order Thinking Skills Innovative Educators Dr. Jennifer Hurd October 13, 2010 Objectives When you complete this workshop, you should be able to: • Write different types of questions that will assess more than knowledge and comprehension on the Bloom’s Taxonomy Scale. • Analyze test items to determine if these test items assess higher order thinking skills. • Avoid signaling the correct answer by knowing common errors in teacher-developed exams. Bloom’s Taxonomy Developing Objectives Using Bloom’s Taxonomy • Objectives must be measurable • Objectives must be attainable • Objectives must have a limiting factor Objectives must: • Contain the same key words that we will discuss for developing questions. • Map to the mission statement or goals for the entire course or department • Demonstrate the skills have been attained for each level of Bloom’s Taxonomy Example of an Objective According to the course material, 85% of the students will be able to design a business plan for a new business of their choice. Example of an Application Objective According to the course content, 90% of the students in this course will be able to complete a mock interview with three of the instructors Learning Outcomes • Are the primary building blocks of good curriculum design. • Support the goal or learning outcome in that each is a small step to arriving at what the learner is supposed to know or be able to do. • Are carefully worded with qualifiers to restrict the conditions and terms under which the learning outcomes are met. • Are measurable! Learning outcomes must contain verbs that describe observable, measurable actions. Learning Outcomes should:  Provide the basis to select or design instructional materials, content or assessments  Provide a framework within which learners can organize their efforts to complete the learning task Planning a test • Determine the method of assessment – Demonstrating knowledge and understanding – Assessing critical thinking skills – Assessing problem solving skills – Assessing performance of procedures – Assessing application skills – Assessing collaborative learning skills – Assessing research skills Planning: Specifying What is to be Assessed • Clarify instructional learning outcomes • Define the purpose of the test – Pre-instructional (pre-test) – Tests and quizzes – Final exams • Specify what will be tested and at what level • Develop a test blueprint Blueprint for a Test Content/ Skill Area Knowledge/ Understanding Critical Thinking Skills Skill Comprehension Application Critical Thinking Problem Solving Performance of Procedures/ Demonstrati ng Techniques Collaborative Learning Skills Integrate Learning Into Professional Practice Assessing Research Skills Basic Tenants of Test Item Writing • • • • Reading Level Write items as simply as possible Test for important details Avoid direct quotes or verbatim statements • Identify any opinions or authority that are used in test items Anatomy of an Exam Question • Should have three parts: – A key word that tells the students what they are expected to do – An object that tells them what general information to use to perform their task – A limiting factor or factors that tells what specific information to include in the answer Bloom’s Taxonomy Let’s look at each level of Bloom’s Taxonomy to see how we can develop questions for each level. Knowledge Questions • • • • • • • remembering; memorizing; recognizing; recalling identification and recall of information Who, what, when, where, how ...? Describe An example of a knowledge question According to the instructor, define the key words for knowledge questions. An example of a knowledge question According to the instructor, list the key words for knowledge questions. Can you change this question into a M/C question? Writing a question According to the instructor, which of the following are the key words for knowledge questions? A. B. C. D. E. list, synthesize, evaluate, define, write List, define, dramatize, solve List, define, duplicate, memorize None of the above A and B only Application Question Key Words • • • • • Imagine Pretend Make Believe Visualize Assume • Put yourself in the situation of • Use your knowledge to • Suppose In groups of three or four write a three-part application question on any topic or subject area using one of the key words listed above. Be prepared to share your question. Imagination Activity • Imagine that you are creating your own company • You can hire 5 key people to help you run your company • These 5 people can be anyone from the past or present • Who would you hire and why? • You have 3 minutes to decide, then you will discuss your response with your table partners. One person from each table will share with the group. Evaluation Question Key Words • • • • • • • • Rank Order Judge Grade Measure Determine Convince persuade • • • • • • • Decide Support Explain Critique Criticize Recommend Suggest Sample Three-part Evaluation Question • According to the number of safety-related features each contains, determine which compact car is safest to drive. Key Word: Determine Object: to find out which compact car is safest to drive Limiting Factor: cars must be compact Only safety-related features to be examined A/B Visit Argue Workshop • This workshop allows students to persuade, convince, determine, decide, recommend, suggest, support, and explain parts of course content. • Applying course content to other parts of the course or to their own life makes it even more meaningful and worthwhile. A/B Visit Argue Workshop • Set up the room for A/B visits, or have students get into pairs. • Your topic is to argue whether or not there should be a student success course for college credit on your campus. • The people on the outside (A’s) will take the pro stance. • The people on the inside (B’s) will take the con stance. Synthesis Question Key Words • • • • • • • • Combine Integrate Modify Revise Improve Rearrange Reconstruct Hypothesize • • • • • • • • Substitute Create Generate Devise Design Compose Plan Predict Synthesis Sample Question • Rearrange the amendments to the U.S. Constitution in order from least to most important in everyday life. As well, devise any new amendments that you think would improve everyday life. Justify your amendments according to your text or your notes. Rearrange the amendments to the U.S. Constitution in order from least to most important in everyday life. As well, devise any new ammendments that you think would improve everyday life. • Key Words: rearrange and devise • Object: To discover which constitutional amendments you consider most important and which you would like to add to improve everyday life • Limiting Factor: deal only with amendments to the U.S. Constitution. Other Synthesis Assessments • Ask students to create mnemonic devices to remember course content. • Students can create crossword puzzles that center on a particular part of course content • Create a visual product to show course content. • Example: Create a company logo for XXX company. Analyzing Test Items to Determine if they Assess Higher Order Thinking Skills In the next three questions, use the chat feature to tell me whether the question assesses higher order thinking skills. hy/why not. According to the text, good money management skills include A. Planning, budgeting, and living within your means B. Living within your means and using credit cards to help. C. Setting goals, budgeting, living within your means, and using credit cards wisely D. Living on a budget and using credit cards when necessary E. Living on a budget without credit cards. Use your knowledge from the text and the class discussions of Hitler’s strategies to suggest how he lost World War II. According to the text and class discussions, which of the following would be a reaction of an insecure/avoidant baby to the strange situation research technique? A. The baby would show great upset when the parent departed and would cling to the parent upon his/her return. B. The baby may or may not show great upset when the parent departed, but would cling to the parent upon his/her return. C. The baby would show great upset when the parent departed, but would ignore the parent upon his/her return. D. The baby may or may not show upset when the parent departed, but would ignore the parent upon his/her return. Common Errors Made by Teachers in Constructing Test Items Don’t give the answer away! Avoid these common errors. Test Construction Errors to Avoid • Don’t use “All of the above” or “None of the above” only when they are the correct answer. • Avoid distractors that are a joke or an insult. Test Construction Errors to Avoid • Make sure the stem of the question and the distractors/answer are grammatically correct • Avoid having one choice that is extremely longer than the other choices. Test Construction Errors to Avoid • If all the distractors are numbers, 90% of the time the correct answer is not the highest number or the lowest number. • Avoid using absolute determiners in the answer choices. Test Construction Errors to Avoid • Avoid using more inclusive options. • Avoid answer options that state a reason. Learning how to play the piano while singing occurs in the A. Affective domain B. Motor domain C. Cognitive domain D. Motor and cognitive domains E. None of the above According to the text, those who believe in the work ethic believe that A. All people will do good work. B. Hard work is always rewarded. C. Work is good and important. D. Only good workers should receive pay. The normal adult temperature of 98.6 degrees F is equivalent to A. 18 degrees C B. 27 degrees C C. 37 degrees C D. 42 degrees C Summary • Use three-part questions to make sure you are asking questions that cover all of Bloom’s Taxonomy • Write objectives to measure all aspects of Bloom’s Taxonomy • Have students create questions and grade questions • Use other methods instead of just paper/pencil tests to assess students in higher order thinking skills