TwT Brown Bag - Writing Effective Assessment Questions & Technology Considerations - August 2022
[Dr. Kurzweil] I'm Dr. Dina Kurzweil I'm an assistant professor in the School of Medicine and director of the Education and Technology Innovation Support Office. Today, I'm very pleased to introduce Dr. Lynnette Hamlin, Dr. Linda Macaulay and
Dr. Karen - she goes by Beth Marcellas, who will be joining us today to talk about these topics. [Dr. Marcellas] Thanks, Dr. Kurzweil. Okay.
Well, I will let folks read the objectives here for themselves and talk a little bit about the origin of this session. So you may be wondering why we're talking about multiple choice questions in "Teaching with Technology." And Dr. Macaulay who will be talking a little bit later. Is someone many of you may be familiar with. She does a lot of the Canvas training and she's been doing some Canvas training on assessment and the quiz tool in Canvas.
And she ended up getting a lot of questions, not necessarily just about the Canvas tool, but actually about writing effective questions to put into that tool. So that led us to decide to do a Teaching with Technology Session, specifically about writing effective multiple choice questions, since that's what you're generally going to be using in your Canvas assessment. [Dr. Hamlin] Before we dig into some of the micro on writing effective test questions, I'm going to talk a little bit about aligning it with the goals of assessments in our courses a little bit, kind of not kind of about test blueprints.
So, you know, when we do our assessments, what do we want our students to know when they leave this course? So aligning the objectives for each exam or assessment with both the course objectives and the class objectives for those classes that you are going to do your assessment on. And then once you look at your learning objectives, what kind of assessments are you going to do? So how can you line up your assessments that will tell us faculty, that students have achieved those learning objectives that you've identified in your course and class objectives. Giving some pre thought to this for your assessments helps us insurer as faculty that our teaching learning goals and the assessment practices we use in our classes are clear to our students so that they know what to focus on as we present the content and what they're going to need to excuse me, know and learn as they demonstrate their learning. So one of the first pieces of that is what our learning objectives are and what level. And you'll see some of this is looking familiar. So we're asking our students to recall, recognize or identify information.
Objective test items are good types of assessments to meet those learning objectives and different types between multiple choice can be fill in the blank matching and labeling, so it helps them recall terms, facts and concepts. If we're asking our students to interpret, exemplify, classify, summarize and further compare or explain activities such as exams that require students to summarize the content, compare and contrast, classify or categorize, paraphrase, find examples for me as the clinician, asking students to apply their learning into clinical scenarios is one type of piece that I've been using. Thank you. [Dr. Marcellas] Okay. Thanks, Dr. Hamlin.
So now let's take a look. Dr. Hamlin kind of talked to you about a couple of different types of things you can do with an exam. And I wanted to take a look at the anatomy of the multiple choice questions so that we can all make sure we're on the same page, just in terms of terminology. And what we'll be talking about today during the session.
So you can see an actual multiple choice question here on the page, and you can see there's a couple of different parts that I've highlighted. So the first box is a vignette. And that's that's where not every question is going to have one, but that's where you'll give the background and the case study and the any kind of information that the students might need for these kinds of higher order questions. If all you're asking is recall, you may, you probably don't need that.
You'll just have this second box, the leading question stem, those types of things. So those are two of the parts of the question. And then you also, of course, have to have answer choices that's important for the students. But also you should think about the fact that your students should really be able to answer the question even without seeing the answer choices. And by that I mean you want to make sure you're actually asking them a question that has a clear a clear question with a clear answer.
So if you can if they can look at that question and at least have in their mind, oh, the answer to this is X or the minimum. Oh, the answer to that is a body part. I don't know what it is, but I can tell that that's what it should be.
That's how you know you're on the path to question. Or if you can't do that, you're certainly on the path to a bad question. So before you even get to the answer choices, you'll you'll have a clue as to whether it's a good or a bad question. But again, you have a list of options and answer choices. And then so you'll have a correct answer, which you have here highlighted in red.
And then you have some distractors or incorrect answer choices, which you see here highlighted, which are not highlighted in red. They're just black. And you can see the distractors identifying them. Now, as you look at this, you can think about as we and we as we start to think about writing good, effective assessment questions, multiple choice questions. If you think about it, there's really kind of two ways that the writing of these types of questions can make them ineffective. First off, if people can easily answer the question correctly, even if they don't know the answer, that is a poorly written question.
Or if people who do actually know the answer are unable to answer the question correctly because of the way it's written, that is also a problematic question. So I'll be talking today a little bit about some of the things you can do to try to make sure that you don't have either of these problems with your questions. Again, questions should design so that students who know the material can find the correct answer. So again, I'd advocate avoiding trick question questions that are designed to lead students to an incorrect answer.
That's that's really kind of defeating the purpose of the exam. You really want to know in the question. You want to know that students can show you what they know. And trick questions are really kind of just, you know, making sure they can read the question correctly or there's there's something else going on with this. So we want to avoid those in good exams.
So first, moving to talk a little bit about some things, characteristics of a good vignette for an assessment question. The important thing here is that you give in your vignette only the facts needed to answer the question, and that's going to depend on the type of question. It is obviously in a sort of a a case based question. And there's a lot of information that could be in there, but you really want to pair it down to what they need to answer the question correctly. So, only the needed facts. If there's clinical data,
you want to include units so you don't get them confused -- is this liters, milliliters, there's going to be there's going to be a big difference. Obviously you're probably not using liters but different units or it's going to make a difference in their answer. You may need physical findings. You may need results of diagnostic test. All these are going to help the students get focused on the right answer.
Another thing to keep them focused is you don't want to use names. Generally, there's not really a need to use names in the question unless it's some sort of a dialog where you want to show a doctor and a patient speaking because names can be a distraction. They may make people think about the person rather than the the question or condition that you're talking about.
So try to keep it as generic as possible. Certainly don't use a real patient name with real patient details. But again, not using a name and even gender, race, those may be things that will affect the answer choice. If they are, you can include them, but if not, you want to leave those out because again, those are going to distract people, make them think either differently about the differently about the question itself. If it's not relevant, you shouldn't include it. Okay.
So now let's take a look at some things that can help you write a good question or Stem. So first, as I said before, make sure it's a question is it's just something like something more abstract, like "what do you think" or or, you know, "what's what's going on here," that's a little bit vague. So you want to be as specific as possible. And also, especially now as we're talking, all of our students are advanced level students.
You want to keep focused on on really higher order questions. You don't want to use true false questions at this level for a couple of reasons. First off, because they have a 50/50 chance of getting it right, even if they don't know the answer.
So that's always problematic. But even so, when you write a true false question, you have an answer or a specific answer in mind that your readers may or may not have the background knowledge, or they may not know what you know. They may not really be thinking in the same line you're thinking.
So they really kind of have to guess what you had in mind. And so that's really going to cause problems. So you really want to try to avoid something like a true false question because there's too, too much vagueness in it.
So multiple choices are really helpful because you have clear, generally defined answers. Also, you want to avoid imprecise terms in your question. Like usually, generally in the question you see here, even most likely might be a little bit imprecise because again, different people maybe may have different ideas of what that means.
So that's going to abstract in the question. If you have this imprecise term, that may affect the student thinking about the answer. So you want to avoid that.
Also avoid negatively phrased items. So you don't want to say, you know, which of the following muscles is not most likely. You as manager for a couple of reasons. First off is especially if most of your questions are not phrased that way. If you throw in one that's phrased with a "not," students are likely to overlook it. Right? So you don't want to throw them off.
It's going to make it easy for them to miss the question simply because they didn't see that key question, that key part of the answer. So instead, you'll want to try to rewrite the question so that it's a positive choice rather than a negative choice. So yeah, so if you do use a not if you use not in your answer, in your question, you absolutely need to highlight it, make it big, bold, red, but you really want to avoid it if at all possible. So answer choices. Let's take a look at what you can do to make effective answer choices.
First off, one of the there's different schools of thought about how to order your answer choices. But one of the one of the rules I've learned that I've always followed is you want to list them in alphabetical order or numerical order, depending on the type of question, for a couple of reasons. The biggest reason is then the students, you know, we talked about whether students can figure out the correct answer, even if they don't know it. And if they're if the answer choices are not in alphabetical order, the students might be trying to look at that to see whether that's giving them an answer choice. And I know before I learned this rule, if I was writing a multiple choice test, I would sit there and say, Oh, wait, the last three answer choices were C, so I need to make sure this answer choice is D because I've had too many C's in a row and I would I would reorder the answer choices based on that. If you use something, if you use multiple choice or numerical choice for numerical order if you use alphabetical order or numerical order for numerical questions, then students will notice, okay, everything's in alphabetical order.
I can disregard the order of the answer choices because that's not going to give me any information at all. So it really helps you say that's one less thing you have to worry about. Also ensure that there is, as I said before, only one correct answer so that the students know exactly what they should select. You want to avoid overlapping choices, which may happen in a couple of ways in numbers you might you don't want to say 1 to 5 and 5 to 10, because then if the answer is five, they don't know which to choose. You know, any overlapping number sets will will cause problems can also be a problem.
For example, in anatomy, if you have a map, anatomical structures with larger structures that contain smaller structures. And then what if you're if you're mixing different types of things and one structure is included in another, that's that can cause problems. So definitely make sure that you have one clear, clear answer each time and that the answers do not overlap. And then you want to avoid including terms like always or never in your answer choice, especially if it's only in some of them, students will immediately discount those because it's it's rare that you're going to have something that's that definitive.
This is you know, this is this is always going to be a diagnosis of X because whatever or, you know, students just disregard that. They realize that they're it's less likely to be true than anything that doesn't have that kind of absolute term. Also, you want to avoid all of the above or none of the above questions.
Again, for a couple of reasons. The biggest one is that if you have an all of the above answer choice and depending on how many answer choices you have, you may have a lot. You may have a few, but if students see more than one answer that they know is correct, no matter how many answer choices you have, if you have all of the above, they know that that must be the answer. So they don't even need to go through the process of of looking at the answer choices.
So you don't want to do that. You want to make sure they're considering everything. And similarly, if it's a none of the above question and they see one that they know is false, and that's they see two that they know are false, then they know that none of the above is the correct answer as well. So all of the above and none of the above allows students to who don't have a full knowledge to select, which based on partial knowledge, they can sometimes get the correct answer.
So again, that means they didn't they didn't necessarily know the right answer, they just guessed it. And that's a that's an ineffective question. And again, none of the above actually turns it into a kind of a true false a true false test. So series of true false question. No, we already said we don't want to do that.
All right. So the big question that comes up when we're writing multiple choice questions is how many answer choices do I need for a multiple choice question? And to show this, I wanted to to do a question that I personally wrote. So you'll see it's not really a higher order question because I am not a medical professional, health care professional. So I wrote a question just to kind of show you what happens when you start writing too many multiple choice answer choices. So here's a few.
They look like they might be reasonable and then suddenly, well, I got to four and I could not figure out what to do. So I just I just threw in Dracula. Now, you know, it looks like some of the other words.
So, so for for someone with my level of knowledge, it might actually might actually work. But generally, that's that's not going to be effective. In fact, research suggests that three answer choices is sufficient.
There's studies, the literature reviews, the reference on this page is actually a review that studied a lot of different research on multiple choice questions. And there's no significant difference in terms of student performance, whether there's three answer choices given or four answer choices, five answer choices. You don't need to feel like you should come up with five answer choices or even four answer choices.
If they're not good answer choices that it's not going to distract the students and it's not really worth your time. So three answer choices is plenty as long as they're good ones. If you want to do more, you can.
Extra credit. Anyone? Anyone knows the picture on the right? Anyone know why we have that on here? Any any Monty Python fans. The chat got a this is actually the holy hand grenade of Antioch. If you've watched Monty Python and the Holy Grail, you know the number of the answer choices should be three.
You can actually go a little bit higher than three if you want, but but not fewer because that would make it a true false question. So three answer choices is plenty. A couple more things to talk about with answer choices before I let you all attack some questions on your own.
First off, take a look at the question. Just go ahead and read through the question and read through the answer choices. And I'll talk a little bit about about the question after I've given you a chance to read through it. All right. So as I said, I'm not a health care professional, but even not being a health care professional, I can read this question and say the first step in management should be intravenous administration of so I can discount and see right off the top. They're not grammatically consistent with the stem, so that's an important piece.
People can discard answer choices if they're not grammatically consistent with the STEM. I might, if I do a little bit more, know that some of these wouldn't be administered intravenously. But still, the important thing is that they're they're inconsistent with the question itself. Also, you want your answer choices to be homogeneous. So as you can see, some of these are procedures.
Some of these are are tests, some of these are medications. So you want each answer choice to be sort of the same category so that you're not letting people discount again, if they know if they know that the answer has to be something in some sort of thing, like a medication, and you include an answer choices that aren't medications, then clearly that's going to be wrong. So you've just made it easier for them to get to the question, correct without knowing the answer choice. So they want to be homogeneous.
They want to be similar in length. So these are all important things as you look at your answer choices. These are not too bad in terms of similarity in length, but definitely the two medications at the end are a little bit longer or a little bit shorter than some of the others. So that that might let people give people some clues, the different answer links of the answer choices. All right. Let's take a look at another question.
This is a little bit longer, so I'll give you some time to read the question in the answer choices. Okay. Now, this one, it's not quite as obvious. And let me tell you the correct answer choice is D. And if you read through the question, you can see that there's the there's some information in the question about heating. So,
again, an intelligent, clever test taker could read through and say, well, I see heating in the question. I see heating in the answer choices. So I'm going to guess it's heating. This is this gives them a clue that that the answer choices gave them a clue because it reflects some of the answering the question, some of the information in the question. So you want to make sure that your answer choices are not using similar language to the to what's in the question that might give the students the answer.
And that's true even if you are using a different language, right? If you're using sort of Latin in the answer choices and it's in English in the question or vice versa, you know, your students are going to know enough to recognize the Latin roots and stems and things so, So that doesn't necessarily get you out of it. You want to make sure that your your language in the answer choices don't mirror in any way the language of the questions in in either either language. So I want to move on now. I think I've given you most of the tips, and I want to take a few minutes to go through a few examples so now folks can type into the chat for the next few slides. I'm just going to put a question up that you all can type into the chat where any comments you have about the question answer choices each of them should have.
At least one of the issues that I have discussed in the course of the presentation. So go ahead and type in type into the chat. What you see is wrong with this question. It's like people have done it. So first off, it's it's not really a question. Good point.
And then the language in some of the answer choices, C and D, you can see that people are going to discount that. So you're down to three answer choices, which is fine. Not it's not a terrible thing to have three answer choices, but you didn't.
The person writing the question didn't really need to spend time on writing answer C and D because they know the students are going to write those out. So yeah, good job. Let's take a look at another question. So take a quick, quick look at this question. The answer choices... Right. So you see the answer choice is none of the above.
You don't want to do that. Dullness is used twice that can be confusing that people the answer choices aren't entirely distinct. So yes, some of the language is a little unclear. One other thing is that the first the first answer choice, this is a little bit more subtle, but the first answer choice has two different things, two different content items. Included in the rest of them only have one so that that can throw people off a little bit as well.
So some so yeah. All right. Let's take a look at look at a couple more questions and then I'll let Dr. Macaulay take over for a minute. All right.
Example three But what issues so you see with this question? It or the answer choices? We see a few people and again, you can see just from these examples, it's really easy and a lot of people do not do just omit the question itself. Right. A lot of people will just sort of say, oh, people know what I mean, right? I have a statement and people know what the question is based on the statement. But that's not really that's not really effective.
Your learners are really going to be focused and they may not know what you're asking in that question. So if you don't put an actual question in, it's going to just increase the cognitive load and make it harder for them to to use the exam to demonstrate their knowledge. It becomes more of a guessing game for them as to what exactly you're looking for.
That's usually used in here, right? So say language. And the other thing, just because I was an English major once upon a time, I always notice. I always notice the parallelism, right? Severe obesity in early adolescence, poor prognosis that might be kind of a one of those ratio type questions or the but it's not an actual poor prognosis doesn't grammatically follow like the rest of these things right and 75% chance that's that's a good point. The 75% chance seems very specific. Right. So again, we talked a little bit about having the answer choices be similar.
And that's sort of, you know, are most of these are are just verbal kind of talking about prognosis and things. And this is suddenly you're getting into percentages and and something that makes this one pretty different than the others. So people are likely to to not get the answer. That's going to look very different to them. And again, one of the things I found when I was looking for sample questions to use here, since I don't know all the ins and outs and I wanted relevant medical and health care questions, a lot of times they don't tell you what the correct answer choice is.
The sources that I found. And so I don't actually know what the correct answer choices for this one. I just know that there's a lot of issues with the answer choices that are provided. So take a look at one more example. So for this one, right, choice C is not homogeneous lack of specificity.
So yeah, this one, the correct answer is actually B I did when they did give me the correct answer for this one, which is interesting because I know from you know, as we talked about, you want you want the answer choices to be similar. And the correct answer is the only one in this case that's different from the rest. So that's you know, that's something to keep in mind as well. You're you're you want to make sure that the answer choices have enough differences between them that people aren't going to notice a difference that doesn't actually they're going to notice something is different but it's not it's not actually relevant. That's a problem. So you want to use that similar language as we've talked about and also I manipulated some of the answer choices a little bit from the original source and folks may or may not know just from looking at it from research, vowel sounds is a subset of lower pitched sounds.
So that's I think someone identified that lower pitched sounds was was not particularly specific. So again, these aren't necessarily distinct answer choices. There's a lot of a lot of kind of noise, as it were, going on in here that's going to potentially throw some students off. So anyway, I'm sure we could we could go on all day with me putting putting questions up and you all tearing them to shreds. And you're welcome to you're welcome to spend some time looking at your own exams now. Now that you know some of the tips and tearing your own questions to shreds.
But at this point, I'm going to hand the microphone over to Dr. Macaulay so she can talk a little bit about test security. [Dr. Macaulay] So once you have your perfectly written multiple choice questions, you also want to make sure that you consider test security in your test settings, particularly if you're using an online platform like our learning management system.
So we wanted to spend a little bit of time to go over some of these pieces of information, keeping in mind that we do have that session on September 1st at 9:00 in the morning, which will go into a lot more detail than this. But this is your 10,000 foot overview, high level of the types of things you're going to think about. So definitely take a look at those quiz settings that you have in your learning management system.
Look at things like shuffling your answers, adding a time limit, allowing multiple attempts or not showing one question at a time or not, because all of those things in concert with each other can add to your test security. Because if we shuffle the answers, then students sitting next to each other likely won't have the same test. If we put a time limit. They can't look up answers as easily, but I will caution you make sure that time limit is reasonable so that you're not testing their speed with which they can take the test instead of the content they're supposed to know. Also, considering that multiple attempts, should they be able to do it more than once so they actually can show mastery or not? And then really the most important one is the bullet point here, that says "let students see their quiz responses in Canvas." It is checked by default.
So that means whenever students take a quiz in Canvas, automatically they will see their quiz responses after they submit the quiz. So if you don't want that, I would recommend checking those quiz settings and leave that particular setting unchecked until after everyone takes the quiz and you're ready to review the answers. Then you can let them see the responses. You also may want to use question or item banks so that students are seeing different questions on their test, specifically, if they're using multiple attempts, that's a good way to make sure each attempt is unique.
It also helps students sitting next to each other in the same room not have the same test as well. You can use availability dates to make sure that the test is only visible when you want it to be visible. So during that testing window, only not before and not after. And you also want to consider potentially adding an honor statement as your first question on the test.
You can actually put a statement that they have to respond to, that they are the person taking the test and they only are the only person checking test. They did not receive any assistance or something like that as they completed the assessment. So that way you are more assured that the work is their own. [Dr. Kurzweil] Thank you for watching today's Teaching with Technology Brown Bag Session. You can view all teaching with technology Brown Bags on the USU YouTube channel under the full ETI Playlist.
Have a nice day.