Assessment and Feedback

Day 3: Crafting Quizzes, questions and feedback

Backwash on the beach

A key principle of assessment is that it meets the conditions of Validity (gauging if a student has met the objectives) and Reliability (gauging how much we can trust the results).  There is also ‘backwash’.  Hughes (2002) wrote: “The effect of testing on teaching and learning is known as backwash, and can be harmful or beneficial.  If a test is regarded as important, if the stakes are high, preparation for it can come to dominate all teaching and learning activities.  And if the test content and testing techniques are at variance with the objectives of the course, there is likely to be harmful backwash.” (2002, p. 1)  Later saying:  “Too often the content of tests is determined by what is easy to test rather than what is important to test.” (p.27)

Quiz Creation

An important thing to consider before you launch into writing questions is “How can I give my students opportunities to show what they know and how can they apply this?”  Aim to align your questions to the course learning outcomes – the knowledge, skills and attitudes they are expected to achieve during the course.  Below is advice offered in the ANU Med School’s Formative Assessment and Quiz Design course converted into an H5P ‘column’ (with text, an image hotspot and a drag text quiz question).

Raising ‘Reliability’Reliability

Many have written on the subject of reliability including Hughes (2002, p.44-50).  Tips for increasing reliability while also preventing harmful backwash can be summarised as follows:

  • Include a sufficient number of questions to give the student enough opportunity to demonstrate mastery but not so many that they get fatigued or distracted.  Where possible, ensure the effort put in is reflected in the points awarded.
  • Start with easier items so the student does not panic then increase in difficulty thus differentiating student performance.
  • Write auto-marked questions (and instructions) in a precise and unambiguous manner that does not give the answer away but allows a limited number of responses and discourages students from clicking without thinking.
  • Ask peers to take the test and identify any errors or potential confusion then address these. This may also bring to light alternative answers you had not considered.
  • If you opt to use the longer ‘Essay’ question type, you can provide a composition template so students do not stray too far from what you intend them to write and will be easier for you to manually mark.

 

question mark

Discussion

Have you experienced good and bad backwash from quizzes?  What do you think are the characteristics of good and bad quiz question or quiz design?  How would you go about planning for a successful quiz?

Question Crafting

A lot of these tips from or based around tips from Vanderbilt University for designing multiple choice questions can be applied to other question types.

Extra tip:  It is possible to create single gap-fills with MC questions but it makes more sense to use a gap-fill/cloze for this.  Also, it is better to use matching rather than MC if you intend to reuse the same distractors anyway.

Feedback

Yeaaah...I'm gonna need some mroe detailed feedback. Mmmkay?A great beauty of using technology to deliver online quizzes is that you can provide immediate feedback.  If set up correctly, students never have to wonder:  Which questions did I get right and wrong, and why? and How could I improve?

Everything you ask your students to do should be a learning experience.  Exams seem to be the antithesis of this but quizzes don’t have to be.

Here are some tips for providing effective feedback for your quiz:

  • Write specific feedback for each option (where appropriate/relevant).
  • Don’t just tell students the correct answer. If possible tell them why their answer was correct or incorrect.
  • Similarly, don’t just tell students the incorrect answer(s). If possible tell them why their answer was incorrect.
  • In some situations, you may wish to amalgamate all the feedback from each option and provide the same feedback whichever option is selected. This gives the students the opportunity to learn everything you have to say about each option.

For more on feedback see our previous Coffee Course, Feedback for Learning.

question mark

Discussion

How do you or would you provide quiz feedback to your students?  Do you have any other tips on crafting questions or feedback?

Resources and references

15 thoughts on “Day 3: Crafting Quizzes, questions and feedback

  1. I’ve definitely had some backwash from past quizzes. The most common I have found is that often the teachers will select to randomize the questions so that you can’t take the test in the same order as a peer (I guess this is to prevent working together or “cheating”?). While I understand this on one level, on the other hand, it does mean that you can’t have that nice increase in difficulty and instead seem to be swapping from one topic to the next indiscriminately. I can also definitely see how the number of questions or time taken for the quizzes needs to be closely considered. It seems to be a fine line between addressing all the relevant information and adequately assessing the students’ skills, vs. not losing their concentration or interest. Something that takes only 15-20mins to complete requires a lot more preparation to make it a success then I previously thought!

    I think the idea of providing feedback for incorrect answers is great. If you formulated the distractors in such a way that it would be the answer the student would arrive at if they skipped a certain step in their thought process then you could point where they went wrong and give more constructive formative feedback. This might also help in the construction of the quiz and it gives me a better idea of what I could put for some reasonable distractors!

    1. Hi Sarah,

      Thanks so much for sharing your experiences! Randomising questions and giving a time limit are indeed common methods to prevent cheating but it is interesting to hear about the bad backwash from randomisation. Ordering can be very important in a quiz and, as you say, it is nice to increase in difficulty and answer by topic rather than jumping around all over the place.

      I recently taught an academic how to build random Wattle (Moodle) quizzes. As I recall, she had 20 questions across three topics and she wanted a specific number from each. So I showed her how to create sub-categories and set up her quiz to draw at random: 3 questions from the first, 3 from the second and 4 from the last. If the overall quiz setting ‘shuffle’ is not selected, they would appear by topic without jumping around. If you made all questions in each category a similar difficulty from low to high then that would help with building up from low to high. Do you think this would be a good compromise?

      You’re right. It is tricky when there is a lot of content to cover in the quizzes but concentration may flag if the quiz is too long. I suppose shorter but more frequent quizzes might be the answer to this. Or you could break them up by topic and have a few quizzes available in the same period so students can have a rest and study between them. Do you think that would help?

      I’m so glad this course is giving you ideas on how you will design quizzes for your courses! It sounds like your use of distractors and targeted feedback will be extremely helpful for students! I’d love to hear how you go with that!

  2. Sarah, I hadn’t thought of this drawback on random assignment of questions ! I’ve been thinking of adding more quiz questions; but so as not to overload students, I would randomly assign the questions (I have 5-10 questions per weekly quiz). You are probably more experiencedon the randomisation of questions – I thought I could control the order (bummer).

    I am not so concerned about “cheating” : the quiz mark is not “worth” much, but The “doing” of the quiz seems to be helpful in getting a good mark in the exam. So I thought giving different sets of questions to different people might get students to collaborate /work/learn together. (Again,my quiz questions are often quantitative (with random inputs), so no two answers are the same – although the method will be the same).

    1. Hi Edie,

      Perhaps you could also use the random within sub-categories solution (as described in my response to Sarah just now) to give them some order. I just replied to your comment from yesterday and suggested you could create an ungraded quiz with all the questions (or a random selection) that is not visible to the student until the assessed quiz has been closed and they have completed it. That may be a way of letting the super keen students have a go at everything as many times as they want. What do you think?

  3. Like any assessment, good quiz design should constructively align with the Learning Objectives and activities and requires appropriate planning. The main bad backwash I’ve experienced was when students realised they could copy-paste answers, instead of learning from/through the quiz. I realised this was due to poor design, and attempted to address this by creating higher-order questions in subsequent quizzes.

    Randomised sub-categories have been fabulous for our quizzes. My sub-categories were 1) auto-markable (usually lower-order), 2) short answer (middle-higher order), and 3) extended answer (higher-order). This way students got randomised questions, there was a progression, and best of all, students got the same number of the same types of questions. Took a bit of planning, but was about as equitable as I could muster on a first attempt at quiz building.

    One thing I would like to change is finding a way to say if a student received question X in one sub-category, then they would be excluded from getting question Y in the next sub-category. It didn’t happen very often, but sometimes there were similar questions in different sub-categories. Students who got both of the similar questions may have had an advantage. Another potential backwash, I suppose.

    In crafting feedback, the correct responses usually explain why it is correct (although, I am guilty of occasionally simply writing “That is correct”). The incorrect responses for the formative quizzes link students back to the relevant readings and course materials. In terms of summative quizzes, given so many of our questions require manual marking, I tend to give personalised feedback. Time-consuming, but those type of questions best align constructively with our LOs.

    1. Hi Bhavani,
      It sounds like you’ve done a marvellous job with your quiz building so far!
      Unfortunately, there is no way to set up flows to stop the student getting a certain question from one subcategory if they have already done a similar one in another. All I could suggest is to have similar questions in their own subcategory and choose to randomly select one from it.
      I’m glad to hear that you provide lots of feedback on your quizzes! When you’re creating essay-type quiz questions, you can provide general auto-feedback so you don’t have to repeat the same info when you manually mark the items. For instance, you could put in a suggested answer for the student to compare with their own. This could potentially save a fair bit of your time. Do you think a combination of general and specific feedback would work for those kinds of questions?

  4. Hi Rowena,

    Definitely something I need to explore more. We give both general and specific feedback for essays, so would be happy to incorporate this into our quizzes as well.

  5. I chuckled while reading through your tips on how to formulate questions. Only yesterday did I take a multiple-choice test that had a few of your ‘avoid’-tips in it. For example, it asked ‘Which of the following does NOT apply to …?’. Which was really tricky, because your natural instinct is to read a statement and think ‘Yeah, that’s true, so I’ll tick it’. The quiz also had a few questions that had ‘All of the above’ as an option. As soon as I had read that, I immediately checked if at least two answers were true and ticked that option (and found that it worked). So the former made me a confused student, while the latter made me a lazy student. Definitely reinforced that I would want to avoid doing this when I craft multiple-choice questions!

    1. Melde I like the point you make about this sort of question structure not fostering effective learning practices in students – students spend more time interpreting how the questions are worded than actually thinking about the content and what it means!

  6. I have included immediate feedback after every question — with the wrong answer, a clarification/tip and request to try a gain and with the correct answer, congratulations and a rewording of the question and answer to repeat the right content, once again. In the end of the quiz, I have provided an overall assessment and also another overall assessment across all the 10 quizzes my unit has.

    1. Hi Hanna,
      Feedback is something that lots of people overlook when creating quizzes so it is great that you have paid so much attention to this. I’m sure your students really appreciate the time you have put into providing encouragement and tips to get them back on track!

  7. Hello All,
    Hanna, we seem to have a similar approach 😉
    I include immediate feedback after questions and write specific feedback for each question (where appropriate/relevant) telling them why their answer was correct or incorrect. In the beginning when I started setting quizzes I didn’t include feedback and students would email asking for clarification for their own thinking as to why their answer was right or wrong. But sometimes the emails highlighted that the question design needed some improvement. Through trial and error, and I have not mastered this yet, I have developed a better approach to question design that ranges across the different levels of “Blooms Taxonomy” (apologies my previous post said DeBono!). I am sorry to have stepped into this coffee course late and miss the associated workshop!

  8. This course has made me realise that I can do a lot more with quizzes than I previously thought! The idea of randomised subcategories seems like a really good strategy for assessable quizzes. I also really like the idea of providing explanations in the auto-correct feature. I can see that this can really help to make the exercise a valuable learning experience regardless of whether the student gets the right answer. In itself this might also encourage students to complete low-stakes or ungraded quizzes.

Leave a Reply to Melde Cancel reply

Your email address will not be published. Required fields are marked *

*