Assessment and FeedbackQuiz Design

Day 2: Tour of Question Types

Question Mark
Image by Barney Moss

Today weā€™ll take you on a tour.Ā  In part one, we’ll explore the most used question type and its variations.Ā  Now what could that be?

You guessed it:Ā  Multiple-choice (MCQ)!

Then we’ll proceed with the second part of the tour exploring other lesser used but valuable question types.

But before we start the tour, letā€™s have a look at some common terms weā€™ll be using by completing this H5P (drag and drop onto image) matching quiz question.Ā  Don’t worry!Ā  You don’t need to know anything about anatomy to answer this question and you can have another go if you need to!Ā  So have a go and if you would still like to see an explanation, hover/click here.

All aboard for our two-part tour!

These two ‘Presentations’ created in H5P will show you some working examples of question types and offer tips on when and how to use them.Ā  The first tour contains questions that should be super easy to answer (just to show the question types) and the second tour has questions that are slightly more taxing.

If you have a small screen, you may wish to click the double ended arrow to make it fullscreen then click it again (or press Esc) when youā€™ve finished to return to the post.

 

MC Hammering

While concluding that standard MC questions are best for occasional testing of large cohorts, Hughes (2002, p.76-78) listed these drawbacks of multiple choice (in bold with extra details underneath):

  • The technique only tests recognition knowledge
    We should not only focus on receptive skills and neglect productive ones.
  • Guessing may have a considerable but unknowable effect on test scores
    Having four or five options will decrease this.
  • The technique severely restricts what can be tested
    Absence of viable distractors may mean important aspects are not tested.
  • It is very difficult to write successful items
    Many assume that because they are easy to score they are also easy to create then end up with ambiguous questions or ones that give the answer away through poor stems or distractors.
  • Backwash may be harmful
    Unless students get feedback and pay attention to it, the quiz may reinforce misconceptions.
  • Cheating may be facilitated
    It is important to consider the use of settings to randomise questions, shuffle questions and options within them as well as restrict when students can access the quiz and for how long.

Good thing there are variations and other question types at our disposal in part two of our tour:

 

question mark

Discussion

What did you think of our two-part tour of question types and tips?Ā  Which question types or ideas presented are you keen to use in your teaching?Ā  Do you have any question types or ideas to add?

 

Learning through quizzing

Owl with one eye closed

Auto-marked quiz questions lend themselves to experimenting in low or no-stakes quizzes where the purpose of the quiz is to learn by having a go.

Instead of giving students a list of terms and definitions, images and labels, words and equivalents or phrases and translations to passively absorb, we can get the students to make educated guesses and their retention will be much greater.

The highly popular (and gamified) language learning app, DuoLingo essentially teaches through quizzing.Ā  People clearly donā€™t mind not knowing all the vocab and grammar before being quizzed on them.Ā  They actually embrace the challenge.

Over the last six months, the author of this post, Rowena, has been enjoying the process of learning Indonesian in her free time via a number of platforms including DuoLingo and the online version of The Indonesian Way which both use a variety of quiz question types to great effect.

question markDiscussion

Can you tell us about your experiences of learning through quizzing?Ā  Have the quizzes you’ve encountered (as a student and/or educator) had a good variety of question types?Ā  How would you like to use quizzes in your courses?

 

Want a quiz knight to come to your rescue?

Come to our Wattle Quiz Building Workshop!

For those of you at the ANU, we would like to invite you to a Wattle Quiz Building Workshop which will be held 10am – 12pm Thursday 5 September 2019.

Weā€™ll guide you through how to:

  • create a variety of question types using the Moodle (Wattle) Quiz tool
  • categorise questions
  • tweak quiz settings and scoring

We hope to see you there!Ā  Click here for more information and bookings.

Resources, readings and references

Clearing up Question Name Confusion

Moodle

H5P

The activities featured in this course were made using H5P.Ā  The ANU is planning to activate the H5P plugin for Wattle later this year.Ā  It is hoped it will help teachers to create formative online learning activities for their students.

28 thoughts on “Day 2: Tour of Question Types

  1. Sorry, I posted my answer to the first question on day one, instead of day two. I went back to see if I missed the bit about what a “key” was. To answer the second question, I find the idea of “learning through quizzing” unacceptable. Having students make “educated” guesses seems to be a way to discriminate against, and humiliate, those who do not have what the teacher considers the “correct” upbringing. As someone who can’t spell, I was subjected to this form of systematic humiliation throughout my school years. It is not something I wish to inflict on my students.

    1. Hi Tom, thanks for sharing your experiences with quizzes – I’m really sorry to hear that happened to you, it sounds awful! I’m wondering if your experience is something that others have found as well? I’d love to hear more from the other participants about how they have found ‘learning from quizzes’ as a student?

      One of the things I am reflecting about, reading your comment, is the scaffolding and intention behind the use of ‘learning from quizzes’ – how we as teachers set up and introduce quizzes, and incorporate them into a wider teaching format and style. I have had some positive experiences when I was learning languages where we had ungraded formative quizzes done using technologies, where I could click as many times as I wanted and get some feedback on each response. From the safety of my own home, I could anonymously try and try again without any repercussions. What about everyone else?

    2. Hi Tom,

      No worries. Iā€™ve responded to your comment there too. It is truly awful that your school days were so traumatic. That must have been very hard for you. I completely agree that teachers should never single out students and quiz them on things they donā€™t yet know in front of others. That is guaranteed to make them feel small so we must never inflict that on our students. However, that is very different to the instantly-marked (very low or) no-stakes quizzes I was talking about in the ā€˜Learning through Quizzingā€™ section.

      As I mentioned in the post, I am studying Indonesian at the moment and I use ANKI flashcard sets (made by other users) on my phone to learn and test myself on key vocab. Sometimes I can work out what words are (eg. ā€˜informasiā€™ is obviously ā€˜informationā€™) but I donā€™t know most of the words at all because Iā€™ve never seen them before. However, as I cycle through them, (first Indonesian to English then vice versa) I end up remembering them all. Obviously it would be very stressful to have to do this as a high-stakes assessment or in front of others but in private with no such pressure, I can take all the time I need until I get them all right.

      Similarly, as Iā€™m going through the quizzes on DuoLingo and The Indonesian Way, I donā€™t feel disheartened when I donā€™t get everything right the first time because nobody needs to know and I can always try again. If I were completely lost and answering purely at random, that would be pointless but if I know enough to get most things right then that reinforces the things I do know and I’m able to have another go at the ones I donā€™t.

      When I taught English to adult migrant and refugee students, I designed, created and evaluated my own non-assessed suites of web-based, self-paced activities based around touching and amusing stories. I created them with the now lesser used but remarkably versatile ā€˜Hot Potatoesā€™ quiz maker which allowed me to tweak the code and come up with an even wider variety of question types. When I trialled them with my classes (with each student at their own computer), I loved seeing the look of delight on my studentsā€™ faces as they completed the exercises and they were perfectly happy to have another go if they didnā€™t get everything right. They were not being assessed but they were clearly learning through quizzing and having fun while doing it.

      I would not suggest that people should rely entirely on learning through quizzing. I doubt anyone would claim that you can become fluent in a language by using DuoLingo alone but it is certainly helping me improve while using a variety of other inputs.

      1. Rowena, my school days were not that bad, but it does make me want to do better for my students.

        Ironically, after doing very poorly in languages at school, I find myself teaching 300 international masters students how to write a job application.

  2. I particularly liked the final example in the two-part tour of question types as I have never seen the method of chronological ordering steps before. I can see it being very useful in multi-step math and physics problems! While I understand there are some pitfalls and limitations of quizzes that need to be accounted for, I think they can generally be a useful tool to check-in with where the students are at. Previously when Iā€™ve been a student using quizzes in assessment it was only worth a small fraction of our grade (~1% of our grade for each weekly quiz) which I think was good as it put less pressure on us doing well but still gave us some incentive to do the 20min quiz to test our knowledge. While Iā€™ve taken a few quizzes, Iā€™ve never created one before and it seemed quite daunting so Iā€™d definitely be keen to come to the workshop!

    1. Hi Sarah – I had the same experience the first time I had to create a quiz myself! Despite having been subject to them for years and years as a student, I really struggled to put together the questions myself. Coming up with reasonable distractors is the most challenging part for me – finding alternatives that were plausible enough but not toooooo similar was really difficult! Definitely register for the workshop if you are keen – it’s not until September so registering will ensure we can send you reminders / updates as the session approaches. šŸ™‚

      Here are the details: https://services.anu.edu.au/news-events/wattle-quiz-building-workshop

      1. Yeah, the distractors seem to be the most difficult part of quiz design to me so far. However, Day 3 has already given me some good ideas on how to make effective distractors while also being able to provide informative feedback to the students. I’m primarily involved with maths and physics courses where the questions for the quizzes will typically be to calculate something, so by completing the questions missing certain steps and getting the “incorrect” answers could be an effective method of distractor creation while also enabling me to provide a note when the student is marked wrong saying which step they missed so they can learn for the future!

        I would love to go to the workshop but I just realized that I’ll be out of the country on the 5th of September! šŸ™

    2. Hi Sarah,
      Iā€™m so delighted the tour exposed you to a new question type! It is great to know the possibilities so we can add more variation to our quizzes and really help our students get to grips with what we need them to understand. Iā€™m also glad your experience of quizzes has been positive. Low-stakes quizzes are definitely a handy way of gauging if our students are on track. I really look forward to meeting you at the Wattle Quiz Building Workshop! The Wattle Quiz tool may look daunting but once you know how, Iā€™m sure youā€™ll soon become a quiz whiz!

      1. Hi Rowena, I’d really hoped to be able to make the wattle quiz workshop, but unfortunately, I’ll be out of the country! Do you have any sources you can recommend for quiz building? This course is already giving me lots of ideas and I’ll make sure to keep checking out the additional resources and references too. šŸ™‚

        1. Hi Sarah,
          Oh no! What a pity! There are some resources for building Moodle Quizzes in Day 3ā€™s list. Iā€™ll definitely be sure to let you know if I come across anything more as I prepare for the workshop and put my own resources together for it.

  3. I use weekly wattle quizzes in Two science courses (physics and chemistry), and these are a mixture of definition and logic, but mostly quantitative quizzes. To avoid the pain/anxiety of wrong answers, I give students several attempts to answer and wirh each wrong answer, I give feedback (or hints). In the first few weeks sphere is no penalty for successive attempts, but later I make 10% deductions for each wrong attempt – and with each wrong attempt, I give them a step in the solution. Students seem to like the quizzes – some email me if the quiz is released a little late….even though it counts a very small fraction of their course grade. The quizzes are administered along with lecture videos and both video/quiz must be completed before workshop.

    We also made the HPO/ASE portion of one course based upon Wattle quizzes – however, some of these require manual grading, so they are not as automated.

    Next year I am hoping to have enough quiz questions so that I can assign students a random set of questions per quiz – (I already use a random number generator in my quantitative questions). Some students just eat these up and I am guessing that they will work together so as to see more questions.

    Sorry for late submission – I am travelling,,,

    1. Hi Edie,

      No need to apologise! These posts go out on particular days but anyone is free to answer at any time after that.

      Your weekly Wattle (Moodle) quizzes sound great! Iā€™m sure your students really benefit from the feedback and hints! Youā€™ve obviously thought about them very carefully. I reckon your students would also appreciate not being penalised for wrong answers in the first few weeks and Iā€™d imagine the small deduction and clue each time works very neatly and prevent wild guessing. It is a great sign that students are watching your Wattle site to see when the next one is released. Do any students ever fail to do the quiz before the workshop? What happens if they donā€™t complete it on time?

      Yes. The manual marking of ā€˜Essayā€™ questions can be time-consuming but sometimes questions we need to ask that simply cannot be automatically marked.

      It is very clever that you have the random number generator in your numerical questions! It sounds like many of your students would be keen to try out all the questions! It is actually possible to tweak the settings to set up the quiz as youā€™ve described above and then have another quiz with all the questions in it that is not visible until the first quiz has closed and they have completed it. This quiz would not count towards the final grade but it would allow students to have a go at all the questions. Iā€™m about to post more about randomising questions in the comments for Day 3 so have a look there too. Hope youā€™re enjoying your travels!

  4. Yes Rowena, A few students donā€™t complete the quiz by the deadline. That means they cannot review or even see the questions, let alone the solution to the questions. Most will ask me to re-release the quiz for them – and I ALWAYS do that, but in fairness to others, they get no mark for that weekā€™s quiz. However, everyone knows that the ā€œdoingā€ is more valuable than the marks garnered from the quizzes. And to emphasise that, the mid-term has at least one question very similar to a quiz question. So if students have a CRUNCH time, I tell them not to worry – they can still get the quiz re-released and will not be disadvantaged further (but they dontā€™ get Quiz points…). This is simply keeping to the aim of the quiz – to keep up with the material. The points are only to reinforce a timeline….

    1. Hi Edie,
      That sounds very sensible. Thanks very much for explaining your system! It is great to share so others can benefit from your experiences!

  5. We try to frame all assessments as learning opportunities. Our quiz questions are designed to facilitate learning, rather than to trick students. As such, we design our quizzes such that a student could technically get 0% and yet have still learnt something. This is because every single question provides information or reiterates key concepts and terms. Convincing students of our pedagogical approach is a whole other ball game.

    Could we get a similar tour, demo, or step-by-step instructions on how to actually set up the question styles requiring coding? I have always avoided them on Wattle, limiting the options I can use for my students. Despite reading the Moodle docs, etc, I still lack confidence. Alternatively, I hope these less-common question types will also be explored in the Quiz Building Workshop.

    1. Hi Bhavani,
      It sounds like you and Edie have very much the same aims with an emphasis on the ā€˜doingā€™ of quizzes being more valuable than the marks gained from them.
      Absolutely! Iā€™m in the process of designing a tour of Wattle quiz question types alongside instructions for how to create them for the Wattle Quiz Building Workshop on 5 September 2019. The aim of the workshop is to expose participants to a greater variety of question types and give them the confidence to build them on their own. I really hope you can come!

  6. I really liked that I got to experience the different types of H5P quizzes in this module. Very cleverly done, Rowena! I agree, non-assessed activities with automated feedback are great for the ESL (blended/flipped) classroom.
    When I developed online activities for my ESL class, I discovered that seemingly simple formats can be used in a variety of ways. E.g. ‘drag & drop text’ can be used to put steps in the correct order (like you showed in the second H5P activity in this module) but also to order whole stories (which allows students to understand text structure and linking words). Similarly, ‘mark the words’ can be used to test students’ understanding of grammar (like you demonstrated with ‘click on the adjective’) but it can also be used for a ‘spot the mistake’ exercise – which could be either a listening exercise (practising decoding skills) or a proofreading exercise (practising life skills :-))
    I did this sort of thing using HTML-templates, so I’d have to test and see if it can all be done with H5P as well.

    1. Hi Melde,
      Iā€™m glad you like the examples! When I was teaching ESL, I used HotPotatoes to create suites of learning activities based around amusing stories. WebSequitur was perfect for rebuilding stories and summaries by using contextual clues. This could be replicated with the ā€˜Summaryā€™ question/content type in H5P but would a bit more work than just deciding where the line breaks go because you need to enter every set of distractors with each answer. I also tweaked the code of WebRhubarb so students would be given the first and/or last letter of longer words to give them clues and they could type full words in and decode a whole passage or set of questions and answers about the story theyā€™d just explored. It would be brilliant if H5P offered something like that. The closest thing at the moment is probably Dictation but that involves the audio files so it isnā€™t quite the same thing.
      Please check out my H5P Explained site: https://wattlecourses.anu.edu.au/course/view.php?id=27401
      Would love to get your feedback on that too .
      Rowena

  7. I had the wattle quizzes in 2017 as an assessment item (5 quizzes in total) and this was a nightmare. The wattle died every time the students had two wattle windows open (e.g., one for the quiz and one for browsing relevant lecture slides) and of course, this issue was not know before we had experienced it. This caused a lot of completely unnecessary stress to the affected students and a lot of overhead for me to organise opportunities for them to take the quiz later.

    Hence, I took the quizzes away from marking, and have had them as an option in 2018 and 2019. The students seem to love them, and even now expect other units to have similar optional quizzes.

    Also I find spying how many students have taken the quizzes and how well they do a great way to see how the class is traveling through the weeks. It seems to me that the students who do a little bit every week tend to get better marks and more enjoyable ride by dividing an conquering the work load.

    The only problem I have is to calibrate their expectations — that many quiz questions are a lot easier that exam questions and getting good quiz marks does not guarantee a good exam mark.

    1. Hi Hanna,
      Iā€™m very sorry to hear about the issues you had with quizzes in 2017. Iā€™d think things must have improved a lot since then as I havenā€™t heard of such issues happening recently.
      Iā€™m so glad you didnā€™t give up on them and your students now love and get great benefit from them. It is also great that you can track their progress this way!
      You raise a very good point that one must be careful not to lull students into a false sense of security before an exam. In addition to the weekly quizzes maybe you could set a Practice exam quiz based on the Exam from the previous year.

  8. I really like the examples using HP5 and the comparative Wattle view, thank you! HP5 seems to offer a few more options to create quizzes and get students involved. It does take a lot more thought to create the variety of quiz questions you demonstrated, but it seems worth the effort. I am also hoping to create enough quiz questions to build up a bank so that I can randomise a set. My largely international Masters students seem to like quizzes as it provides them with immediate feedback on learning. I used quizzes in one course, and because of student feedback have now incorporated them into another course. I look forward to when we can access HP5 and try this out šŸ™‚

    1. Hi Emmaline,
      I am definitely all for giving students variety. It irks me greatly when I hear people talk about ā€˜Multiple Choice Quizzesā€™ as if that is the only type of quiz there can be. What about the other 17 question types available in Moodle? How did you go with building up your bank of questions to randomise them?
      So glad you enjoyed seeing the H5P and Moodle questions compared! Have you started to explore H5P yet? Please check out my H5P Explained site for more info: https://wattlecourses.anu.edu.au/course/view.php?id=27401

  9. * Can you tell us about your experiences of learning through quizzing?
    I’ve had some terrible educational mismatches between the assessment task (MC) and the preparation for said task in my class environments. So bad that I’ve had to scale results up more often than not, so I ended up abandoning the MC as an assessment tool – particularly where I was getting great essay and short answer questions and 30% accuracy on the MC. That said, I’ve often wondered if the solution is trying to link training to the education outcome.

    At the same time, I’m also somewhat notorious for
    (a) having set a 30 question multichoice quiz with 29 A answers, and 1 C answer, with the C answer being Q27, so any student gaming the quiz pattern would change their answers (they did).
    (b) once having place the answer key to the exam quiz in plain sight in my office as abstract art work, revealing it to the students after the semester, and many of them commenting that they’d seen it, and not believed it could possibly be an answer key. Good abstraction in the art helped

    For serious though, I am interested in trying to build a multi-choice set that helps explore student uncertainty – eg help the students to self identify strengths and opportunities to improve – I’ll be commanding the intro to marketing course, and setting up good fundamentals would be useful for the later years.

    1. Hi Stephan,
      Thanks for sharing your experiences. It sounds like you might like to explore the ā€˜Deferred Feedback with CBM (Confidence Based Marking)ā€™ and ā€˜Deferred Feedback with explanationā€™ options in Question behaviour settings for quizzes.

  10. Similar to a lot of the other responses, I would prefer to use quizzes as a learning or practise tool rather than associating them with significant marks. I imagine students could either use them as practise tests or we could try and introduce elements of gamification to encourage students to persevere with some of the more boring aspects of course content (e.g. vocabulary memorisation in language courses). I have found the use of quiz-based apps, like Anki which I saw was already mentioned, very helpful personally in learning languages. A flash card deck for a language course, as William Maish created for medical students that was mentioned on Day 1, would be invaluable in adding students in memorisation.

    1. Hi Alison,
      I really like creating opportunities for self-paced study too. It would be great if more people used quizzes and H5P to provided these opportunities to students. I agree that Anki is excellent for learning vocab and terms but following up with using that vocab in context is also essential.

Leave a Reply to Sarah McIntyre Cancel reply

Your email address will not be published. Required fields are marked *

*