A key principle of assessment is that it meets the conditions of Validity (gauging if a student has met the objectives) and Reliability (gauging how much we can trust the results). There is also ‘backwash’. Hughes (2002) wrote: “The effect of testing on teaching and learning is known as backwash, and can be harmful or beneficial. If a test is regarded as important, if the stakes are high, preparation for it can come to dominate all teaching and learning activities. And if the test content and testing techniques are at variance with the objectives of the course, there is likely to be harmful backwash.” (2002, p. 1) Later saying: “Too often the content of tests is determined by what is easy to test rather than what is important to test.” (p.27)
An important thing to consider before you launch into writing questions is “How can I give my students opportunities to show what they know and how can they apply this?” Aim to align your questions to the course learning outcomes – the knowledge, skills and attitudes they are expected to achieve during the course. Below is advice offered in the ANU Med School’s Formative Assessment and Quiz Design course converted into an H5P ‘column’ (with text, an image hotspot and a drag text quiz question).
Many have written on the subject of reliability including Hughes (2002, p.44-50). Tips for increasing reliability while also preventing harmful backwash can be summarised as follows:
- Include a sufficient number of questions to give the student enough opportunity to demonstrate mastery but not so many that they get fatigued or distracted. Where possible, ensure the effort put in is reflected in the points awarded.
- Start with easier items so the student does not panic then increase in difficulty thus differentiating student performance.
- Write auto-marked questions (and instructions) in a precise and unambiguous manner that does not give the answer away but allows a limited number of responses and discourages students from clicking without thinking.
- Ask peers to take the test and identify any errors or potential confusion then address these. This may also bring to light alternative answers you had not considered.
- If you opt to use the longer ‘Essay’ question type, you can provide a composition template so students do not stray too far from what you intend them to write and will be easier for you to manually mark.
Have you experienced good and bad backwash from quizzes? What do you think are the characteristics of good and bad quiz question or quiz design? How would you go about planning for a successful quiz?
A lot of these tips from or based around tips from Vanderbilt University for designing multiple choice questions can be applied to other question types.
Extra tip: It is possible to create single gap-fills with MC questions but it makes more sense to use a gap-fill/cloze for this. Also, it is better to use matching rather than MC if you intend to reuse the same distractors anyway.
A great beauty of using technology to deliver online quizzes is that you can provide immediate feedback. If set up correctly, students never have to wonder: Which questions did I get right and wrong, and why? and How could I improve?
Everything you ask your students to do should be a learning experience. Exams seem to be the antithesis of this but quizzes don’t have to be.
Here are some tips for providing effective feedback for your quiz:
- Write specific feedback for each option (where appropriate/relevant).
- Don’t just tell students the correct answer. If possible tell them why their answer was correct or incorrect.
- Similarly, don’t just tell students the incorrect answer(s). If possible tell them why their answer was incorrect.
- In some situations, you may wish to amalgamate all the feedback from each option and provide the same feedback whichever option is selected. This gives the students the opportunity to learn everything you have to say about each option.
For more on feedback see our previous Coffee Course, Feedback for Learning.
How do you or would you provide quiz feedback to your students? Do you have any other tips on crafting questions or feedback?
Resources and references
- IT Training Services, Penn State University (15 March 2017). Writing Effective Quiz Questions. Retrieved on 11/06/19 from: http://personal.psu.edu/bxb11/QuizQuestions
- Hughes, A. (2002). Testing for Language Teachers (Cambridge Language Teaching Library). Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511732980
- Brame, C., (2013) Writing good multiple choice test questions. Retrieved 12/4/17 from https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/.
- Centre for Teaching Excellence, University of Waterloo, Designing Multiple-Choice Questions. Retrieved 09/07/19 from: https://uwaterloo.ca/centre-for-teaching-excellence/teaching-resources/teaching-tips/developing-assignments/assignment-design/designing-multiple-choice-questions
- Armstrong, P., Bloom’s Taxonomy. Retrieved 12/4/17 from https://cft.vanderbilt.edu/guides-sub-pages/blooms-taxonomy/