Assessment and Feedback

Day 4: Lessons learnt from the field

Post written by guest author Alexandra Webb, ANU Medical School.

We identified on Day 1 of this Coffee Course that practice tests (quizzes) and spacing learning over time are the most effective strategies for student learning. But what do students actually do when it comes to study?

In addition to a reliance on relatively ineffective study strategies and minimal use of effective study strategies such as practice testing and using flashcards, Blaisman et al., 2016 found that students mass their study to the day or two before a (summative) examination rather than spacing learning over time. So, by providing students with quizzes throughout your course(s), you can encourage students to apply effective study strategies for their learning.

This sounds simple! But how can we achieve this in practice? Here are a few tips to consider in designing and delivering quizzes throughout your course(s).

Steps in time

Test your quiz

Ask a colleague(s) or tutor(s) to review your questions, answers and feedback to identify any typographical errors, content errors, or misinterpretation before the quiz is released to students. If using an online quiz tool it can also be helpful to do the test a few times to make sure everything works OK.

Educate your students and staff

Educate your students about the best strategies for learning and stress the importance of scheduling regular study time. Share these discussions with your tutors and colleagues as Morehead et al., 2015 found that 41% of instructors told their students to use rereading as a study strategy.

Example from an ANU anatomy course: The number of students using an online weekly quiz increased from 79% (2018) to 93% (2019) when the course commenced with an introduction to effective study strategies. However, the effect was not sustained throughout the semester. So, reinforcement and/or incentives may be required.

Timed quiz release

Online quizzes have the benefit of automated timed release i.e. the quiz can become available and then closed to students at a specific time on a specific date. This can be used by the teacher to compel students to use the quizzes AND space their learning. It is important to ensure students are given explicit instructions regarding the quiz availability.

Example from an ANU anatomy course: 50% of students accessed and used a weekly quiz, aligned to the topic of that week, that they had 1-week to complete. In comparison, 18% of students attempted a quiz that was provided for them to complete any time during the course.

Incentives

Even with low-stakes quizzes, it is possible to provide incentives for your students to regularly participate in the quizzes you provide to them. This may include: minor contribution to the end of course grade, gamification/competition, explicit alignment to summative examinations.

Examples: For each student their top 3 marks for 10 quizzes during the course contribute 5% to their end of course grade. The student team with the highest collective number of marks at the end of the course receives a prize. A selection of quiz questions will be used in the end of course summative examination.

Analytics

Review and respond to the learner analytics to address any misconceptions that your students have had. In order to obtain useful analytics, you may have to set your quiz so that students can only respond once which in some quiz systems may mean that they cannot explore other answer options or immediately see the feedback for each answer option (in a multiple choice question). So you need to carefully weigh up what your goal for the quiz is. To get the best of both worlds you can do both: set the quiz with one answer option only to be completed by a specified deadline – review and respond to the analytics – re-release the quiz with the option to answer multiple times and view all feedback options.

The quiz results can tell you how to adjust your teaching based on how students are dealing with the material. Use the reports to feed-forward into your next face-to-face session. What strategies would you use in-class to address the quiz result reports? If students consistently get some answers wrong, this could mean that the question is confusingly worded. If students get it right every time, it might be too easy.

One of the Coffee Courses coming up later in 2019 is focused on learning analytics – more information available soon.

Reflect

Reflect on your quiz questions, feedback and delivery method; student participation, responses and results; and identify any issues that you can solve to improve the quiz for next time.

 

question mark

Activity and discussion

  • Draft a quiz question and post it here for feedback from your peers.
  • How do you plan to deliver a quiz  in your course?  e.g. Moodle (Wattle) quiz, Echo 360 ALP, hands in the air, paper-based activity?
  • What ideas do you have for initiatives to maximise student participation?

Give us your feedback!

Our team is currently running an evaluation of the coffee courses, and we would really appreciate if you could take our survey and share your experiences. Read more.

14 thoughts on “Day 4: Lessons learnt from the field

  1. I have been making increasing use of quizzes, and other small assessed tasks over the last few years. Almost all students participate, and indicate they like this.

    It had not occurred to me to discuss the quiz results in-class, so that is something I could do next semester.

    One worry I have is is students are gaming the quizzes. I don’t mind so much if a group of students decide to do the quiz together, as they will learn a lot from each other that way. But I wonder how many outsource the quiz to someone else, as a favor, if not for money.

    Ultimately the student will be the looser from this, as they will not learn from the quiz, and so not do as well in the other assessment. Should I put in place technical countermeasures: such as checking the location of the device being used, or sending a code to the student’s phone to confirm it is them? Should I use some form of social engineering to discourage cheating?

    1. I think this is one of the tensions in deciding whether to make a quiz contribute to a course mark or not and weighing up which is most relevant/appropriate to your course context. If it is decided that the quiz will contribute marks then the higher the mark the more you may feel it is important to police potential cheating. I believe there are a number of technological solutions that can be used in situations where the quiz needs to be monitored to prevent cheating e.g. ExamSoft – https://examsoft.com/
      There have already been some useful comments in the Discussion Forum on previous days of this Coffee Course where others have shared their experiences regarding the value of quizzes for learning vs cheating.

    2. Hi Tom,

      This idea of discussing the quiz results in class was new to me as well. I intend to try it as a mechanism to highlight how powerful these optional learning resources are. How did it go for you? Any experiences that you could share?

  2. I was wondering if anyone has used the analytics in Wattle (Moodle) quiz? I believe it’s called Quiz Statistics. I’ve not had the chance to really look at them in detail but from my reading it sounds like they could provide quite a bit of useful feedback to the teaching staff about which questions are effective? Here is some information about what sorts of things are included: https://docs.moodle.org/35/en/Quiz_statistics_report#Reasons_to_look_at_the_statistics_report. I think this sort of thing might be particularly helpful if using quizzes regularly for summative assessment. Has anyone got some tips about using these? 🙂

    1. Hi Katie,

      For our third summative quiz, we repeated a couple of questions from the first 2 quizzes that the Quiz Statistics identified students did “worst” in. The analytics were also useful in helping us identify both topics/concepts that were collective strengths or weaknesses across the class, and question formats that might have been too easy or required tweaking. Definitely looking forward to the learning analytics course to explore this more.

      1. I really like this idea. This could easily be coupled with the suggestion above that you can advertise that some questions from the quizzes will be repeated in the summative assessment to create incentives to do the quiz. Repeating the question that students found most difficult could also check that the measures you have taken post-quiz to clarify that question and explain the answer were effective.

  3. Thank you for this informative course. I was particularly surprised by the effective vs preferred study strategies.

    A couple of questions from today:
    1) How can you get colleagues to test your quiz on Wattle without enrolling them into the course? There are certain features (mainly around feedback) that do not seem to work for me, even in student mode.
    2) In a previous Coffee Course, we looked at metacognition (?) briefly. While I am all for educating my students about learning strategies, and explaining why our pedagogies and philosophies are the way they are, I find my students aren’t all that interested. They believe they already know what to do, and find such discussions irrelevant to the course. Any suggestions on how to do this effectively?

    1. Dear Bhavani,
      I’m so glad you have found the course informative! I can answer your first question but I’ll allow Alex (the author of this post) to answer your second question. You can enroll a colleague as a student in your course so they can see exactly what your students would be able to see. To stop students seeing a quiz before it is ready, set up a section that is only visible to your colleague so they can take the test. Once you’ve got their feedback and had a look at their results, you can delete their attempt and edit the quiz and questions. Finally, move the duplicated quiz up to where your students can see it. Please let me know if you’d like help with this.

  4. The way I used online quizzes in my ESL course:
    – I employed two types of staggered online quizzes, formative and summative ones
    – formative quizzes focused on lower-order thinking skills and had to be completed before class. Students had to retake the quiz until they achieved a score of 80% or more. They received tips and feedback if they submitted a wrong answer.
    – summative quizzes focused on higher-order thinking skills and were made available after class. Students could take the quiz multiple times, but only their first attempt was graded. They received feedback at the end of the quiz.
    I analysed the quiz attempts before each class, and if I saw trends in which answers my students had got wrong I made sure to address that in class. It might have meant that a certain topic was not clear to them, and that we needed to spend some more time on it. Or it might have meant that the quiz was poorly designed. Either way, very useful information!
    I used this method in conjunction with in-class tests and take-home writing assignments. Worked well for me and my students!

  5. I found it really curious how practice testing was ranked last in the study strategies list. That was one of the primary methods we used in prep for our math and physics exams – as usually we were provided with a backlog of previous years exams to get a feel for the question types.

    This post has given me another idea on how to incentivize the use of quizzes. In addition to the minor contribution to final grades (as I find so far that this method works well in my field), a further incentive could be that at the end of each short quiz there will be a question similar in difficulty to one that could be on the exam. This way the entire quiz will not intimidate the students as the difficulty could be leading up to this final exam-like question, with the added benefit that it could hopefully reduce the students anxiety before the final exam as they would be familiar with the question format, while also allowing me to analyze how students are responding to the exam type questions.

    I think that this continual building on formative assessment with a view towards the final summative assessment could help the students better absorb the knowledge and incentivize learning throughout the course rather than 2-days before the exam.

  6. Dear Sarah
    Thank you for your comments. I just want to clarify your first comment – which I hope I have interpreted correctly (apologies if I have not).
    Practice testing is one of the most effective learning strategies (see Day 1 of this coffee course – https://anuonline.weblogs.anu.edu.au/2019/07/08/day-1-the-science-of-learning-and-the-value-of-quizzes/). Studies have shown that students have good intentions for using effective learning strategies but when it comes to doing a course of study, they do not apply this to their own learning. More information can be found in this article (see below) which covered a range of disciplines/courses. They did not compare disciplines/courses but that would be interesting to investigate if math/physics is approached by students differently compared to other disciplines.
    Rachael N. Blasiman, John Dunlosky & Katherine A. Rawson (2017) The
    what, how much, and when of study strategies: comparing intended versus actual study behaviour,
    Memory, 25:6, 784-792, DOI: 10.1080/09658211.2016.1221974

  7. I have not used analytics or Quiz Statistics in Wattle (Moodle). I have been using Moodle to generate quizzes because it links directly to gradebook. I did not know that there was such a tool and will take a look in more detail soon. Thank you for the resources and now the incentive to join the learning analytics coffee course!

  8. Dear Emmaline. Great to hear you have discovered some additional tools to further enhance your use of quizzes. Enjoy the learning analytics coffee course! Best wishes Alex

Leave a Reply to Bhavani Cancel reply

Your email address will not be published. Required fields are marked *

*