Open Education: From Resources to Practice
Written by Adrian Stagg and Emma Power, University of Southern Queensland
Reminder! Today at 10.30am QLD time (11.30am NSW/ACT/VIC/TAS time), USQ will live stream ‘Practical Openness‘; results from our OEP Grants Program. You’ll be able to post questions and have them answered by folk who have grappled with OEP for almost a year. Find out more information here.
Welcome to Day Three, everyone. In the previous days, the grounding for Open Educational Practice, and the issue of quality have been explored, which dovetails very nicely now with open assessment. Today is a slightly different format as the learning activities are at the end of the post, not scattered throughout. This is intentional, because there is a little bit of ‘front-loading’ that needs to occur. I want you to engage with the reading and videos purposefully though, and as you do so, think about this question:
What are the opportunities for universities that embrace open assessment? Would this practice effectively position universities as providing a ‘societal good’, as well as equipping students with life-long skills?
It’s a big question, but revisit it after each section and reflect. You might even post your reflections as a ‘stream of consciousness’ in the comments, or keep notes as you progress so that you can see the progression of your ideas. I’ll come back to these points at the end.
Why open assessment?
Open Educational Practice (OEP) tends to ask deeper, more complex questions of our practice, and open assessment certainly qualifies. Essentially, open assessment occurs whenever students are co-creators of openly-licenced content that meets an authentic need. Whilst such assessment is possible within closed systems, it means that you need to appropriate student copyright, and that the sharing is extremely limited. If the material created by students is monetised in some manner, then it essentially reinforces that knowledge is simply a commodity for purchase â instead of as asset owned as part of a ‘common wealth’ of society.
Rajiv Jhangiani, a professor of psychology (now working at the University of British Columbia as an advocate for OEP), describes this aspect of open pedagogy in very human terms:
“Incorporating openness into pedagogy is simultaneously liberating and terrifying. It challenges instructors to reflect on their practices and move away from the traditional top-down model of pedagogy by assigning open-ended problems and empowering students to act as co-creators (Rosen & Smale, 2015). But whereas it takes a degree of courage to untether oneself from the security and predictability of the staid research essay, once accomplished, the benefits to the learning process are sizable. For one, students and instructors work collaboratively towards creating resources for public consumption, adding tangible value to the world outside of their classroom. Second, students tend to invest more effort and care more deeply about the product when they know that their work has a larger potential audience than just their instructor (Farzan & Kraut, 2013). Third, open pedagogy unleashes the studentsâ creative potential, allowing them to ascend the rungs of the cognitive process dimension in Bloomâs revised taxonomy (Anderson & Krathwohl, 2001)”.
From: Jhangiani, R (2017). E-xcellence in Teaching Essay: Ditching the âDisposable Assignmentâ in Favor of Open Pedagogy [blog post], Society for the Teaching of Psychology, http://www.teachpsych.org/E-xcellence-in-Teaching-Blog/4583103
Rajiv explains this further in a video interview from last year:
I want you to keep these benefits in mind as we explore some examples of open assessment, as I’ll use them as a type of criteria for assessing fit-for purpose open assessment.
The disposable assignment
It’s also worth explaining the term ‘disposable assignment’: it was coined by David Wiley to describe any piece of assessment that was constructed by the student, and marked by the lecturer, only to have the student read the mark and the feedback â and then throw it away. Wiley (and many others that we’ll meet in the examples below) argue that this does not lead to deeper student learning, nor does it invest students in their learning – you can read more about this here.
At this stage, some readers may consider this a worthy argument against the much-maligned essay. In the last decade, the essay (like the exam), has been criticised as outmoded assessment unfit for ’21st century learners’ (whatever one of these might be). What is often lacking in the argument against essays is a discipline-nuanced critical discussion about the role of this assessment in developing core disciplinary skills, and the way in which essay writing is embedded in the curriculum. Often the rationale and process of essay writing is opaque to students; that is, the purpose to write one, the structural elements, and the link to the broader disciplinary narrative are not explained â so the format is perceived as ‘disposable’ when this simply isn’t the case.
I don’t want us to get side-tracked on essays, but it serves as a very good example of why we need a critical, discipline-based lens when examining assessment, and not simply dispose of entire formats due to prevailing fads.
What does it look like?
Open assessment can take many forms, but luckily, there are emerging examples of good, successful practices. Let’s take a look at three:
- Amin Azzam (UC-San Francisco School of Medicine) and his students recognised that for many people, Wikipedia is the first source of information for public health. In fact, public health articles on the site are viewed by 200 million people per month for a variety of reasons. The course assessment documented quality improvement changes made by students to Wikipedia articles in this discipline; and the articles improved by twenty-eight students were viewed more than 975,000 times during the semester.
- Rajiv Jhangianiâs Social Psychology course includes assessment whereby students design and contribute question banks linked to the course learning outcomes for future students to use as revision. Students were provided with learning activities about writing effective questions, and providing constructive peer feedback. One of the drivers for this activity was the closed nature of commercial textbooks that offer question banks, slides, case studies, and other resources as ‘value added services’. Students who can’t afford or access a textbook though, are denied access to these resources.
- Robin De Rosa (Plymouth State University) realised that the anthology of early American literature prescribed for her course was simply a collation of readings in the Public Domain, packaged and sold by a publisher. To make the book more freely accessible, and to engage students with the literature, Robin asked students to source the Public Domain versions of the examples, and then write an introduction to each reading explaining it’s background and relevance. The result has been freely shared here.
In all these cases, the assessment ‘lives on’ beyond the course. Students can refer others to their work, it does ‘make a difference’ for other learners, and the content can be updated in future course offerings by other students – making it ‘renewable assessment’ instead of ‘disposable assessment’.
I think we’ve worked hard enough today, so let’s go back to the original question driving our reading and start discussing this one in the Comments. I really look forward to hearing your thoughts, as I’m currently working with a few lecturers on open assessment ideas, so I have an authentic context to learn from you too.
Learning Activity 6
Please post your responses to this question in the Comments.
What are the opportunities for universities that embrace open assessment? Would this practice effectively position universities as providing a ‘societal good’, as well as equipping students with life-long skills?
Until tomorrow, we’ll see you in the Comments!
Licence
The text of this work is made available under a Creative Commons Attribution 4.0 International Licence. All images and videos retain their respective licences.
15 thoughts on “Day Three: Open Assessment”
Good morning everyone, and welcome to Day Three. Yesterday was a challenging set of activities, and the conversation gave me plenty to think about too. Perceptions of, and actual quality, tend to be difficult discussions, and there is a lot precision required when articulating a ideas – but the respondents certainly did this. We spent some time unpacking ideas, especially standards, criteria, indicators, and evidence, and the purpose of quality in education. We didn’t reach any particular conclusions about whether openness required a separate set of quality standards, but overnight people have added to the Comments (for both Day 1, and Day 2 – which is great to see).
Today we’ll take a look at how open educational practice empowers assessment – what types of assessment are made possible by free and open resources? You might already be using some of these ideas in your work, or in courses – if so, share links!
This morning, USQ is also hosting a livestreamed event called ‘Reflections on Practical Openness’; presentations and discussion from our 2017 OEP Grant awardees as they reflect on the challenges and outcomes of the grants. You’re all very welcome to join us at 10.30 (Qld time). Details are at: https://www.usq.edu.au/open-practice/open-education/OAW2017/2017-reflections
Hope that you can make it to the livestream, and – as always – I look forward to hearing from you on this topic.
Wow–thank you for sharing some of my work, Adrian and Emma! I wish I could participate in your discussion or livestream but instead I will be following your posts with great interest. Please reach out if I can be useful in any way.
Thank you for your comment Rajiv, your Social Psych open assessment was particularly interesting to me. I can see the value of challenging students to develop meaningful questions themselves and also peer reviewing their classmates’ questions. Including some of the students’ top quality questions in the exams too seems like a great motivation for students to engage deeply with the content and how it could be applied.
Hi Rajiv, thanks for taking the time to drop by.The examples I used in the post were ones that I thought people could relate to, and possibly re-use in their own discplines. The need for test banks (especially ones that address deeper learning of topics) is a gap that I think open education can certainly fill. If we can also include students teaching other students, then it becomes even more powerful.
Q. What are the opportunities for universities that embrace open assessment?
A. Limited.
The term “open assessment” is new to me. I had to re-read the notes several times until I found a definition: “… occurs whenever students are co-creators of openly-licenced content that meets an authentic need…”.
This differs from the definition I found in the Wikipedia:
“Open assessment is a method for making impact assessments
where anyone can participate and contribute.”.
That is a process used for the ANU Techlauncher students I tutor. As well as the tutor, the student’s client, mentor and other students are all involved in the assessment. However, none of this is “open” in the sense used in the Coffee Course, as the students privacy and that of their client must be respected.
The open assessment discussed in the coffee course appears to have two aspects: 1. open license, 2. meets a need.
I routinely set assessment tasks which meet a need, where the student can do the assignment in their workplace (I have been running an “ICT Sustainability” course at ANU since 2009 this way), or for a real clients (I have 13 students doing soft3are development for clients this semester).
However, I rarely attempt to set assessment with an open license. I have done this with one-off research projects, where an individual student undertakes work which might be released as a report or as open access software. In some cases I have had three students, or teams of students, refine the material in sequence over years.
But doing this with dozens, or hundreds, of students over months, seems to be a high risk, high staff resource exercise. This is a risk as the students may be seen as being used as free labor, producing course-ware. Also anything released publicly exposes the students, especially international students, to public scrutiny. In an extreme case, something a student writes in an assignment in Australia could get them prosecuted when they return home, or executed.
If students are to undertake “open assessment”, then their instructors and themselves, will need to be trained in what it is and how to do it safely and effectively.
Q. Would this practice effectively position universities as providing a âsocietal goodâ, as well as equipping students with life-long skills?
A. No. The priority, I suggest, should be having students undertake realistic assessment tasks. Ideally these will be in the real-world, but some in a simulation. The scope for the students to produce “open” materials will be limited. Government agencies and commercial companies are willing to have designated university instructors read reports written by students for assessment purposes. However, they are naturally reluctant to have details of their operations, including sensitive security and financial details, made public. Much of this material will also be of little general value.
Hi Tom,
Thank you for your comment. I agree that Open Assessment (and indeed all OEP) needs to be appropriate for the context. With issues like privacy, designing valuable assessment tasks (from a student perspective rather than lecturer perspective), and considering student context risks (like overseas students). I think it may be more applicable to some fields of study over others too. I agree that authentic and real-world assessment tasks can be quite valuable for students, and ideally open assessment is designed to be not only open but useful for students. I can see the examples provided as being quite valuable for students developing a more nuanced understanding of the content whilst providing them and their peers a unique tool for revision and other learning purposes.
Hi Tom,
What you highlight here is that context matters, and I agree. Open assessment won’t suit all disciplines, nor will it suit all applications. In the case where students partner with commercial, proprietary bodies, it is entirely reasonable that there is commercial-in-confidence information that can’t be shared, or that the outcome of the student work can’t be shared as it may either share this in-confidence knowledge, or could give away a competitive advantage. In a completely commercial environment, this is understandable, and I’d expect it.
There have been examples of students working in team-based assessment on free and open source software, often from a list of user requests.
Now, this does fit the open assessment operationally. There is an open licence involved, and this is actually a really important part of the process. The students’ work is identified legally as theirs – they receive the full credit for the work, and have the ownership rights to determine what can and can’t be done with it. Creative Commons licences (and GNU for software) do have a range of options for creators to select from. They could choose to lock it down with a Non-Commercial No-Derivatives Licence, or they could pick any of the other five licences. Part of the course (or program) design would include information literacy components that would cover ethical use of information resources.
However, the due diligence would need to rest with the lecturer in expressing the expectations of the assessment and how/where the sharing would take place.
This ‘due diligence’ would be built into the assessment rubric. You obviously don’t want the work of all students being shared openly, irrespective of their grade. When designing open assessment, it is common to include a section of the rubric that explains the level required for the resource to be shared. Those that do not meet the criteria set by the marking rubric are ineligible to be shared. This is also why peer-review cycles, and formative assessment design works well in this instance.
You’ve deftly shown that there are a range of very deep considerations and contexts that need to be considered, rather than the uncritical acceptance of openness as ‘A Good Thing’. My intent with this course was very much to have people think about this deeply, and reflect on suitability of aspects of openness, so I do welcome your thoughts.
I must admit I got lost in daydreaming about all of the possibilities with this one! Perhaps it helps that I’m also involved with a group within a professional association where we have already identified that there is a lot we would love to achieve but simply don’t have the manpower. There are also a lot of practicalities about setting up something of this nature, where students contribute to something that is useful externally to the class – issues around the quality of the content, for example, although having an added step of reviewing what someone else has contributed before it goes live would be a good way around this and could be another step in assessment. It’s like a controlled crowdsourcing.
So my ideas now are around research skills – we have students learning information literacy, more advanced students learning how to teach information literacy, others again learning research and reference skills and setting up user education. A nice mix.
Identified gaps in the accessible information locally – we have already put out an open textbook on the library and information industry in New Zealand http://bit.ly/1vyRJRK but there is no organised biography of New Zealand Librarians. There will be other gaps too – e.g. outlines of specialised library services in NZ, collections of research tools for the local context (that one is already underway!) etc. A couple of NZ librarian biographies appear on Wikipedia, but the length and quality is variable, and entries are not co-ordinated and coverage is not methodical.
Assessment for students learning about information literacy could include researching about a NZ librarian and writing a short entry (like a Wikipedia entry) which includes links to all the information sources etc. That would serve as an assessment for that class. The advanced students learning to teach information literacy could perhaps pick up an entry each, critique it, write a note to the creator (probably not to be delivered though) explaining what else could be done, and then improving the entry before it goes live.
So – information creation and quality control spread over two classes using students in two classes to create and quality check entries, resulting in a Wiki full of quality local information for our local profession. Win-win.
Now just to that more tricky task of trying to sell the idea to the actual people who are involved in writing those assessments.Or finding modified ways to do similar things for the courses where I have the option of creating the Asessments myself.
I think at this point the ideas are easy but putting them into practice involves a whole team of people who may need to be brought up to date with OEP and ideas beyond traditional student assessments first. And external constraints and requirements will need to be considered in the mix, of course.
Hi Alison,
What a great idea for an open assessment task! I think that it would add value beyond just for students, but also for others in the community looking for that information too. I think there is a lot of value for students performing peer review and receiving constructive feedback from peers. It’s good to see you consider the practicalities of implementing an open assessment like this too, however I’m sure if its something that is beneficial for students, staff and the community it wouldn’t be too hard to get people on board. Let us know if you’re able to make it happen!
Controlled crowdsourcing… I like it!
The example that you give is really quite an exciting one. I particularly like the idea of students within a discipline working together across year levels – it’s almost like an informal community of practice for the assessment. The connections students forge across the year levels could also transcend the assessment, with the more experienced students able to share some of their experiences. Librarians do tend to be a social lot, and sharing is (in my opinion at least) built into the profession, so this could work given the alignment with professional beliefs.
It would also give students a chance to contribute to the profession that they may eventually join, and gives them professional contacts that may be useful should they need to approach libraries for placements and the like.
Overall, I can see a lot of potential with this idea, but obviously there are some assessment design issues that would need to be carefully examined. If you get traction for this idea and need someone to facilitate some discussion with your group, I’d be most keen to be a part of the conversation.
But in the meantime, please, continue daydreaming about the possibilities. It’s only by creating hypothetical cases that we’re able to critique and refine ideas.
I must admit I got lost in daydreaming about all of the possibilities with this one! Perhaps it helps that I’m also involved with a group within a professional association where we have already identified that there is a lot we would love to achieve but simply don’t have the manpower. There are also a lot of practicalities about setting up something of this nature, where students contribute to something that is useful externally to the class – issues around the quality of the content, for example, although having an added step of reviewing what someone else has contributed before it goes live would be a good way around this and could be another step in assessment. It’s like a controlled crowdsourcing.
So my ideas now are around research skills – we have students learning information literacy, more advanced students learning how to teach information literacy, others again learning research and reference skills and setting up user education. A nice mix.
Identified gaps in the accessible information locally – we have already put out an open textbook on the library and information industry in New Zealand http://bit.ly/1vyRJRK but there is no organised biography of New Zealand Librarians. There will be other gaps too – e.g. outlines of specialised library services in NZ, collections of research tools for the local context (that one is already underway!) etc. A couple of NZ librarian biographies appear on Wikipedia, but the length and quality is variable, and entries are not co-ordinated and coverage is not methodical.
Assessment for students learning about information literacy could include researching about a NZ librarian and writing a short entry (like a Wikipedia entry) which includes links to all the information sources etc. That would serve as an assessment for that class. The advanced students learning to teach information literacy could perhaps pick up an entry each, critique it, write a note to the creator (probably not to be delivered though) explaining what else could be done, and then improving the entry before it goes live.
So – information creation and quality control spread over two classes using students in two classes to create and quality check entries, resulting in a Wiki full of quality local information for our local profession. Win-win.
Now just to that more tricky task of trying to sell the idea to the actual people who are involved in writing those assessments.Or finding modified ways to do similar things for the courses where I have the option of creating the Asessments myself.
I think at this point the ideas are easy but putting them into practice involves a whole team of people who may need to be brought up to date with OEP and ideas beyond traditional student assessments first. And external constraints and requirements will need to be considered in the mix, of course.
Today is difficult to answer that question, because it’s no fully clear how open assessment can be implemented, there many questions connected with culture, technological viability, even moral and ethical questions, as it has been explained in other comments. But one thing is clear: oppennes is THE trend, and it include assessment, because it is part of the all OEP, who decided to join it, will have better chances to success.
I agree Rey, there is definitely a lot of contextual elements to consider before implementing open assessment in a course. Despite there being situations where openness is not suitable I think it is fascinating the possibilities and examples of where it is a good fit for the course due to the value students and academics could get from it.
Thank you for your attention and your efforts (and Adrian’s) for deciphering my comments. I hope you both to success in your professional activity.
Best regards,
Rey.
“What are the opportunities for universities that embrace open assessment? ”
In undergraduate science teaching labs, it might be possible to adapt some experiments so that they relate more closely to real-world problems. A limitation to openly publishing the lab reports of such experiments is that it becomes too easy for the next year’s students to just copy those reports.
Also, in some cases, inserting an element of “authenticity” won’t work because it’s necessary for students to conduct experiments about fundamental concepts that are not immediately apparently related to real-world problems. In those cases, the “open” part of the assessment could take place as a written assessment. When students do experiments, they have to write lab reports, and they’re usually done individually. An adaptation could be that the students submit the report they wrote individually, and then work together by peer-reviewing the reports (or working with a study buddy) and then re-submit the improved work along with an explanation of what they learned as a result of the collaboration. The reason for the two-step process is for students to show the individual work they’ve done.
In lab reports, students are often required to answer questions where they apply the concepts in the experiment to real-world issues. In an open assessment, the students could be asked to collaborate and propose a solution to some real-world issue. They might even be asked to follow through, implement the solution, and examine its impact in another lab report.
One issue is that in the sciences, students may not want to collaborate since they might be in competition with each other to get into medical, pharmacy, nursing or other such professional programs.
Another issue is funding. Implementing any real-world solution comes with costs.
In some science degrees, students have to complete a major research project conducted over two or three semesters. These projects could be related to current real-world problems. One limitation to this idea is that students don’t normally accomplish much in such a short time frame. Plus, they’re still just learning about research. Their learning has to come first. Another limitation is that some projects involve patents such that the results cannot be made immediately public. Until the results are published, the only people who know the details about this kind of project are the student and their supervisors.
One way that openness can be included in most science courses and labs is to have students seek out openly licensed material (where it’s necessary and useful) and insert it or refer to it in their work. The completed work can be submitted in the traditional closed ways or with a CC license, depending on the situation.
“Would this practice effectively position universities as providing a âsocietal goodâ, as well as equipping students with life-long skills?”
I agree with Tom that the priority is the students’ learning. Also, for universities to advertise their use of open assessment, this practice would have to be implemented on a large scale – likely at the scale of the department. I don’t think it can be advertised as being an institution-wide practice given the limitations in some departments. The practice of open assessment would also need to prove to be beneficial for the institution and for the students before it is advertised as an advantage. Proving such things takes market research which first would require administrative approval. These steps take time and lots of convincing, so it could be years before a university got to the stage of implementing and advertising its “societal good” of practicing open assessment. Now, suppose all of this did actually take place. How do you advertise “open assessment” in a clear way that convinces people that it is beneficial?
Another point is how “open assessment as a societal good” factors into university rankings or university reputation. If this practice cannot be realized, it’s one more obstacle to implementing open assessment. I think the more realistic approach is to communicate with university instructors and course-design teams and implement open assessment as a small-scale initiative. If anyone is willing to do research on these initiatives to reveal their weaknesses and benefits so that a subsequent iteration can be implemented with improvements, all the better!