Digital Content

Day Two: ‘I’ll know it when I see it’, or, usefully defining quality in openness

Open Education: From Resources to Practice

Written by Adrian Stagg and Emma Power, University of Southern Queensland

Common Good Books by Kathleen Tyler Conklin CC-BY, downloaded 20/10/17

I’ve used the image on the left in a number of my presentations and it never fails to elicit chuckles in the audience when they read through the categories this book store offers. As a former librarian, I’m still not certain what ‘Quality Trash’ is (and it might be better left as a mystery), but it makes one wonder what criteria a book needs to be shelved under that heading.

There are a few topics that are absolutely guaranteed to generate robust and vigorous discussion in teaching and learning – and one of them (at least among my network), is ‘quality’. If you cast back to Susan D’Antoni’s report from yesterday, quality was the fifth highest priority globally (after Awareness raising, communities, capacity development, and sustainability) and cited as a priority in all geographical regions.


Learning Activity 3

Before we try to make sense of the debate around quality standards, measures, and proxies, I’d like you to reflect on what quality in learning and teaching means to you. It’s a big question, so when answering in the comments, pick one indicator and explain why it is important to you.

For example, you might think that strong teacher-student interaction is a sign of quality, or that the resources selected for the course are particularly current or topical. There aren’t any right answers here, we’re establishing some basic perceptions of a very complex concept.


Building on the notion of OER and OEP is how the discussion around quality is framed. In the earlier course, we touched on a number of barriers to OER adoption. One was the perception that institutions would only share resources that could not be commercialised – the adage ‘you get what you pay for‘ was the root of this criticism.

Direct and indirect measures of quality pervade higher education and often the two are conflated. David Wiley, an influential scholar in Open Education, more precisely states that there are direct measures, and indirect proxies for quality in OER. That is, a number of criteria are often used to encourage the perception of quality, but do not always have a direct correlation to it.


Learning Activity 4

Read David Wiley’s short post about OER quality standards, and in the comments below reflect on two aspects of the post:

Firstly, in the context of your own work or practice, do you agree with his perception of measures and proxies? Through the lens of open educational practice (the focus being on learning design that leverages openness, organisational cultures that value and recognise openness), do you think that the measures and proxies refer to OER, OEP, or both?

Secondly, have a look at the date of the post (I’ve deliberately chosen it based on the currency). Is this applicable to discussions at your institution, or has there been significant change?


Following on the learning activity is a much more recent post on the International Council for Distance Education (ICDE) blog by Professor Daniel Burgos. He discusses the role of quality in light of an abundance of OER, and in possibly normalising quality measures used in mainstream educational systems.

Many repositories for open resources – and especially open textbooks – have included a peer-review process for the resources, as a way of supporting lecturers in their choice of learning materials. Here is one example from the BC Campus Open Textbook Collection (scroll down to the bottom of the page to see the reviews).

Quality, by Tony Hawk, CC-BY-NC, downloaded 20/10/17

When discussing OER, there are some clear attributes that can be examined – most of them drawn from other mechanisms that have been used for closed materials. Traditionally, Faculty, Librarians, and Learning Designers have exercised a role as arbiters of quality in various guises; and now the same is true in open education. Whether OER needs to be ‘fenced off’ separately to other resources in the quality debate is still under discussion.

However, open education practitioners need to consider additional facets of quality in their work. The Australian Open Education Licencing Toolkit provides a visual representation of these facets, framed by Legal, Technical, and Accessible concerns. The ‘most open’ resources are those that adhere to the criteria at the top of the list, and the level of openness decreases as you move down the page. We’ll revisit this on Day 4.

It would appear that there is a perception that open practitioners need to demonstrate that their work still meets quality standards in order for OEP to be a viable and sustainable practice. This leads us into the last learning activity for the day.


Learning Activity 5

Today has really only skimmed the surface of quality in OEP, and there are many types of discussions waiting to happen. From the list of three questions below, select one, and post your thoughts in the comments below. I’d encourage you to also engage with other participants, remembering that this is a professional space.

  • Do open educational resources need their own version of quality?
  • Ultimately, who decides what quality looks like in open educational practice?
  • Do you think that mainstream measures of quality (such as peer-review) need to be applied to open resources in order to normalise the practice of using them?


We look forward to speaking with you all in the comments before we move on to a (perhaps) less contentious issue tomorrow – open assessment!


The text of this work is made available under a Creative Commons Attribution 4.0 International Licence.  All images and videos retain their respective licences.

18 thoughts on “Day Two: ‘I’ll know it when I see it’, or, usefully defining quality in openness

  1. Hi everyone,
    I’m a little late to the party, but have been catching up on the comments from yesterday. Firstly, thanks for the great conversation; there are a lot of threads that I’d like to pick up, and hopefully continue this week.

    Rey started us off with a great quote about ‘standing on the shoulders of giants’. Interesting there is quite a lot of controversy as to who actually said this, but that’s an aside for later. I like the term as it especially captures that we are trying to work with the resources and practices of others to not only learn in our own practice, but extend that practice. We’re accustomed to doing this in research (in fact we’re required to review the literature), but it doesn’t always happen in learning and teaching. Thanks for the reminder, Rey.

    Tom reminded us of the place for recognising previous educational experience, and – if I can extend the notion – essentially the recognition of prior learning. Open education does provide some complex challenges in this space, especially in light of MOOCs/mOOCs and professional accreditation and recognition. If we are true to the idea that learning can happen ‘anywhere, anytime’ then are we are called upon – as educational institutions – to consider how that fits with our enterprise?

    Alison raised a great point about Market, State, and Commons (perhaps the new ‘iron triangle’ for business courses?), and she is completely correct – the balance of these inter-related agencies creates a lot of tension and complexity for universities. Ideally, the Commons should sit apart from these, as a ‘common wealth’ of knowledge for the good of society, but convincing governments that this is worthwhile is not always easy when education is viewed as an export commodity (as it is in Australia).

    Thanks for the interaction, and the continuing conversation (I’m keen to explore Tom, and Rey’s comments about MOOCs further), and look forward to speaking with you all during the week.

  2. The first learning activity asks that we reflect on “what quality in learning and teaching means to you” but then goes on to ask us to pick an indicator of quality. To me those are completely different conversations. A quality course provides accurate and significant information or skills at an appropriate level and in an engaging manner. I don’t know of any indicator or metric that measures quality, especially when applied to a wide range of course subjects and levels. As the day two link states, I’ll know it when I see it. While I realise that it’s harder to manage an organisation or to compare different approaches without metrics, bad metrics are not better than no metrics.

    1. Hi Jack. You raise some good points about the ‘I know it when I see it’ challenge in quality. I was at a conference last week and attended a stream about quality in online courses that was a little worrisome because the main message was ‘we know our courses are high quality because an external commercial provider says so’. They proudly displayed the badges of certification, but there wasn’t a lot of discussion about what that actually means. In some ways, I wonder if this is essentially ‘outsourcing’ some of the course design process, or whether it’s a type of peer review (or perhaps both?).

      It really depends on what words we use to describe concepts. You mention that a quality course

      “provides accurate and significant information or skills at an appropriate level and in an engaging manner”

      and I’m of a similar mind. When I unpack that, it looks like a series a questions, such as:
      – Accurate and significant information: does the information support the learning activities? Is the information conceptually of an appropriate level for the audience? Does the information connect to the broader disciplinary narrative? IS the context of the information appropriate?
      – Significant skills – Does the course build upon skills developed in earlier courses, and how does it contribute to skills in later courses? (if we’re talking about a degree program). Are the outcomes aligned well with the assessment, activities, and content? Do students have opportunities to apply the skills in authentic situations?
      – Engaging manner – Who/What does the student engage with during the course? (lecturer, other students, content). Are the course materials presented in a range of formats? Is active learning the focus of the course design?

      I’m sure that you’d be able to add to this list. I suppose that my questions are about gathering evidence that these indicators exist appropriately in a learning experience, and to what extent; and then the presence of a certain number of key indicators would indicate quality. However, I’d also state that you’d need to know the purpose of the course. There is nothing wrong with a course that privileges student engagement with content; provided that this is the design intent and that it is appropriate to the level, and discipline. So, you’d need to ensure that there was nuance to the quality evaluation – a single set of metrics applied to all disciplines, at all levels is not an appropriate measure.

      As you say, no metrics are better than bad metrics (but I would add that bad metrics are a catalyst for creating better ones).

      What do you think? Are we using different words for the same concept; are we approaching this from very different perspectives, or something else? I’m keen to hear your thoughts, Jack.

  3. Quality in learning and teaching means achieving the planned outcomes. That is, can the students demonstrate they have the required knowledge and skills, as a result of their learning? This view perhaps comes from being trained in software development, which uses a definition of quality derived from manufacturing industry. When designing a course I first set out what the learning outcomes are, then how to test if the students meet those. Quality is simply if, or how many, of the students achieve the outcomes.

    1. Hi Tom. The focus on application as a way of providing evidence of learning, and meeting the intended outcomes is certainly one I can appreciate. Answering the question ‘why does this course exist?’, and ‘what do students do in this course?’ sometimes requires a lot of ‘unpacking’. Your approach to design (I still think it somewhat amusing that it’s referred to as ‘backwards design’ because to me it is common sense) resonates with my own approach – you can’t design much if you don’t know who it is for, or what the intended outcomes of the design are (at least in my opinion).

      So my question for you would be – do we need any extra indicators for open content, or do we evaluate open content/courses with the same tools or questions as we do their closed counterparts?

  4. Sorry, I missed the second paragraph of Learning Activity 3 (it was below the fold): I don’t think strong teacher-student interaction, or the resources selected are a sign of quality. These may well result in better outcomes for the course, but they are just part of the process, not outcomes which is where quality has to be measured.

  5. I help decide what quality looks like in educational practice for my profession. I have a say in the standards for practitioners, including the requirements formal qualifications issued by higher educational institutions in Australia, and around the world. It does not matter if the materials are open, or not, in terms of the quality of the final product: qualified professionals.

    1. Hi Tom,
      I agree with you – ” It does not matter if the materials are open, or not, in terms of the quality”. I think it is the education that is being judged for its quality, not the fact that it is OER or not.

  6. Learning Activity 3
    What does quality in learning and teaching mean to you?
    My own view – ‘getting results’ which is successful learning by the students, and that may be something learned for use right now or something learned for the longer haul where learners can come back to what they’ve learned again and again. (I guess that’s surface learning and deep learning in a way).
    The problem is – when we (teachers) build something, or even when we deliver something, how do we know if it will result in students learning something? If it’s an iterative process, then how many times do we get feedback? And how do we know if students improve their learning each time a course is run? Formal assessments might not be the best way to test that…

  7. Activity 4
    I agree in general terms with David Wiley’s list. I like the division between the two sides of the list, and agree that the first (left) side is the one to use for measuring quality. To me, a course only works if the course works. i.e. if something is learned from it.
    However, I don’t like the terms ‘direct’ and ‘indirect’. I know ‘direct’ means this is a direct measure of quality, but it can only be measured indirectly. Likewise, the indirect measures are the ones that can be measured directly.

    Re Is this applicable given it’s date – Yes. Each institution or organisation and person is in a different place in their journey around openness. This applies to some now like it would have then. The world may have moved on but not everyone in the world has moved at an equal pace, and I think this is still relevant to some near the beginning of their journeys.

  8. Activity Five:
    I’ve picked an interesting question: Ultimately, who decides what quality looks like in open educational practice?
    It really depends! The creator/compiler decides what quality is when they use OEP in a teaching resource.
    The purchaser of that teaching resource decides what quality is when they purchase it or not.
    The student decides what quality is when they engage with the course and work their way through it (or give up, if the quality is really bad).
    The employer or profession decides what quality is when they see how well the graduate of the teaching resource performs in a workplace situation.
    They might be four different indicaters of quality that we are talking about, but I think everyone will make some form of judgement about the ‘quality’ of a teaching resources built with OER as they do with any other teaching resource.

    1. I agree Alison, there are so many definitions or views of quality depending upon the person assessing the resource. Do you think there would be any indicators of quality in common with these different viewpoints? Or do you think it is best to conceptualize quality as a more nuanced concept that is different depending on the ‘audience’?

  9. Hi all,
    Here are my answers:

    Learning Activities 3 and 4 combined
    I agree with Wiley that student learning somehow needs to be measured so that it can serve as an indicator of OER quality. There is pre-test/post-test research. There is also descriptive research where students are observed and asked about their learning experience. Some examples of questions off the top of my head are: what hooked you into the material? Which explanations were not clear enough? What did it take for you to “get” a particular concept (accompanied by a measure to determine that the student did actually “get” the concept)? Does any part of the resource make you want to learn more independently? How do you think you can apply what you’ve learned?

    With regards to the timing of David’s blog post, I agree with it even though it’s five years old.

    All of Wiley’s listed proxies apply to OER. I just think that research resources are better spent directly involving students than on proxies. To me, OEP are actions and resources that support OER development, so I don’t think the proxies apply.

    Learning Activity 5
    Do OER need their own version of quality? I’m not sure. Closed and open online resources have a lot in common: it’s better if they’re highly interoperable, easy to use technically for learners including those with physical, vision, and hearing issues…

    In open education, it’s the whole community that should have the opportunity to decide about quality. For informal use, the individual teacher or learner can decide on the quality. For formal credit, the OER should be officially recognised by an accredited institution.

    “Do you think that mainstream measures of quality (such as peer-review) need to be applied to open resources in order to normalise the practice of using them?” I think it would help, but honestly, there are too many OER that are poorly organised in way too many OER repositories for a peer-review system to be valid or practical. A three-star rating out of five stars means nothing to me. I need to look at the resource myself to judge whether it suits a learning purpose. So what would really help is a way to find relevant OER more easily.

  10. Activity 3.
    I think, from the most general point of view, quality is the essential determination of the object (object is understood here in wide sense, not just a material thing).
    But, when one is talking specifically on quality in education and its components, I am agree with Tom’s point of view: “Quality in learning and teaching means achieving the planned outcomes”. And I add: quality can be measured by the changes, taken place in learners and their correspondences with the aims of activities, which should correspond to some needs.
    I think there are three different, but connected “moments” of quality in teaching-learning process: quality of design (it takes places before beginning the learning process), quality of process, and finally, quality of the results, the outcomes. The last one depends from previous ones, but it doesn’t mean that if they are the measure of final quality, although they could work as indicators, heuristic of final quality. Frequently they are based on correlations, obtained from good experiences of particular courses.

  11. Activity 4.
    Regarding first question: I agree with author’s perception, in fact the right column corresponds to indicators of design quality, as I wrote in Activity 3 post. When considering the design of OER, those are indicators that can garantee “a priori” an expectation of quality. But when refering to OEP, they don’t garantee the quality of result.
    In my Institution it is a problem we have to deal. Mostly quality measerument is done based on indicators in the design phase.

    1. I like your description of ‘a priori’ expectation of quality; I see this happen when an institution ‘trades on its’ name’ as an indicator of quality. MOOC providers in particular exploit this when they make decisions to only work with ‘the top 300 research institutions in the world’ or similar. The idea is that the prestige of the university rubs off on the MOOC provider, and being involved with MOOCs guaranteed (at least a few years ago) media coverage. I agree that it certainly doesn’t apply in an open sense.
      What I also recognise is your statement about quality standards in the design phase. I’ve used indicators (I won;t say standards as that’s a completely different level of audit) to not only review existing materials (with an eye to enhancing them), but also as a gauge when creating resources. In many ways, this type of work involving quality indicators needs to be very flexible. It can also give a practitioner aspirational goals for practice and resources. I’ve had many conversations that included a phrase such as ‘the resource really does meet all of the base indicators for good practice, but have you also considered….?’, so the emphasis here is not just on reviewing current quality, but also setting future targets.

      Also, I wanted to comment about your three areas of quality (design, process, and outcomes) because I think you’ve been able to articulate something that many of us might implicitly do, but perhaps struggle to describe. I really like the delineation. One query for you though – is it possible for one of these three to have low quality, but evidence of high quality to be found in the other two? Or do you see these as inter-dependent with a low quality indicator immediately disadvantaging work in others? I’m going to need to think about this and return to it, so your thoughts would be most welcomed.

      Do hope you have a good weekend.

      1. I think there several variants, and among them could be the case when a bad design is present (or there is not design at all), but the proccess work fine (it could be fixed, for example, by means of good communication during activities, supporting student activities, helping them, including new resources, omited initially).

        Even might be, that one has a bad design (or not design), a bad proccess (because, for example, teachers and tutor don’t support, help, advice students). But the outcome might be good, even excelente, due the selfregulated activities of student, who, as an active actor, having the possibility to search, make choice, etc., can fix, can complement the lack correct design and teaching. In this regard I think, OEP can help to overcome situations like described.

        There is some dependence in those phases, but they are not total, complete dependencies. There are factors, acting outside,as part fo the context, as it was explained in post 5.

  12. Activity 5.
    OER are a kind of ER. Then they have some general properties, shared with all ER. But they have specific properties, related mainly with the concept of open, as it was discussed in Day 1. Consequently, OER need a new, specific version of quality, but preserving several aspects of general conceptualization of quality for ER.

Leave a Reply

Your email address will not be published. Required fields are marked *