Defining quality in online education: engagement and outcomes

This article presents a model for quality in online education to support decision-making by learning designers, digital media specialists and academic colleagues in the design and development of online courses. Whilst there are many standards and, in the case of formal education, regulatory frameworks that provide criteria against which online courses may be measured, these are often broad and require significant interpretation. 

There are many aspects to defining quality in and of an online educational experience, and there is the risk that a unifying model elaborates to include a myriad of pedagogical, technical and operational factors. The model presented here simplifies criteria into just two categories: engagement and outcomes. The efforts of the learning designer are therefore to guide the design and development of an online course to ensure both that learners are engaged and learners are able to successfully achieve outcomes. However, rather than measuring quality of course production outputs, quality is defined by a collaborative process. Decision-making in design, development and teaching is therefore grounded with two questions:

• In what ways are learners supported to be engaged with learning?
• In what ways are learners able to achieve outcomes?

Outcomes are defined here deliberately vague, meaning both individual outcomes as well as educational, institutional and socio-political outcomes.

The following explores definitions of quality online learning and online education, drawing upon regulatory frameworks, recognised standards, educational research and the importance of multiple viewpoints bringing quality through process.

An introduction to quality for online education

Quality may be interpreted in different ways by different educators teaching within different educational contexts. This is not a new idea. Garrison (1993), in exploring quality in distance education, made this point with reference to whether there is an institutional predisposition towards scale (and an emphasis on pre-packaged materials with an implied determined learning path and outcomes), or interaction (to simulate a face-to-face approach, and an emphasis on higher-order learning and discussion), and how this influences how quality of the educational experience is defined. The quality argument presented is between access (scale) and interaction (quality), with ‘computer-mediated communication’ (as was known in the early days of online education) as a potential bridge in fostering communication between educator and student, and student and other students (Garrison, 1993). Since then, creating online education at scale and fostering quality interaction are no longer opposite ends of a spectrum, with advances in both pedagogy and technology, combined with learning design, leading to demonstrable excellence and reach of online learning. However, quality extends beyond the educational approach. 

As a specific model for online higher education, Vivolo’s (2018) Pyramid of Online Learning Support demonstrates the interplay and interdependence between learners, educators, educational professionals and other specialists. Learning designers are one of these roles, providing support in online pedagogy and adopting quality assurance responsibilities, in particular where quality is defined by alignment of outcomes and assessment, level of critical thinking and learning collaboration (Stucklen, 2018). However, involving a range of educational expertise needs relevant supporting course design and development processes, so that specialist contributions can be brought in at key stages of review (quality assurance) (Melton, 2002). Embedding quality throughout the design process sets an expectation of iteration, collective endeavour and shared responsibility of those involved. This is different from, and complements, quality control points where typically quality is assessed by those outside of the development team and after potentially significant time has been invested, but both approaches are necessary for ensuring quality (Melton, 2002).

With this in mind, quality is defined not by descriptions of outputs, but by the processes that underpin them. To support decision-making in these processes, educators and educational professionals draw upon frameworks that provide prompt questions, checklists and standards, guiding choices over activity, assessment and platform. It is argued here that these choices fall largely into decisions about how learners are supported to be engaged in the learning process and how they are able to achieve outcomes.

Policy context for quality

Within the UK, the Office for Students (OfS) is the regulatory body for higher education in England, and has a legal duty to hold institutions to account in terms of quality to ‘sector-recognised standards’ (Higher Education and Research Act 2017). The regulatory framework defines indicators of positive outcomes as students: continuing in their studies; completing their studies; progressing into managerial or professional employment, or further study; any other areas as determined by the OfS (OfS, 2022). These measurable and dispassionate outcomes are a proxy for value attributed to the educational experience, but are not by themselves a measure of quality. Indeed, there are some possible contradictions on quality as defined by baseline expectations and metrics, and the scope for risk-taking and innovation to push forward education in disciplines and for new professions as yet to emerge.

“The conditions of registration for quality and standards that apply to individual providers are designed to ensure a minimum baseline of protection for all students and the taxpayer. Beyond this minimum, autonomous providers are free to pursue excellence and innovation as they see fit, and the OfS will use its sector level tools to create the space for this to happen… The OfS has adopted the TEF [Teaching Excellence Framework] as a sector level intervention to promote excellence in teaching, learning and student outcomes beyond the minimum baseline.”

(OfS, 2022, p.25)

Within the regulatory framework, specifically Part V, Quality and Standards Conditions, the broad language defining quality leaves much to interpretation. There are some fundamental basics to quality which are defined, such as logical progression of curriculum design, appropriate level of challenge and currency of subject matter. References, albeit as examples, to specific forms of learning, “an appropriate balance between delivery methods, for example lectures, seminars, group work or practical study, as relevant to the content of the course” (OfS, 2022, p.91), may set a precedent for somewhat traditional approaches. However, with a more optimistic view, there are some opportunities for creativity and educational research to influence practice, as evident by defining ‘up-to-date’ to mean both subject matter and “being appropriately informed by recent… developments in teaching and learning, including learning resources” (OfS, 2022, p.92).

“B1.3 For the purposes of this condition, a high quality academic experience includes but is not limited to ensuring all of the following: a. each higher education course is up-to-date; b. each higher education course provides educational challenge; c. each higher education course is coherent; d. each higher education course is effectively delivered; and e. each higher education course, as appropriate to the subject matter of the course, requires students to develop relevant skills.”

(OfS, 2022, p.90)

To support interpretation of broad criteria, some examples and counterexamples are provided and these include direct reference to online education, though, of course, the specific pedagogy is not defined. This maintains the value of the highly collaborative endeavour of the design and development of courses between educators, learning designers, learning technologists and educational developers, each bringing their own expertise and awareness of discipline and educational research and practice.

“In relation to ‘up-to-date’ and ‘appropriately informed’… b. The pedagogy of a course is not representative of current thinking and practices. For example, a course delivered wholly or in part online that does not use pedagogy appropriate to digital delivery, would likely be of concern…

In relation to ‘effectively delivered’… a. The range of learning activities is too narrow… b. Learning activities are not delivered effectively. For example, insufficient opportunities for students to engage directly with teaching staff, including where parts of a course are delivered remotely, would likely be of concern… e. Assessment is not designed into the course. For example, assessment activities not being scheduled to consolidate students’ learning, or feedback not sufficient or timely to support learning, would likely be of concern.”

(OfS, 2022, p.93-95)

The OfS framework draws attention to outcomes and whilst it does include reference to engagement, the interpretation is around student and institutional engagement. For example, involvement in governance and programme committees, provision of student experience feedback and contribution to the development of the academic experience (OfS, 2022, p.99-105), rather than engagement in learning. Therefore specific forms of engagement and interaction at the course or activity level are not defined, beyond the counterexample shown above. 

The Quality Assurance Agency (QAA), formerly with the quality responsibilities now adopted by the OfS in England (though continued responsibilities for Scotland, Wales and Northern Ireland), provides a Quality Code which is complemented with established Subject Benchmarks. Defining a quality online educational programme is therefore a combination of a number of factors, including discipline practice, student experience, learning environment, institutional processes and resources. Deliberately using the language of the OfS framework, the Quality Code is introduced as a means for institutions based in England “to satisfy themselves that their quality processes support enhancement above the baseline” (QAA, 2023). The QAA supporting guidance is extensive and provides prompt questions and sets expectations of how quality can be met through process. This is key and a clear distinction from metric-driven assessments of quality. By adopting an approach which looks at the development of education, rather than measuring outcomes, quality is by definition able to be more attune to specific needs of disciplines, learners and adopt iterative educational methodologies. 

Quality frameworks relevant to online education

There are two notable frameworks with international prominence which offer the next layer of detail for defining quality, Quality Matters (QM) and Online Learning Consortium (OLC) Quality Scorecards. Both adopt checklist-style approaches to assessing not just the quality of produced online courses, but the support, infrastructure and processes for effective operations in online or higher education. The broad headings under which criteria are provided for course design and development are listed below:

QM Higher Education Rubric, Seventh Edition

• Course Overview and Introduction
• Learning Objectives (Competencies)
• Assessment and Measurement
• Instructional Materials
• Learning Activities and Learner Interaction
• Course Technology
• Learner Support
• Accessibility and Usability

(Quality Matters, 2023a)

OLC Quality Course Teaching and Instructional Practice

• Course Design
• Accessibility, ADA Compliance, Universal Design
• Course Learning Outcomes
• Course Content
• Assignments
• Instructor Role
• Class Discussion and Engagement
• Building Community
• Communication
• Continuous Course Improvement

Online Learning Consortium (2016)

Both standards break down these areas into detailed criteria, with both outcomes and engagement forming a significant proportion of the checklists. Indeed, both QM and OLC define quality objectives and outcomes across a number of criteria and sub-sections, for example relating decisions over activities, assessment and materials to how they support learning outcomes. Knowledge and experience of good practice in education, learning design, instructional design and technical expertise is still required to implement these within an educational context to create effective online courses that are student-centred. Engagement is grounded in active learning, requiring clear expectations of participation and task outputs to be defined, instructor involvement and OLC go as far as explicitly stating reflective and critical thinking in quality criteria.

The Quality Matters framework is accompanied by an iterative model that underpins a quality process, setting the expectation of feedback, course review, course revision and meeting quality expectations (Quality Matters, 2023b). This is an important addition to the concept of quality, not as defining outputs, but a quality cycle that leads to adaptation and change. Indeed, this echoes back to OfS standards of currency and relevance in a more broader sense.

Pedagogy, learning design and quality

The quality frameworks available are largely defined independent of disciplines and specific pedagogies, although encourage quality of process in learning design. This allows for both measurably good learning design, but also latitude for ineffective teaching and learning practices to thrive. The implementation of any of these quality standards requires an understanding of the discipline, its practices and its learners. 

Pedagogic frameworks assist in learning design, providing a structure to think about the activities of a learner, their cohort and the educator. There are some highly regarded and effective frameworks that have been applied to online education, such as Laurillard’s (2012) Conversational Framework and Conole’s (2015) 7C’s Learning Design Framework, which support thinking about the design for learning. However, applying a particular design or pedagogic approach without discipline awareness can actually lead to an incongruous learning experience. For example, applying a social learning pedagogy to mathematics may be less effective than a framework that represents learning through modelling, peer teaching, think aloud or other more discipline-aligned approaches. Similarly, a didactic pedagogical approach in social science disciplines may limit the opportunity to discuss nuance of context and interpretation possible through social learning, discussion forums and critical reading. Therefore, the lack of definition over specific pedagogical or design frameworks in quality standards is very much in keeping with the idea that best practice is not generalisable (Steele, et al., 2019).

Beyond pedagogy, there is also the intrinsic relationship between the learning environment and the learning experience. The choice of tool, method of interaction, space and timeliness for participation all impact on the learning experience, defined by the capabilities of the online learning environment. It is for these reasons that applying a sound pedagogical framework alone does not define a quality online learning experience. In fact, in each of the three standards above, OfS, QM and OLC, the quality indicators of course experience are broader than the pedagogy and expand into practical and organisational considerations of teaching and learning online. There is recognition that no matter how well-designed and produced course materials and activities are, the structures supporting online education must also foster learner engagement.

Quality engagement

Educational activities, instruction, the relationship and interactions between educators and learners, and the experience of learning online all play a part in learner engagement (Bergdahl, 2022). Learning designers and educators draw upon a wide range of approaches to engage learners in the process of learning. The science of learning and research in educational neuroscience offers a basis for understanding the relationship between motivation to learn, emotional engagement, learning with others and reward. Technically, this form of engagement is neurobiological, an effect in terms of brain response, rather than externalised engagement in class or participation in activity. However, research is showing links between this form of engagement and learning in terms of memory.

For example, the effect of ‘novelty’ leading to short-term gains on motivation (Schomaker and Meeter, 2015), the connection between ‘motivationally important events’ and long term memory (Shohamy and Adcock, 2010), the potential for curiosity and unexpected answers to lead to better recall (Kang, et al., 2009), and increased engagement through uncertain rewards in gamified learning (Howard-Jones and Jay, 2016). These acts are intended to stimulate the learner to be motivated and interested in the experience of learning. There is of course a balance required in the instructional design of such activities, noting theories of cognitive load (Sweller, et al., 1998) and the need to scaffold complex concepts. Engagement through novelty of online interaction frequently leads to the trap of distracting interfaces or frustrating locked-in sequences of clicks to progress through materials or activities. Arguably, quality may be defined in the way the learning and instructional design process seeks to adopt educational research in the most appropriate way for any given context.

Complementing the scientific perspective of learning as a neurological process, engagement in the educational process considers the broader learning context, including who and where learning is taking place. Digital accessibility is another aspect of engagement that can be considered, in terms of both the functionality of platforms and structure of learning content. These considerations are again reflected in the OLC and QM standards, but also reflect the relationship between quality of learning experience and an understanding of the learners. 

Quality of interaction and developing motivation of learners to participate extends beyond the scope of learning materials or activity design, and requires an awareness of the participants of the learning experience. For example, Bergdahl, et al. identified learner competences in online learning as a factor of engagement and disengagement in otherwise distracting technology-enhanced learning environments, and noted how “high-performance students… [who] have developed digital competence that support them to use technology for learning in productive ways” (2020, p.11). This finding was true for both individual and collaborative activities, and is a strong indicator that there is a role for educators to explicitly support learning engagement for all learners, fostering self-efficacy and ability to learn online, along with well-designed instruction.

Engagement is also an indicator of relevance of the learning experience to individual learners. Looking at four dimensions of engagement (behavioural, social, cognitive and emotional), Bergdahl (2022) discovered conflicting findings of both engagement and disengagement within a single dimension. The implication is that such contradictions reflect learners’ priorities about how and what they will engage with; there is, perhaps unsurprisingly, individual choice and difference. Using the four dimensions engagement model offers an interesting prospect for learning design (as well as evaluation and analysis), considering where educator activity should be prioritised to support engagement. This leads to questions of what content and activities will be acceptable to ‘let go’, and where choice and self-efficacy to prioritise different aspects will be fostered and encouraged, enabling the experience of learning to be more personal and meaningful to learners.

Engagement and outcomes

The connection here between engagement and outcomes is obvious. From both a motivational perspective, learning that aligns with personal interests and curiosity will engage learners, enabling them to achieve the performative aspect of the educational experience for measurable outcomes. Similarly, considering how outcomes may be determined by learners themselves, particularly in professional learning settings, the ownership of the learning experience combined with self-efficacy of learners will increase their ability to engage meaningfully to achieve those outcomes.

From a regulatory perspective, much is made of outcomes-based measures that distil the learning experience into a series of statistics aggregated across a population. Though suggested as indicators of quality, their purpose is to summarise for comparison, allowing for performance tables and, in some cases, metric-driven funding decisions. Further, the upcoming THE Online Learning Rankings, although not yet disclosed in full detail, will measure against resources (relating to dedicated funding), student engagement (some measure of interaction), student outcomes (quantitative), and the learning environment (technical support and some measure of community) (THE, 2023). These types of indicators disguise the many nuances of individual learners’ outcomes and motivations and how learning experiences may be crafted to enable personalised learning, differentiated learning experiences and learning outcomes that are broader and more aspirational. In placing an emphasis on outcomes that represent a single point in time, whether that is the course completion grade or employment statistic six months hence, this leads to quality decisions which ignore the potential opportunities of learning that may not be realised within the confines of the designed curriculum or ambitions that take much longer to enact in individuals’ lives.

The range of parameters that define quality in online education show clearly that quality is not a measure of a produced output. Quality is not solely determined by the glossiness of the multimedia, the meaningfulness of a learning activity, the intuitiveness of the platform or even the expertise of the educator. A key distinctive descriptor of online education is that it is designed and planned, particularly to foster asynchronous engagement and enable flexible learning, and therefore the time and expertise invested in the pre-teaching phase is critical to the success of a course. All educators and educational professionals working in online education should take personal responsibility for the quality of their own contributions, and as a collective, can work to understand their contribution to both learner outcomes and learner engagement through quality discussions and processes. It may be argued then that the quality of the online educational experience is therefore not in the measured outcomes, nor even in quantitative measures of engagement, but in the process of design, development and teaching that provides the structure and scaffolding for individual learners to flourish.

References

1. Bergdahl, N. (2022). Engagement and disengagement in online learning, Computers & Education, 188.
2. Bergdahl, N., Nouri, J., Fors, U., Knutsson, O. (2020). Engagement, disengagement and performance when learning with technologies in upper secondary school, Computers & Education, 149.
3. Conole, G. (2015). Designing effective MOOCs, Educational Media International, 52(4), 239-252.
4. Garrison, D. R. (1993). Quality and access in distance education: theoretical underpinnings. In: Keegan, D. (ed.), Theoretical Principles of Distance Education. London: Routledge.
5. Higher Education and Research Act 2017 (c.29). London: The Stationery Office.
6. Howard-Jones, P. and Jay, T. (2016). Reward, learning and games, Current Opinion in Behavioral Sciences, 10, 65-72.
7. Kang, M. J., Hsu, M., Krajbich, I. M., Loewenstein, G., McClure, S. M., Wang, J. T., Camerer, C. F. (2009). The Wick in the Candle of Learning: Epistemic curiosity activates reward circuitry and enhances memory, Psychological Science, 20(8), 963-973.
8. Laurillard, D. (2012). Teaching as a Design Science. Abingdon: Routledge.
9. Melton, R. F. (2002). Planning and Developing Open and Distance Learning: A Quality Assurance Approach. London: RoutledgeFalmer.
10. Office for Students (2022). Securing student success: Regulatory framework for higher education in England. London: The Stationery Office.
11. Online Learning Consortium (2016). Quality Course Teaching and Instructional Practice, Quality Scorecard.
12. QAA (2023). The UK Quality Code for Higher Education. Quality Assurance Agency.
13. Quality Matters (2023a). Course Design Rubric Standards (7th Ed.).
14. Quality Matters (2023b). QM Course Review Process.
15. Schomaker, J. and Meeter, M. (2015). Short- and long-lasting consequences of novelty, deviance and surprise on brain and cognition, Neuroscience and Biobehavioural Reviews, 55, 268-279. 
16. Shohamy, D. and Adcock, R. A. (2010). Dopamine and adaptive memory, Trends in Cognitive Sciences, October 2010, 14(10), 464-472.
17. Steele, J., Holbeck, R., Mandernach, J. (2019). Defining effective online pedagogy, Journal of Instructional Research, 8(2), 5-8.
18. Stucklen, E. G. (2018). Understanding and ensuring quality: Designing assessments and evaluation for online learning. In: Vivolo, J. (ed.), Managing Online Learning: The Life-cycle of Successful Programs. Abingdon : Routledge.
19. Sweller, J., Van Merrienboer, J. J. G., Paas, F. G. W. C. (1998). Cognitive architecture and instructional design, Educational Psychology Review, 10(3), 251-296.
20. THE (2023). THE announces a new ranking for online learning. Times Higher Education [online]. November 23 2023.
21. Vivolo, J. (2018). Managing Online Learning: The Life-cycle of Successful Programs. Abingdon: Routledge.