Moore Public Schools Literacy Design Collaborative Training Video

Well-planned online learning experiences are meaningfully different from courses offered online in response to a crisis or disaster. Colleges and universities working to maintain instruction during the COVID-19 pandemic should understand those differences when evaluating this emergency remote teaching.

a person pushing a cube and a person pushing a sphere — *Credit: frankie's / Shutterstock.com © 2020*

Due to the threat of COVID-19, colleges and universities are facing decisions about how to continue teaching and learning while keeping their faculty, staff, and students safe from a public health emergency that is moving fast and not well understood. Many institutions have opted to cancel all face-to-face classes, including labs and other learning experiences, and have mandated that faculty move their courses online to help prevent the spread of the virus that causes COVID-19. The list of institutions of higher education making this decision has been growing each day. Institutions of all sizes and types—state colleges and universities, Ivy League institutions, community colleges, and others—are moving their classes online.¹ Bryan Alexander has curated the status of hundreds of institutions.²

Moving instruction online can enable the flexibility of teaching and learning anywhere, anytime, but the speed with which this move to online instruction is expected to happen is unprecedented and staggering. Although campus support personnel and teams are usually available to help faculty members learn about and implement online learning, these teams typically support a small pool of faculty interested in teaching online. In the present situation, these individuals and teams will not be able to offer the same level of support to all faculty in such a narrow preparation window. Faculty might feel like instructional MacGyvers, having to improvise quick solutions in less-than-ideal circumstances. No matter how clever a solution might be—and some very clever solutions are emerging—many instructors will understandably find this process stressful.

The temptation to compare online learning to face-to-face instruction in these circumstances will be great. In fact, an article in the Chronicle of Higher Education has already called for a "grand experiment" doing exactly that.³ This is a highly problematic suggestion, however. First and foremost, the politics of any such debate must be acknowledged. "Online learning" will become a politicized term that can take on any number of meanings depending on the argument someone wants to advance. In talking about lessons learned when institutions moved classes online during a shutdown in South Africa, Laura Czerniewicz starts with this very lesson and what happened around the construct of "blended learning" at the time.⁴ The idea of blended learning was drawn into political agendas without paying sufficient attention to the fact that institutions would make different decisions and invest differently, resulting in widely varying solutions and results from one institution to another. With some of that hindsight as wisdom, we seek to advance some careful distinctions that we hope can inform the evaluations and reflections that will surely result from this mass move by colleges and universities.

Online learning carries a stigma of being lower quality than face-to-face learning, despite research showing otherwise. These hurried moves online by so many institutions at once could seal the perception of online learning as a weak option, when in truth nobody making the transition to online teaching under these circumstances will truly be designing to take full advantage of the affordances and possibilities of the online format.

Researchers in educational technology, specifically in the subdiscipline of online and distance learning, have carefully defined terms over the years to distinguish between the highly variable design solutions that have been developed and implemented: distance learning, distributed learning, blended learning, online learning, mobile learning, and others. Yet an understanding of the important differences has mostly not diffused beyond the insular world of educational technology and instructional design researchers and professionals. Here, we want to offer an important discussion around the terminology and formally propose a specific term for the type of instruction being delivered in these pressing circumstances: emergency remote teaching.

Many active members of the academic community, including some of us, have been hotly debating the terminology in social media, and "emergency remote teaching" has emerged as a common alternative term used by online education researchers and professional practitioners to draw a clear contrast with what many of us know as high-quality online education. Some readers may take issue with the use of the term "teaching" over choices such as "learning" or "instruction." Rather than debating all of the details of those concepts, we selected "teaching" because of its simple definitions—"the act, practice, or profession of a teacher"⁵ and "the concerted sharing of knowledge and experience,"⁶—along with the fact that the first tasks undertaken during emergency changes in delivery mode are those of a teacher/instructor/professor.

Effective Online Education

Online education, including online teaching and learning, has been studied for decades. Numerous research studies, theories, models, standards, and evaluation criteria focus on quality online learning, online teaching, and online course design. What we know from research is that effective online learning results from careful instructional design and planning, using a systematic model for design and development.⁷ The design process and the careful consideration of different design decisions have an impact on the quality of the instruction. And it is this careful design process that will be absent in most cases in these emergency shifts.

One of the most comprehensive summaries of research on online learning comes from the book Learning Online: What Research Tells Us about Whether, When and How.⁸ The authors identify nine dimensions, each of which has numerous options, highlighting the complexity of the design and decision-making process. The nine dimensions are modality, pacing, student-instructor ratio, pedagogy, instructor role online, student role online, online communication synchrony, role of online assessments, and source of feedback (see "Online learning design options").

Online learning design options (moderating variables)

Modality
- Fully online
- Blended (over 50% online)
- Blended (25–50% online)
- Web-enabled F2F
Pacing
- Self-paced (open entry, open exit)
- Class-paced
- Class-paced with some self-paced
Student-Instructor Ratio
- < 35 to 1
- 36–99 to 1
- 100–999 to 1
- > 1,000 to 1
Pedagogy
- Expository
- Practice
- Exploratory
- Collaborative
Role of Online Assessments
- Determine if student is ready for new content
- Tell system how to support the student (adaptive instruction)
- Provide student or teacher with information about learning state
- Input to grade
- Identify students at risk of failure
Instructor Role Online
- Active instruction online
- Small presence online
- None
Student Role Online
- Listen or read
- Complete problems or answer questions
- Explore simulation and resources
- Collaborate with peers
Online Communication Synchrony
- Asynchronous only
- Synchronous only
- Some blend of both
Source of Feedback
- Automated
- Teacher
- Peers

Within each of these dimensions, there are options. Complicating matters, not all of the options are equally effective. For example, decisions around class size will greatly constrain what strategies you can use. Practice and feedback, for example, are well established in the literature, but it's harder to implement this as class size grows, eventually reaching a point where it's just not possible for an instructor to provide quality feedback. In the case of synchrony, what you choose will really depend on your learners' characteristics and what best meets their needs (adult learners require more flexibility, so asynchronous is usually best, perhaps with optional synchronous sessions, whereas younger learners benefit from the structure of required synchronous sessions).

Research on types of interaction—which includes student–content, student–student, and student–instructor—is one of the more robust bodies of research in online learning. In short, it shows that the presence of each of these types of interaction, when meaningfully integrated, increases the learning outcomes.⁹ Thus, careful planning for online learning includes not just identifying the content to cover but also carefully tending to how you're going to support different types of interactions that are important to the learning process. This approach recognizes learning as both a social and a cognitive process, not merely a matter of information transmission.

Those who have built online programs over the years will attest that effective online learning aims to be a learning community and supports learners not just instructionally but with co-curricular engagement and other social supports. Consider how much infrastructure exists around face-to-face education that supports student success: library resources, housing, career services, health services, and so on. Face-to-face education isn't successful because lecturing is good. Lectures are one instructional aspect of an overall ecosystem specifically designed to support learners with formal, informal, and social resources. Ultimately, effective online education requires an investment in an ecosystem of learner supports, which take time to identify and build. Relative to other options, simple online content delivery can be quick and inexpensive, but confusing that with robust online education is akin to confusing lectures with the totality of residential education.

Typical planning, preparation, and development time for a fully online university course is six to nine months before the course is delivered. Faculty are usually more comfortable teaching online by the second or third iteration of their online courses. It will be impossible for every faculty member to suddenly become an expert in online teaching and learning in this current situation, in which lead times range from a single day to a few weeks. While there are resources to which faculty can turn for assistance, the scale of change currently being required on many campuses will stress the systems that provide those resources and most likely will surpass their capacities. Let's face it: many of the online learning experiences that instructors will be able to offer their students will not be fully featured or necessarily well planned, and there's a high probability for suboptimal implementation. We need to recognize that everyone will be doing the best they can, trying to take just the essentials with them as they make a mad dash during the emergency. Thus, the distinction is important between the normal, everyday type of effective online instruction and that which we are doing in a hurry with bare minimum resources and scant time: emergency remote teaching.

Emergency Remote Teaching

In contrast to experiences that are planned from the beginning and designed to be online, emergency remote teaching (ERT) is a temporary shift of instructional delivery to an alternate delivery mode due to crisis circumstances. It involves the use of fully remote teaching solutions for instruction or education that would otherwise be delivered face-to-face or as blended or hybrid courses and that will return to that format once the crisis or emergency has abated. The primary objective in these circumstances is not to re-create a robust educational ecosystem but rather to provide temporary access to instruction and instructional supports in a manner that is quick to set up and is reliably available during an emergency or crisis. When we understand ERT in this manner, we can start to divorce it from "online learning." There are many examples of other countries responding to school and university closures in a time of crisis by implementing models such as mobile learning, radio, blended learning, or other solutions that are contextually more feasible. For example, in a study on education's role in fragility and emergency situations, the Inter-Agency Network for Education in Emergencies examined four case studies.¹⁰ One of those cases was Afghanistan, where education was disrupted by conflict and violence and schools themselves were targets, sometimes because girls were trying to access education. In order to take children off the streets and keep them safe, radio education and DVDs were used to maintain and expand educational access and also were aimed at promoting education for girls.

What becomes apparent as we examine examples of educational planning in crises is that these situations require creative problem solving. We have to be able to think outside standard boxes to generate various possible solutions that help meet the new needs for our learners and communities. In some cases, it might even help us generate some new solutions to intractable problems, such as the dangers girls faced trying to access education in Afghanistan. Thus, it may be tempting to think about ERT as a bare-bones approach to standard instruction. In reality, it is a way of thinking about delivery modes, methods, and media, specifically as they map to rapidly changing needs and limitations in resources, such as faculty support and training.¹¹

In the present situation, the campus support teams that are usually available to help faculty members learn about and implement online learning will not be able to offer the same level of support to all faculty who need it. Faculty support teams play a critical role in the learning experiences of students by helping faculty members develop face-to-face or online learning experiences. Current support models might include full-course design support, professional development opportunities, content development, learning management system training and support, and multimedia creation in partnership with faculty experts. Faculty who seek support typically have varying levels of digital fluency and are often accustomed to one-on-one support when experimenting with online tools. The shift to ERT requires that faculty take more control of the course design, development, and implementation process. With the expectation of rapid development of online teaching and learning events and the large number of faculty in need of support, faculty development and support teams must find ways to meet the institutional need to provide instructional continuity while helping faculty develop skills to work and teach in an online environment. As such, institutions must rethink the way instructional support units do their work, at least during a crisis.

The rapid approach necessary for ERT may diminish the quality of the courses delivered. A full-course development project can take months when done properly. The need to "just get it online" is in direct contradiction to the time and effort normally dedicated to developing a quality course. Online courses created in this way should not be mistaken for long-term solutions but accepted as a temporary solution to an immediate problem. Especially concerning is the degree to which the accessibility of learning materials might not be addressed during ERT. This is but one reason that universal design for learning (UDL) should be part of all discussions around teaching and learning. UDL principles focus on the design of learning environments that are flexible, inclusive, and student-centered to ensure that all students can access and learn from the course materials, activities, and assignments.¹²

Evaluating Emergency Remote Teaching

Institutions will certainly want to conduct evaluations of their ERT efforts, but what should they evaluate? First, let's consider what not to evaluate. A common misconception is that comparing a face-to-face course with an online version of the course constitutes a useful evaluation. This type of assessment, known as a media comparison study, provides no real value, for at least three reasons:

First, any medium is simply a way to deliver information, and one medium is not inherently better or worse than any other medium. Second, we need to better understand different media and the way people learn with different media to design effective studies. And, third, there are too many confounding variables in even the best media comparison study for the results to be valid and meaningful.¹³

Researchers who conduct media comparison studies are looking at "the whole unique medium and [giving] little thought to each one's attributes and characteristics, to learner needs, or to psychological learning theories."¹⁴

Other approaches to evaluation can be useful in this move to ERT. The success of distance and online learning experiences can be measured in a variety of ways, depending on how "success" is defined from a given stakeholder's perspective. From the faculty point of view, student learning outcomes would be of primary interest. Did learners achieve the intended knowledge, skills, and/or attitudes that were the focus of the instructional experience? Attitudinal outcomes are also possibly of interest, for students and for faculty. For students, issues such as interest, motivation, and engagement are directly connected to learner success and so would be possible evaluation foci. For faculty, attitudes toward online instruction and all that it entails can affect the perception of success.

Programmatic outcomes such as course and program completion rates, market reach, faculty time investments, impacts on promotion and tenure processes—all of these are relevant issues related to the offering of distance courses and programs. Finally, implementation resources and strategies are possible areas of evaluation inquiry, such as the reliability of selected technological delivery systems, the provision of and access to learner support systems, support for faculty professional development for online teaching pedagogies and tools, policy and governance issues related to distance program development, and quality assurance. All of these factors can influence the effectiveness of distance and online learning experiences and can serve to inform learning experience design and program development and implementation.¹⁵ These recommended areas of evaluation are for well-planned distance or online learning efforts and may not be appropriate in the case of ERT. Evaluating ERT will require broader questions, especially during initial implementations.

Next, let us recommend where you should focus your evaluation related to ERT efforts. The language of the CIPP model will be used for structure.¹⁶ CIPP is an acronym representing context, inputs, process, and products (see table 1).

Table 1. CIPP evaluation terms

Context Evaluations	Input Evaluations	Process Evaluations	Product Evaluations
"Assess needs, problems, assets, and opportunities, as well as relevant contextual conditions and dynamics"	"Assess a program's strategy, action plan, staffing arrangements, and budget for feasibility and potential cost-effectiveness to meet targeted needs and achieve goals."	"Monitor, document, assess, and report on the implementation of plans."	"Identify and assess costs and outcomes—intended and unintended, short term and long term."

In the case of ERT, institutions might want to consider evaluation questions such as the following:

Given the need to shift to remote instruction, what internal and external resources were necessary in supporting this transition? What aspects of the context (institutional, social, governmental) affected the feasibility and effectiveness of the transition? (context)
How did the university interactions with students, families, personnel, and local and government stakeholders impact perceived responsiveness to the shift to ERT? (context)
Was the technology infrastructure sufficient to handle the needs of ERT? (input)
Did the campus support staff have sufficient capacity to handle the needs of ERT? (input)
Was our ongoing faculty professional development sufficient to enable ERT? How can we enhance opportunities for immediate and flexible learning demands related to alternative approaches to instruction and learning? (input)
Where did faculty, students, support personnel, and administrators struggle the most with ERT? How can we adapt our processes to respond to such operational challenges in the future? (process)
What were the programmatic outcomes of the ERT initiative (i.e., course completion rates, aggregated grade analyses, etc.)? How can challenges related to these outcomes be addressed in support of the students and faculty impacted by these issues? (product)
How can feedback from learners, faculty, and campus support teams inform ERT needs in the future? (product)

Evaluation of ERT should be more focused on the context, input, and process elements than product (learning). Note that we are not advocating for no evaluation of whether or not learning occurred, or to what extent it occurred, but simply stressing that the urgency of ERT and all that will take to make it happen in a short time frame will be the most critical elements to evaluate during this crisis. This is being recognized by some as a few institutions are beginning to announce changing to pass/fail options rather letter grades during ERT.¹⁷

Also, given the continued evidence of problems surrounding student evaluations of instruction under typical higher education experiences, we recommend that the standard, end-of-semester teaching evaluations definitely not be counted against faculty members engaged in ERT.¹⁸ If an institution's policy mandates that those evaluations be administered, consider amending the policy, or make sure that the results are clearly qualified with the circumstances of the term or semester.

Final Thoughts

Everyone involved in this abrupt migration to online learning must realize that these crises and disasters also create disruptions to student, staff, and faculty lives, outside their association with the university. So all of this work must be done with the understanding that the move to ERT will likely not be the priority of all those involved. Instructors and administrators are urged to consider that students might not be able to attend to courses immediately. As a result, asynchronous activities might be more reasonable than synchronous ones. Flexibility with deadlines for assignments within courses, course policies, and institutional policies should be considered. For a high-level example, the US Department of Education has relaxed some requirements and policies in the face of COVID-19.¹⁹

Hopefully the COVID-19 threat will soon be a memory. When it is, we should not simply return to our teaching and learning practices prior to the virus, forgetting about ERT. There likely will be future public health and safety concerns, and in recent years, campuses have been closed due to natural disasters such as wildfires, hurricanes, and the polar vortex.²⁰ Thus, the possible need for ERT must become part of a faculty member's skill set, as well as professional development programming for any personnel involved in the instructional mission of colleges and universities.

The threat of COVID-19 has presented some unique challenges for institutions of higher education. All parties involved—students, faculty, and staff—are being asked to do extraordinary things regarding course delivery and learning that have not been seen on this scale in the lifetimes of anyone currently involved. Although this situation is stressful, when it is over, institutions will emerge with an opportunity to evaluate how well they were able to implement ERT to maintain continuity of instruction. It is important to avoid the temptation to equate ERT with online learning during those evaluations. With careful planning, officials at every campus can evaluate their efforts, allowing those involved to highlight strengths and identify weaknesses to be better prepared for future needs to implement ERT.

Notes

See, for example: "Information for Ohio State University Students, Faculty and Staff," The Ohio State University, Wexner Medical Center; "President Eisgruber Updates University on Next Steps Regarding COVID-19 to Ensure Health and Well-Being of the Entire Community," Princeton University; and Everett Community College. ↩
"Coronavirus and Higher Education Resources," Bryan Alexander blog, March 17, 2020. ↩
Jonathan Zimmerman, "Coronavirus and the Great Online-Learning Experiment," Chronicle of Higher Education, March 10, 2020. ↩
Laura Czerniewicz, "What We Learnt from 'Going Online' during University Shutdowns in South Africa," PhilOnEdTech, March 15, 2020. ↩
"Teaching," Merriam-Webster. ↩
Daniela Peixoto Olo, Leonida Correia, and Maria da Conceição Rego, "The Main Challenges of Higher Education Institutions in the 21st Century: A Focus on Entrepreneurship," in Examining the Role of Entrepreneurial Universities in Regional Development, eds. Ana Dias Daniel, Aurora A.C. Teixeira, and Miguel Torres Preto (Hershey, PA: IGI Global, 2020): 1–23. ↩
Robert M. Branch and Tonia A. Dousay, "Survey of Instructional Design Models," Association for Educational Communications and Technology (AECT), 2015. ↩
Barbara Means, Marianne Bakia, and Robert Murphy,Learning Online: What Research Tells Us about Whether, When and How (New York: Routledge, 2014). ↩
Robert M. Bernard, Philip C. Abrami, Eugene Borokhovski, C. Anne Wade, Rana M. Tamim, Michael A. Surkes, and Edward Clement Bethel, "A Meta-Analysis of Three Types of Interaction Treatments in Distance Education," Review of Educational Research 79, no. 3 (2009): 1,243–89. ↩
Lynn Davies and Denise Bentrovato, "Understanding Education's Role in Fragility; Synthesis of Four Situational Analyses of Education and Fragility: Afghanistan, Bosnia and Herzegovina, Cambodia, Liberia," International Institute for Educational Planning (2011). ↩
For an explanation of method, media, and mode in online learning, see J. Thomas Head, Barbara B. Lockee, and Kevin M. Oliver, "Method, Media, and Mode: Clarifying the Discussion of Distance Education Effectiveness,"Quarterly Review of Distance Education 3, no. 3 (2002): 261–68. ↩
See "UDL On Campus," ↩
Daniel W. Surry and David Ensminger, "What's Wrong with Media Comparison Studies?" Educational Technology 41, no. 4 (July–August 2001). ↩
Barbara Lockee, Mike Moore, and John Burton, "Old Concerns with New Distance Education Research,"EDUCAUSE Quarterly 24, no. 2 (2001): 60–68. ↩
Mike Moore, Barbara Lockee, and John Burton, "Measuring Success: Evaluation Strategies for Distance Education,"EDUCAUSE Quarterly 25, no. 1 (2002): 20–26. ↩
Daniel L. Stufflebeam and Guili Zhang,The CIPP Evaluation Model: How to Evaluate for Improvement and Accountability (New York: Guilford Publications, 2017). ↩
For a discussion of institutions moving to pass/fail in response to COVID-19, see Allison Stanger, "Make All Courses Pass/Fail Now," Chronicle of Higher Education, March 19, 2020. ↩
For information about issues with student evaluation of instruction, see Shana K. Carpenter, Amber E. Witherby, and Sarah K. Tauber, "On Students'(Mis) judgments of Learning and Teaching Effectiveness,"Journal of Applied Research in Memory and Cognition, February 12, 2020. ↩
"Guidance for Interruptions of Study Related to Coronavirus (COVID-19)," Federal Student Aid, Information for Financial Aid Professionals (IFAP), March 20, 2020. ↩
Elin Johnson, "As Fires Rage, More Campuses Close," InsideHigherEd, October 29, 2019; Jenni Fink, "Florida Universities Cancelling Classes, Closing Campus Ahead of Potential Category 4 Hurricane Dorian," Newsweek, August 29, 2019; and Perry Samson, "The Coronavirus and Class Broadcasts," EDUCAUSE Review, March 3, 2020. ↩

Charles B. Hodges is a Professor of Instructional Technology at Georgia Southern University.

Stephanie Moore is an Assistant Professor of Instructional Design and Technology in the Curry School of Education at the University of Virginia.

Barbara B. Lockee is a Professor of Instructional Design and Technology, and Provost Faculty Fellow, at Virginia Tech.

Torrey Trust is an Associate Professor of Learning Technology at the University of Massachusetts Amherst.

M. Aaron Bond is Senior Director, Professional Development Network and Faculty Digital Fluency at Virginia Tech.