Skip to main content

In September of last year, I left what had been a dream job working in the Johns Hopkins Data Science Lab where I spent a significant amount of time developing a few much-loved online courses. I left Baltimore and accepted an offer to become a PhD student in Philip Guo’s lab at UC San Diego, where I am further pursuing my interests in both creating more humane online educational experiences and trying to develop a better understanding of how scientists do data analysis in order to understand and communicate the results of their own experiments.

Although I had significant experience creating online courses I only had a surface-level understanding of the current state of the art in online education research. In order to improve my understanding of current developments in the field I set off to read every paper published at the Learning at Scale conference, a relatively new venue for educational researchers and computer scientists to present the latest innovations in the world of technology-enabled education. Philip encouraged me to write and submit a meta-analysis type of paper to Learning at Scale, reviewing all of the papers published at the conference so far. I’m happy to announce that the paper was accepted and I will be presenting it at Learning at Scale 2018 in London! You can read a pre-print of the paper here, or you can read the “manifesto” version of the paper below.

MOOC Development is like Space Exploration

One of the central virtues of space exploration is that the depth of thought and level of engineering sophistication required to allow humans to live and work in space has pushed all of the fields involved forward. Like outer space, the modern online classroom is also an extreme environment which requires new strategies and technologies to address what happens when challenges that have been solved in the classroom must be re-purposed for teaching large groups of students online. These challenges present themselves most prominently in Massive Open Online Courses (MOOCs) where unlimited numbers of students can enroll in classes that are often offered for free.

When you are teaching thousands of students from all over the world, how do you do provide them with meaningful feedback for the assignments they complete for the class, let alone fair grades? If you’re teaching computational skills, how can you ensure that all of your students can set up a computing environment that will allow them to achieve the goals of the course? Broadly speaking, it appears that building technologies to help students connect to each other as individuals, and connecting them to a larger community of learning can help solve these problems. Aggregating lessons learned by previous students, and using student feedback in order to anticipate a student’s error can help to steer students towards an understanding of material more efficiently than what might be able to be achieved in a regular classroom.

Despite the fact that the hype around MOOCs reached a fever pitch several years ago, we are finally emerging from the trough of disillusionment around MOOCs. As a result, the tools that have been developed to help assuage the growing pains experienced in MOOCs are finding applications outside of MOOCs themselves.

What Does Scaled Education Really Mean?

Imagine the most intimate and nurturing educational experience you have ever had: maybe an elementary school teacher who picked you as their favorite student, or a college professor whose office hours you would attend religiously. Now imagine if you had that level of personal attention for every course you were ever enrolled in at every stage of your education. I believe this is the ideal we should be striving for when we think about what free, accessible, scaled education could become. Currently this dream is unrealistic because there aren’t enough people with enough hours in the day who know specifically how to help you learn. However, computers may be able to fill in some of those gaps, and they might help teachers be more effective in delivering ideas and knowledge.

Consider the miracle it is today that if you want to learn some skill, for example, how to do algebra, there are many thousands of tutorials, videos, books, and guides that are only a few clicks away from anybody with an internet connection. Scaling the availability of information is easy online, however those learning materials know nothing about your prior knowledge or your goals (unlike a nurturing mentor).

The spectrum between a caring mentor and a YouTube tutorial can be thought to represent to two extremes in education: hard to deliver and very personal, versus easy to deliver and very impersonal. Ideally “Learning at Scale” means maximizing the best parts of these two paradigms by using technology to make personalized learning experiences easy to deliver. Currently these kinds of technologies take on many forms, including personal cognitive tutors, educational recommendation systems, and platforms for facilitating face-to-face interactions between students and instructors. In order to continue creating more immersive experiences for students, we need to empower educators with tools that can help them develop learning materials beyond watching videos and assessments that are more pertinent than multiple choice questions.

Additionally, we need to build technology which is able to adapt to who our students are: with sensitivity to both what their goals are, and how they want their education delivered to them. Currently, if a student enrolls in an online course and then doesn’t complete the course they’re measured as a “dropout.” This method for evaluating whether or not a course is successful is completely antiquated. Students come to an online course with diverse goals: some of them are looking for one piece of information contained in the course, others might just be looking to refresh knowledge they learned in the past. We need to design online learning environments that can serve all of these needs, and we need to measure the efficacy with which we help students accomplish their differing goals. There is also a disconnect in that most of these learning technologies are being developed for desktop computing environments, despite the fact that people around the world are spending an increasing amount of time on mobile devices. The modes for interacting with a mobile device are much different compared to how we interact with desktop computers, and currently it doesn’t appear that current research is capturing these interactions.

The Cathedral, the Bazaar, and the Schoolhouse

In a traditional classroom, dozens of students put an immense amount of work into their classes. They create artifacts in the forms of notes, flashcards, and review guides. They comb through lecture notes and slide decks provided by instructors and annotate them according to what enclosed information they think is most relevant to their success. Over a semester, they not only master the material in the course, but they master the skills required to take the course itself. Unfortunately at the end of the course nearly all of these artifacts and insights are discarded, or at best they’re filed away for future, personal reference. In college courses all of the emotions, opinions, and labors of students outside of their assessments is submitted through a “course evaluation,” a survey that many students don’t have time to submit. These evaluations are very low-fidelity, and they fail to provide a transparent process for how student feedback influences future iterations of a course. One major promise of technologically enabled education is the potential for students to contribute more of their work to the improvement of a course. This way students can use materials that have been developed over time by other students, they can see pathways to understanding that students in the past have taken, and they can propose changes to a course if they believe they should be made. In my course The Unix Workbench, students are actively encouraged to point out errors in materials and to contribute to the course, however they are also taught highly technical skills for contributing to open source software using Git and GitHub, and they use those skills in order to contribute to the course. Expecting students in all kinds of disciplines to to use such technical software is not practical, so we will need to develop software that allows students to easily contribute new content to a course, propose changes to existing content, and to understand their own path through a course so that they can choose whether or not they want to share that path with others.


Thank you to my advisor Philip Guo for your advice, encouragement, editing prowess, and laudable LaTeX skills! Also thank you to the UC San Diego Design Lab for all of your incredible pre-submission advice and feedback.