How Hands-On Assessments Can Boost Retention, Satisfaction, Skill Development, and Career Outcomes in Online Courses

1.

Introduction

Limari, a mother in Puerto Rico, wanted a boost to her career while retaining the flexibility to be an active parent to her son [1]. She began working as a community manager from home, but it was only after her friend’s suggestion to join Coursera that she had the skills and confidence start as a content strategy manager. Northwestern University’s professional strategy massive open online courses (MOOCs) provided a concrete path for her to advance her career.

In Bengaluru, India, Vivek needed a change, and Coursera offered him a lifeline [2]. After his mother died at age four and his alcoholic father abandoned him, Vivek was not able to finish his formal education but had always wanted to start his own company. By completing programming, machine learning, and algorithm courses on Coursera, he was able to start his own business and hire other self-taught experts. He continues to advocate for use of online courses and carves out time for himself and his employees to continue taking MOOCs to augment their skills.

Paulina, a medical student in Ghana, knew software and data science tools could help her family and her patients [3]. By completing more than a dozen courses on Coursera, Paulina is solving community health challenges through data extraction, geolocation, and clever analysis. As the daughter of farmers, Paulina loves her medical work but also wants to apply her newly gained data science skills to food production and distribution techniques. Paulina is a prime example of how individuals can use what they learn from online content to improve their broader community in a variety of ways.

Limari, Vivek, and Paulina are just three of the millions of learners who have engaged with online courses since 2012. Over the past decade, MOOCs have enabled anyone with internet access to take courses offered by top universities and companies from anywhere in the world. These open-access, largely video-based coursework have played an important role in democratizing higher education, providing faculty-created material at a low price point, and connecting diverse groups of individuals from around the globe [4]. The first MOOC was offered by a university in 2007 and just five years later, there were several companies founded with the goal of providing MOOCs to the world, including Coursera, edX, and Udacity [5]. By 2012, more than one million learners had engaged with a MOOC on one of these platforms.

The context of this research was the Coursera platform, one of the largest higher education platforms in the world. Since its founding in 2012, Coursera has been focused on its mission of providing the world’s best learning experience to anyone anywhere. To create the content it offers, Coursera partners with leading universities and companies to leverage their knowledge, skills, and expertise. By the end of 2020, more than 70 million learners from around the world had come to learn skills of the future in 4,000+ MOOCs offered on Coursera [6].

These MOOC learners are a much broader and more diverse group than traditional, on-campus higher education students. The majority of learners in Coursera’s community are 24 to 45 years old [7]. More than 60% of learners who complete at least one course report career impact as their top goal for enrolling, whether they desire getting a new role, changing industries, gaining confidence in their current job, or starting their own company [6]. Representing a truly global community, learners on the Coursera platform enroll from more than 200 countries worldwide [8].

Students enrolling in the fully online degree programs hosted on Coursera are similarly global and diverse. These degree students span the ages 18 to 65 and represent more than 90 different countries [9]. Plus, more than half of the degree students report working full-time while simultaneously participating in these for-credit learning experiences.

While many of these first MOOCs started with only quizzes, Coursera and others quickly added programming assignments followed by peer review assessments to augment the simple exam-style assessments with more hands-on opportunities to demonstrate their skills [6]. In the for-credit courses, Coursera also offers opportunities for tailored feedback from experts via staff-graded individual and team assignments [9].

On the Coursera platform, there are a variety of hands-on activities that can be included. For example, in a web programming course offered by the University of London Goldsmiths, the course team created a “Sleuth” detective game (see figure 1) for students to solve coding challenges [10]. They used the Coursera Labs infrastructure to design and host this series of programming activities and included custom auto-grading to determine students’ score for each part of the assignment.

As a second example, the Google course team created a computer assembly activity to simulate hardware (see figure 2), where learners have put together the pieces of a computer not only in the right place but also in the right order [11]. By using the JavaScript plugin item type on Coursera, the course staff at Google created a hands-on activity for their IT Professional Certificate that had previously only been possible with a physical computer and completed in real life.

As a third and final example, the course team at Michigan State University created hands-on activities for their photography online courses [12]. While hands-on assessments are often associated with science and technology courses, arts and humanities can equally benefit from having learners test out the new skills they have gained earlier in the course (see figure 3). Peter Glendinning and team created a peer review assignment where students first submit their own photographs with accompanying explanations before receiving feedback from their peers. Importantly, the instructor made the grading rubric to help scaffold and guide learners when giving grades and recommendations to their peers.

Crucially, online hands-on assessments provide opportunities for large groups of students to practice new skills and gain valuable feedback. Hands-on assessments involve the active practicing of skills and offering feedback relevant to the task [13]. These learning activities are in contrast with passively watching a lecture or simple quizzes [14]. Another crucial aspect of hands-on assessments is providing tailored feedback, in the form of either automated or expert guidance [13, 14]. These hands-on activities provide the training students need to meet employers’ expectations [15].

Application-based assessments align more closely with real-world work and better prepare students for industry [16]. Industry-relevant projects were better liked by students, especially by those with less traditional backgrounds or less higher education experience [16]. With changing jobs, industries, and career skills needed, online assessments should reflect this dynamic landscape [17]. While often harder to grade, creative assessments with real-world applications lead to greater student innovation across the learning experience [16, 17].

Moving beyond multiple-choice exams into papers, presentations, and group projects better mirrors industry tasks and can not only increase success of students with less higher education experience but also increase all graduates' employability [18]. Alternative assessments can also help build confidence for some of the most vulnerable groups of students in terms of dropping out [16, 18]. De-emphasizing exams can lead to more equitable outcomes across student populations, and lowering stress can increase students’ ultimate learning gains [19]. Broader and varied assessments can also increase transfer to on-the-job tasks after graduating [18].

Since the start of the COVID-19 pandemic, educators have had a greater need to move these hands-on activities online, through the use of simulated laboratories, projects, and virtual environments [15, 20, 21]. However, many instructors have found it difficult to transfer in-person hands-on assessments into successful online activities and found online platforms lacking the capabilities they desired [22, 23]. Flexibility and autonomy given by alternative assessments are especially important with the diversity of adult learners now re-skilling through online courses—this adaptability is crucial and can lead to higher academic achievement across the board [24].

Hands-on assessments are well-documented to improve student engagement and satisfaction, especially in more technical courses. Students self-report significant preference to laboratory activities and other hands-on tasks as compared with other science course teaching methods, whether learning veterinary dental procedures or designing control systems for robotic devices [13, 14]. These students also reported the hands-on assessments as highly valuable for their learning [13, 14]. In a different training, students in an undergraduate medical course were highly dissatisfied with the shift to online because of the COVID-19 pandemic, linked to the sudden reduction in hands-on assessments and tailored feedback [23]. When connected to the learning objectives of the course, hands-on assessments can provide useful scaffolding and motivation to keep students engaged [25]. Alternative assessment and hands-on experiences appeared to be associated with higher learner readiness, engagement, and motivation in online instruction during the COVID-19 pandemic [24]. This increased engagement can be a positive first step toward stronger learning outcomes [20].

Hands-on assessments can increase skill development for learners. Through observation in MOOCs, researchers rated creativity skills and reported significant increases after project-based learning activities were implemented in the course and participants had to brainstorm, problem solve, and discuss their ideas with peers [26]. Instructors can also include hands-on activities that involve team projects and discussion to encourage collaboration and students’ sense of community in the course, having students work together to create a final deliverable [20].

This article explores research conducted on Coursera, a company specializing in online education that partners with more than 250 universities and companies globally to offer MOOCs and full degree programs to its 100 million registered learners. By analyzing course structures and learners’ behavior, team members at Coursera can offer data-backed insights to empower educators.

2.

Methods

At one of the largest online learning platforms in the world, Coursera team members have conducted robust research to analyze what works best for teaching and learning online. This work drew on Coursera’s large number of learners and immense community of partners. The MOOC research looked across 200 million course enrollments from 9 years of data. The for-credit research drew on 17,960 unique students across 19 degree programs offered on Coursera, including Bachelor’s and Master’s programs from diverse domains, including Computer Science, Business, Data Science, and Public Health [9].

Specific outcome metrics related to quality also needed to be operationalized. For retention, the rate of course completion was used for MOOCs, and the rate of first-year persistence (i.e. percentage of students still active in the degree one year later) was used for analyzing the first courses in degree programs. For an indicator of satisfaction, course completers' percentage of 5-star ratings (out of a maximum five stars) was used. For skill development, Coursera’s own “skill score” was used [27]. To calculate the “skill score,” the difference was calculated for each course completer, comparing their documented skill level upon beginning and then completing the course in question. This change in score, or delta, was then averaged across all course completers of that MOOC. For career outcomes, responses from an in-platform survey automatically sent to learners six months after they completed the course were used to assess impact on individuals’ job, role, and pay. The proportion of course completers from each MOOC reporting some type of career benefit was used as this outcome variable.

Throughout both analyses, researchers at Coursera wanted to investigate the relationship between different learning design choices and the outcome metrics discussed above. To accomplish this goal, the relationship between each driver of interest and the outcome metric was assessed. To standardize the strength of each relationship, regression models were used to isolate the impact of each potential driver on the outcome metrics related to quality, including retention, satisfaction, and skill development. Throughout the analyses, the researchers controlled for learner demographic information, including age, prior educational attainment, and previous learning behavior on the Coursera platform, all of which can influence learners’ likelihood to succeed in online courses. They also controlled for course-level information, including the university brand and topic being taught, both of which can influence degree retention in a particular program. While the regression models provided accurate summaries of the relationship between each driver and outcome of interest, it became challenging to assess the relative effect of each one, since each driver has a different scale of observable measures.

The drivers explored included variables relating to course length, course composition, teaching decisions, and item specifics. For this article, the author has focused on the assessment drivers, particularly those related to hands-on learning, which was a subset of these initial data analysis projects for MOOCs and degrees. Thus, the drivers discussed here include staff-graded assignments, peer review assignments, programming assignments, and hands-on exercises (which includes staff, peer, and programming opportunities together).

This variety in drivers created challenges for attempts to standardize the impact of each and compare them. While course length ranges from three weeks to sixteen weeks, the length of a single video varies from 3 min to 50 min. To account for each driver operating on such a different scale, a unique design was used to normalize the effect across both MOOC and degree analyses. Specifically, the estimated difference was calculated for the outcome metric between the 25th percentile and the 75th percentile for each driver of interest [3, 6]. For example, the 25th percentile MOOC engages its median student on only 5 different days while the 75th percentile MOOC engages its median students on 12 distinct days. The difference between these two, the 25th and 75th percentiles, can be reported on a single display for each outcome metric of interest, be it retention or skill development. In addition, to examine the effect that the presence of a certain assignment type has on retention or satisfaction, a binary system was used, meaning either zero or one, where zero indicated all the courses without that assignment type presence and a one represented all the courses with at least one of those assignments. By comparing the satisfaction at zero, meaning none of those assignments present, and at one, this method provides an estimate of the impact for including that assignment type in the learning experience. Moving from the 25th to 75th percentile measures the standardized scale of each driver in percentile differences instead of the absolute values.

This method provided the additional benefit of moving only within the space of what had already been observed within the sample data, instead of extrapolating beyond the observed sample. A consequence of the chosen technique is that the driver effects can be reasonably changed, given they are occurring in existing online course designs already live on the Coursera platform. By restraining these findings within the observable data, there is sufficient evidence that these changes are both fruitful and possible.

3.

Results

Hands-on assessments emerged as a significant driver across outcome metrics of interest. In both open-access MOOCs as well as closed degree courses, the inclusion of hands-on activities drove increases in retention, satisfaction, skill development, and career outcomes.

4.

Discussion

Overall, hands-on projects can provide authentic projects mimicking industry use cases of the skills and adds purpose to the new knowledge learners have gained. The appropriate use of peer review, programming, and staff-graded assignments each enable learners to demonstrate the skills from the course through application-based projects. These hands-on opportunities show them the benefits of applying these skills to real-world scenarios and help to make the link between academic learning and on-the-job tasks they will need in their career. Thus, hands-on assessments can increase individuals’ persistence through the material, satisfaction with the course, ultimate skill development from the learning experience, and eventual impact on their careers.

Acknowledgments

Appreciation and thanks to Alan Hickey, Eric Karsten, and Laura Wandres at Coursera for making this article possible. Many thanks to Dr. Sharon Mistretta whose leadership made this entire edition possible.

Conflict of interest

The author declares no conflict of interest.