Mastery-Gated Levels, Not Time-Based Points or Plain Lessons: What an AI Coach Rewards and What Learners Can Still Do Three Months Later

Abstract

In a 10-week workplace course on spreadsheet analysis taught by an AI tutor, 348 employees from seven organizations were randomly assigned to one of three reward designs. In "mastery-gated" accounts (n=117), points and new levels came only from passing short applied checks set by the tutor, with unlimited retries and a hint after each failed attempt. In "time-points" accounts (n=116), points came from minutes practiced and lessons completed, and levels from accumulated points. "Plain" accounts (n=115) had the same lessons with no points or levels. Time-points learners logged the most practice, a mean 142 minutes a week, against 118 for mastery-gated and 112 for plain learners, and 71% of them finished the course within 10 weeks, against 58% of mastery-gated learners and 61% of plain learners. Three months after the course ended, 331 learners took an applied task scored by two raters blind to design; mastery-gated learners scored 64.2/100, against 52.8 for time-points learners (a 22% gap) and 55.1 for plain learners (a 17% gap). Mastery-gated learners rated the course as more frustrating, 3.6/7 against 2.9/7 for time-points learners (a 24% gap) and 3.1/7 for plain learners (a 16% gap). Mastery-gated learners who did not finish within 10 weeks still scored 60.1, above the time-points average.

1. Introduction

Mastery-gated learners did worse than learners earning points for time on the measures that course operators usually report, and we should say so before anything else. They practiced fewer minutes than learners earning points for time, finished the course less often, and found it more frustrating. The design we expected to do best, and which did best on the one outcome a learner's colleagues would notice, looks worse on an engagement dashboard.

The question behind the comparison is what the points in a game layer pay for. An AI tutor can award points for almost anything it can measure, and minutes on task and lessons opened are the easiest to measure. Points for time reward the hours a learner puts in, whether or not the skill improves. Points for passing an applied check reward something closer to the skill the course exists to teach, at the price of holding learners at a level until they can show it. We compared both against plain lessons, and measured what learners could still do three months later.

2. Method (Summary)

The course taught spreadsheet analysis (lookups, summary tables, basic charts, and checking a workbook for errors) over 10 weeks, delivered by an AI tutor that explained each topic, set exercises, and answered questions. Employees of seven organizations who enrolled were randomly assigned to one of three designs, stratified by organization.

In mastery-gated accounts (n=117), each of the course's 12 levels ended with a short applied check, such as building a summary table from a supplied data set, which the tutor scored against a fixed answer. Points were awarded only for passed checks, and the next level opened only after a pass. A failed attempt produced a hint and a fresh version of the check, with no limit on retries and no loss of points. In time-points accounts (n=116), points accrued for each minute of active practice and each lesson completed, and levels opened at fixed point totals; the same checks appeared as ordinary exercises. Plain accounts (n=115) had the same lessons and checks with no points or levels. All three designs used the same tutor and the same material.

Practice minutes and course completion were drawn from the tutor's logs, with a minute counted as active when the learner had entered input in the preceding two minutes. Completion meant finishing the course's final lesson within 10 weeks. In mastery-gated accounts, reaching that lesson required passing every level's check; in the other two designs it did not. Three months after the course ended, learners completed a 45-minute applied task, a workbook analysis of the kind the course taught, in a supervised session without the tutor. Two raters, blind to design, scored each submission from 0 to 100 against a written rubric, and their scores were averaged. At week 10, 329 learners rated how frustrating they had found the course (1 = not at all, 7 = extremely) in a survey.

3. Findings

Of the 348 learners, 331 completed the three-month task (111 mastery-gated, 110 time-points, and 110 plain learners), and the scores below are for those learners.

Time-points learners logged a mean 142 active practice minutes a week, 20% more than mastery-gated learners (118) and 27% more than plain learners (112). Within 10 weeks, 82 of 116 time-points learners (71%) finished the course, against 68 of 117 mastery-gated learners (58%) and 70 of 115 plain learners (61%).

On the three-month applied task, mastery-gated learners scored a mean 64.2/100, against 52.8 for time-points learners (a 22% gap) and 55.1 for plain learners (a 17% gap). Time-points and plain learners were close to each other. The gain was not confined to learners who passed every check. Mastery-gated learners who did not finish within 10 weeks (45 took the task) scored 60.1, above the time-points average, against 47.2 for time-points non-finishers (31) and 51.5 for plain non-finishers (42). Among finishers, the scores were 67.0 for mastery-gated (66), 55.0 for time-points (79), and 57.3 for plain learners (68), though in the mastery-gated design finishing itself required passing every check.

Mastery-gated learners failed a median of six check attempts over the course. They rated the course 3.6/7 for frustration, against 2.9/7 for time-points learners (a 24% gap) and 3.1/7 for plain learners (a 16% gap).

4. Discussion

Before claiming anything for the mastery-gated design, we note what it cost the learners in it: more frustration, a slower route through the course, and a lower chance of finishing it on schedule. A failed check with a hint is a gentler obstacle than a failed test, but it is still an obstacle, and it came up a median of six times per learner. If a program judged the three designs on its usual figures, minutes and completions, it would choose time-points.

What mastery-gated learners received for that cost was skill they could still use three months later, without the tutor present. Points for time produced the most practice and the highest completion rate and no better retained skill than plain lessons, which suggests that the additional minutes were spent on activity that earned points without building much skill. The retry design gave learners as many attempts as they needed, and mastery-gated learners who were still held at a level when the course ended scored above time-points learners overall, so the design did not appear to leave behind the learners it slowed down. A design that rewards demonstrated skill asks more of learners in the short term, and the people who rely on their spreadsheets later are among those who benefit.

5. Limitations and Future Directions

Learners knew their design, and those in mastery-gated accounts were shown the applied checks as the route to points, which may have made them better prepared for an applied final task than learners who saw the checks as ordinary exercises. The three-month task was a single 45-minute session and did not measure whether learners used the skill at work. Seventeen learners did not take the task, and their outcomes are unknown. Because finishing meant something different in each design, the finishers-only comparison favors the mastery-gated design by construction, and we give more weight to the overall and non-finisher scores. The active-minute definition may have counted minutes of input that were not practice, such as typing questions to the tutor. All seven organizations paid for staff time in the course, so the effect may differ where learners study in their own time and frustration more easily leads them to stop. A follow-up will collect managers' reports of whether learners used the skill in their work.

Editorial Note

This report underwent CfAHC's Rapid Review Protocol, including AI-assisted literature cross-checking and a two-reviewer human sign-off.