A course and a coach, side by side
Earlier this year we wrote about coaching versus courses, looking at the research on digital business training. The pattern in those studies was that support personalized to the business and sustained over time tends to work better compared to digital courses that deliver the same content to everyone on a fixed schedule. That post was about other people's studies. This year we had the chance to watch both formats side by side, with the same people.
The setup
In one of the countries where MAIA operates, a partner organization had us build two things for the small business owners in its network. The first was a set of short courses on a standard online course platform, on topics the partner chose (eg saving and credit, using technology in the business). We adapted the partner's training material into short, mobile-friendly lessons with videos and reflection questions, and the partner's team reviewed every course before launch. The second was MAIA, set up for this group, with the partner's course material in its knowledge base.
Both were launched at the same in-person event, where staff helped participants sign up for the courses and start a conversation with MAIA. Our team then monitored the course platform using its data exports, and we track MAIA's usage the same way we do in every country. The numbers below compare the two groups from launch to the same cut-off date, several months later.
The course
Almost all of the course enrollments happened within three weeks of the launch event, and most students were enrolled in every course at once. The content itself held up: 39% of those who started a course finished it, and the lesson-by-lesson data showed no point where large numbers of people quit.
What the courses could not do was keep people coming back once the launch event was over. In the first full month after launch, 16% of students completed at least one lesson. In the second month it was under 4%, and in the most recent month it was 0.2%. Most people never came back even once: 93% of course enrollments never opened the first lesson, so for most participants, signing up at the event was the only contact they had with the course. This is the same pattern as the digital courses in the studies we reviewed earlier: good content, offered once, with nothing to bring people back to it.
MAIA
The MAIA group is everyone who started a conversation with MAIA in the same launch window. MAIA lost most of them after the event too: 57% sent at least one message in the first full month and 17% in the second, which is a steep fall and one we are working on (more below).
The difference is in who stayed. In the three months before the cut-off, 19% of the MAIA group were still talking to MAIA, compared with 1.8% of course students who completed any lesson in the same three months. A MAIA user was about ten times as likely to still be engaged.
The most likely reason is what MAIA does between conversations. A course waits to be opened, and so does a general-purpose AI chatbot: it is there when you think to use it, and it never comes looking for you. MAIA reaches out. It checks in the day after a conversation to ask how the plan went, sends a weekly message built on what that person has told it about their business, and follows up on the specific step the owner said they would take, on the day they said they would take it. Each of those messages gives the person a reason to come back that is about their own business, and coming back costs nothing more than answering a message in a chat they already open every day.
Sending people to the course through MAIA
About six weeks after launch, as part of the program's follow-up plan, we used MAIA to send a message with the course links to the program's MAIA users. If being on WhatsApp were what mattered, this should have helped: the same people, on the channel they were already using with MAIA, with a direct link to free courses built for them.
It barely registered. Of the course enrollments that had never been started at the two-month mark, 0.5% were started over the following three months. Monthly course activity kept falling.
So reaching out on WhatsApp was not enough by itself. That message was the kind of nudge the studies we reviewed found does little: the same reminder to everyone, pointing to the same content, with nothing in it about the person's own business, and it sent them away to another site, another sign-in, another app. When a partner brings us course content, our approach is to load it into MAIA's knowledge base so MAIA can bring in the relevant piece inside the conversation, adapted to the person's business, at the moment it is relevant to them.
Caveats
This was not an experiment. Nobody was randomly assigned, and the two groups overlap but are not identical: the course group is everyone who was enrolled, and the MAIA group is everyone who started a conversation. "Active" also means something different on each side. For MAIA it means sending at least one message, which is a lower bar than completing a lesson. And MAIA messages people first, with a weekly coaching touch, while a course platform waits to be opened. That difference is part of what we are comparing, since a course has no easy way to check in on someone, but it is worth stating.
Staying engaged is also only a means to an end. As we wrote in engagement versus impact, what we care about is whether businesses improve, and in our data the users most likely to report higher sales are the ones who come back in at least four separate weeks of their first two months. Some minimum level of engagement is necessary for any impact, but it does not guarantee it.
What we are doing about it
Engagement with MAIA at 10x a course is encouraging, but 19% is still lower than we want. MAIA's steepest losses happen in the first weeks, the same as the course's. We recently changed how MAIA works with new users in their first three weeks: an unanswered check-in no longer delays the next weekly message, and new users get a second message each week during that period. We will report on how it does.