What does early engagement look like?

In the first class of my first semester as an undergraduate student, Professor Larry Boland sat in front of the class and just said ‘questions?’ over and over. After a few sessions we got used to his unique Socratic method, and most ended up enjoying the class. But it was an extremely awkward way to get started, particularly for a room full of new students.

An LLM is a lot like that- its an open text box with a blinking cursor, awkwardly waiting for questions. It has access to a mind-boggling amount of knowledge, but its up to you to come to it and ask the right question in the right way. Its not obvious how to make this powerful tool useful.

When you give a microenterprise owner in a developing country access to an LLM for the first time, they ask a few questions to test its knowledge and are impressed with the results, but similarly its not clear to them how to make it useful over time. What we’ve seen is that unaided, their pattern of usage looks very much like most people when they first try ChatGPT- they ask a few questions to test it out and are amazed with the results, but then move on with other priorities. There are some power users that will come back to it again on their own, but that is the minority, and you can see similar patterns in other LLM studies like this one for entrepreneurs in Kenya and this one for teachers in Sierra Leone, even with reminders and incentives.

Many traditional entrepreneurial training programs are happy to find that minority of power users and focus on them, that is why selection is a core part of how they work. But, to empower the majority of MSMEs, you need to make it useful for them, not just sit back and wait for questions. Our initial launch and focus group testing showed that they don’t want a guru sitting back and waiting for questions: after helping with their first set of priorities, they want a tool that proactively offers advice and ideas that they can engage with and interrogate when they want, and are free to ignore if they are busy.

So that’s the approach we have taken so far. Every week the tool sends out a ‘tip of the week’, as well as a second message that is a rotating mix of system tips, poll questions, or special guided exercises like long-term goal setting or identifying ‘quick wins’.

What we’ve observed so far with this approach is that for every 10 MSME owners who get access to MYPE Asesor IA, there is 1 power user who uses the tool proactively and engages with almost every message, there is 1 inactive user who maybe reads the message but never really engages with the tool, and the remaining 8 engage about half the time. Messages from the system have about a 50% response rate, but its not the same 50% of users week to week.

Last week we put out a poll to see what users thought of this approach, asking if they wanted more, the same, or fewer messages from the system. Here are the results:

So no one finds the system-initiated messages annoying enough to click the ‘I want fewer messages’ response in the poll. And among the half that did respond, they either like the current approach or want more. Eventually the system should adjust to those preferences- there isn’t any reason why everyone should get the same number of messages, or even the same tips of the week.  The benefit of AI-based training is that the tool should adjust its approach for each small business owner. But its clear that the next experiment for A/B testing will be in the direction of more, rather than fewer, messages from the system.