Usage Patterns Update!

Now that we have over a thousand users, 4 months of open access, and some new ideas on metrics thanks to our friends at the Agency Fund, there are a few things that we can say about engagement so far.

Overall from all those who connect to MAIA about 25%-30% continue to use it month-on-month over the long term. We lose a lot of people at the beginning for one good reason and one bad reason. The good reason is that in our recruitment campaigns (more below) the promoters can often connect someone who doesn't actually have a business (eg an employee), and we also have lots of people from potential distribution partners, funders, researchers etc who are just curious and connect to MAIA to try it out. So non MSME owners. The bad reason is that too many MSME owners connect but don't know how to use it, maybe they ask a google-type question (eg what's the current price of X) and get disappointed with the results, or a very general question (eg how do I sell more) and don't get into a more detailed conversation, which is when the value becomes more apparent.

This is the group we are focusing on: how can we make it easier for MSME owners to get started with generative AI. So evolving from a text-heavy open-ended onboarding message (ask me anything!) to showing a couple more structured and specific use cases,  without losing the ability for entrepreneurs with other priorities to choose their own adventure, is the direction we're pushing on.

Another striking thing are the patterns behind this retention curve, in particular how different it looks depending on how users were recruited, and what pre-filters that implies.

Most of the users in the chart above came through recruitment campaigns. While some are through distribution partners, many of these came from our own, where we sent out promoters door-to-door in a few neighborhoods to tell MSME owners about MAIA. 

This gets closer to a representative sample of MSMEs, as its pitched to literally every store and restaurant on the block, and the majority of those do initially connect. But before doing these campaigns, we first tested the tool out on smaller groups of entrepreneurs. Our first group before our public launch was bodega owners who participated in a previous study we ran, who were recruited through the national bodega association. So these were active members of an association, and passed typcial attention filters for our research study. Unsurprisingly, their retention curve looks really different.

The initial fall-off from this group was smaller and leveled off at 45% monthly active users. And note they started in the first days of MAIA, when we didn't have the functionality described in this blog post and it was more of simple 'chat with the training manual' tool. As those features were rolled out, you can see that the MAU numbers for this group rose to 60%+.

So two key take-aways:

1) Recruitment & pre-filters matter a lot. Many entrepreneurial training programs and impact evaluations heavily pre-filter, to minimize study attrition and focus expensive training on those who most want it. This all makes lots of sense, but that is a pretty different group from your average MSME, and if you want to build a product for the hundreds of millions of typical MSMEs out there you need to get at more representative users and be worried about external validity.

2) Features matter a lot. Its not shown here because of small #s, but we can see what usage looks like for a simple chatbot with no additional functionality on average MSMEs because Meta has this 'experiement' where some randomly selected numbers can't receive any messages from WhatsApp business accounts unless they are a reply to a user, so none of this functionality reaches those entrepreneurs. The retention curve for this group seems to drop significantly and level off below 10%. Making a chat-based business adviser useful for the majority requires accumulating a bunch of small improvements in user experience.