Blog

Now that we have over a thousand users, 4 months of open access, and some new ideas on metrics thanks to our friends at the Agency Fund, there are a few things that we can say about engagement so far.<figure id="figure-g7l9ypi6w8m4e8dema" class=""><img id="g7l9ypi6w8m4e8dema" src="https://cdn.durable.co/blocks/12CiNG0XR5XSSHgT6lyWZyiQwdtgMzX8ufyBA05EyQACySyEGNaPxyG5u5tzNGlT.jpg" alt="" title="" class="object-cover bg-gray-100 " style="aspect-ratio: 16/9; object-position: undefined undefined;" data-corners="no-rounded" data-aspect="16:9" data-caption="MAU Periods 0 - 5" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/12CiNG0XR5XSSHgT6lyWZyiQwdtgMzX8ufyBA05EyQACySyEGNaPxyG5u5tzNGlT.jpg" data-mce-style="aspect-ratio: 16/9; object-position: undefined undefined;"><figcaption class="body-normal text-gray-500 mt-2">MAU Periods 0 - 5</figcaption></figure>Overall from all those who connect to MAIA about 25%-30% continue to use it month-on-month over the long term. We lose a lot of people at the beginning for one good reason and one bad reason. The good reason is that in our recruitment campaigns (more below) the promoters can often connect someone who doesn't actually have a business (eg an employee), and we also have lots of people from potential distribution partners, funders, researchers etc who are just curious and connect to MAIA to try it out. So non MSME owners. The bad reason is that too many MSME owners connect but don't know how to use it, maybe they ask a google-type question (eg what's the current price of X) and get disappointed with the results, or a very general question (eg how do I sell more) and don't get into a more detailed conversation, which is when the value becomes more apparent. This is the group we are focusing on: how can we make it easier for MSME owners to get started with generative AI. So evolving from a text-heavy open-ended onboarding message (ask me anything!) to showing a couple more structured and specific use cases,&nbsp; without losing the ability for entrepreneurs with other priorities to choose their own adventure, is the direction we're pushing on.Another striking thing are the patterns behind this retention curve, in particular how different it looks depending on how users were recruited, and what pre-filters that implies.Most of the users in the chart above came through recruitment campaigns. While some are through distribution partners, many of these came from our own, where we sent out promoters door-to-door in a few neighborhoods to tell MSME owners about MAIA.&nbsp;<figure id="figure-rl060g199nm4e93zml" class=""><img id="rl060g199nm4e93zml" src="https://cdn.durable.co/blocks/3xhfETXIWqUyQWHvfG33hPXyrTtxWNHUvtbD7R4RDW3LpBubHMCHot9IF2UNGnAj.jpg" alt="" title="" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="null" data-positiony="null" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/3xhfETXIWqUyQWHvfG33hPXyrTtxWNHUvtbD7R4RDW3LpBubHMCHot9IF2UNGnAj.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure>This gets closer to a representative sample of MSMEs, as its pitched to literally every store and restaurant on the block, and the majority of those do initially connect. But before doing these campaigns, we first tested the tool out on smaller groups of entrepreneurs. Our first group before our public launch was bodega owners who participated in a previous study we ran, who were recruited through the national bodega association. So these were active members of an association, and passed typcial attention filters for our research study. Unsurprisingly, their retention curve looks really different.<figure id="figure-89rz9rsywm4e9g2hh" class=""><img id="89rz9rsywm4e9g2hh" src="https://cdn.durable.co/blocks/5gSNPCCYYALlU3l1puCKlJcHADXUNf0XKbnBBHJbomkNqlVJ4Wc1CM9XznBmAXkJ.jpg" alt="" title="" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="MAU months 0-5, 2024Q1 cohort" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/5gSNPCCYYALlU3l1puCKlJcHADXUNf0XKbnBBHJbomkNqlVJ4Wc1CM9XznBmAXkJ.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"><figcaption class="body-normal text-gray-500 mt-2">MAU months 0-5, 2024Q1 cohort</figcaption></figure>The initial fall-off from this group was smaller and leveled off at 45% monthly active users. And note they started in the first days of MAIA, when we didn't have the functionality <a data-mce-href="https://mypeasesoria.org/blog/beyond--chat-with-the-manual--ii" href="https://mypeasesoria.org/blog/beyond--chat-with-the-manual--ii" target="_blank" rel="noopener">described in this blog post</a> and it was more of simple 'chat with the training manual' tool. As those features were rolled out, you can see that the MAU numbers for this group rose to 60%+. So two key take-aways:1) Recruitment &amp; pre-filters matter a lot. Many entrepreneurial training programs and impact evaluations heavily pre-filter, to minimize study attrition and focus expensive training on those who most want it. This all makes lots of sense, but that is a pretty different group from your average MSME, and if you want to build a product for the hundreds of millions of typical MSMEs out there you need to get at more representative users and <a data-mce-href="https://mypeasesoria.org/blog/worry-about-external-validity" href="https://mypeasesoria.org/blog/worry-about-external-validity">be worried about external validity.</a>2) Features matter a lot. Its not shown here because of small #s, but we can see what usage looks like for a simple chatbot with no additional functionality on average MSMEs because Meta has this 'experiement' where some randomly selected numbers can't receive any messages from WhatsApp business accounts unless they are a reply to a user, so none of <a data-mce-href="https://mypeasesoria.org/blog/beyond--chat-with-the-manual--ii" href="https://mypeasesoria.org/blog/beyond--chat-with-the-manual--ii" target="_blank" rel="noopener">this functionality</a> reaches those entrepreneurs. The retention curve for this group seems to drop significantly and level off below 10%. Making a chat-based business adviser useful for the majority requires accumulating a bunch of small improvements in user experience.

Usage Patterns Update!

Here is an interesting video of a trainer (I believe related to the ILO SIYB program but could be wrong) using ChatGPT's voice mode with a small business owner in the Philippines (via Starlink no less).&nbsp;<figure id="figure-0czbddx29p3vm3ojhw2n" class=""><iframe id="0czbddx29p3vm3ojhw2n" src="https://www.youtube.com/embed/fZggTm95fYw" class="w-full bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9;" allowfullscreen="" data-url="https://www.youtube.com/watch?v=fZggTm95fYw" data-corners="default" data-fullwidth="false" data-mce-src="https://www.youtube.com/embed/fZggTm95fYw" data-mce-style="aspect-ratio: 16/9;"></iframe></figure>In our first round of experiments users liked the ability to send their questions as audio, but preferred written responses, particularly as the issues discussed became more detailed. But that was before the new voice mode. We plan to do another round of experiments with audio replies next year, but that will stay within the WhatsApp interface. Its great to see more experiments like this, to figure out how to make AI more accessible and usable for MSMEs in emerging markets!

Voice Mode in PHL

At its most basic level, an AI chatbot on WhatsApp is just an easier front end to ChatGPT, with ChatGPT instructed in a particular way and supplemented with particular documents to make it more relevant and reliable for a particular group (like Peruvian microenterprise owners). It will then sit there and wait for questions. But as discussed in a <a data-mce-href="https://mypeasesoria.org/blog/beyond--chat-with-the-training-manual" href="https://mypeasesoria.org/blog/beyond--chat-with-the-training-manual" target="_blank" rel="noopener">previous post,</a> this "chat with the manual" bot isn't very useful, and won't get much use.Why? Its placing all of the burden of the user to know what questions to ask, how to use AI, and when. Small business owners are incredibly busy, they have very limited time, and limited experience with technology. Making these tools actually useful and usable isn't their job, its our job.We're only at the beginning of our experiments to learn how to do this better, but so far have found a few really useful things:1) A gradual onboarding process. One week after they first connect with Maia, we send them a list of the 3 most common questions across all users in their partiuclar sector, so they get a sense of the breadth of topics they can explore. The next week they get a guide on how to ask better questions to get better answers (provide more context, follow-up questions, etc). Two weeks later, they get a message they can forward to fellow MSME owners to share Maia, if they found it useful.&nbsp;2) The 'tip of the week'. Every week, users get a different tip built from the best practice manuals in the knowledge base. This surfaces new ideas to explore that they may not have thought of. Users can then discuss those topics further if they are relevant (with quick reply buttons, which are a must!) and very quickly get to a specific actionable plan to implement a particular improvement that they design with the help of the tool.3) Follow-up messages. Its one thing to know that we should eat well and exercise, doing it is another matter. Small business owners are busy, and many things come up that may need immediate attention but distract from important but non-urgent things like improvements. If users explored a particular topic the week before and identified some improvements they wanted to make, the following week they will get a follow-up message, to see if they implemented them (and if so, how it went and what are the next steps), and if not, an opportunity to refelct on and possibly revise the plan.

Beyond 'chat with the manual' II

Here is a presentation I gave today to the ILO's global network of SIYB master trainers. SIYB is the largest small business training program in the world, and it was a very interesting discussion!<figure class=""><img id="yrydhlkd4m1iao1v2" src="https://cdn.durable.co/blocks/18p7ZA33fXSdObi6mYMgjF2u8NDsMHxaUqK7ydtr5oP4xhOpEVufFHWggd5MqopM.jpg" alt="right arrow sign on wall" title="right arrow sign on wall" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/18p7ZA33fXSdObi6mYMgjF2u8NDsMHxaUqK7ydtr5oP4xhOpEVufFHWggd5MqopM.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure><figure id="figure-837va9m65m1iap9ed"><img id="837va9m65m1iap9ed" src="https://cdn.durable.co/blocks/73OKRA7Xb3wjJBicLj5AcpOIb4zirmY5iS8rmZ4PLUZtDxAaS3n8yDvHao8Tn7cV.jpg" alt="black smartphone near person" title="black smartphone near person" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/73OKRA7Xb3wjJBicLj5AcpOIb4zirmY5iS8rmZ4PLUZtDxAaS3n8yDvHao8Tn7cV.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure><figure id="figure-qdudzyw9erm1iappdy" class=""><img id="qdudzyw9erm1iappdy" src="https://cdn.durable.co/blocks/1bT37W1TZexY9S9Voxtx1NL24wnxfc6GZLjXrxMd4nAhKQOtYuGCfjfQu9zXNahq.jpg" alt="person holding compass selective focus photography" title="person holding compass selective focus photography" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/1bT37W1TZexY9S9Voxtx1NL24wnxfc6GZLjXrxMd4nAhKQOtYuGCfjfQu9zXNahq.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure>(Box 1 not covered today, but there is a ton of exciting ways that trainers could use AI to help them with things like lesson plans, exercises, role-playing, etc. See https://www.youtube.com/watch?v=8FnOkxj0ZuA to get started on AI for educators. The best source of ideas are the experiments of your fellow master trainers)<figure id="figure-gu1p0bki49m1iaqogu" class=""><img id="gu1p0bki49m1iaqogu" src="https://cdn.durable.co/blocks/d3PhcEBAyEm1XSveK4KgZBiwuWqA0vFqa2HlsNIkaQx5rlMXGnKfhtXraIFj3hqc.jpg" alt="woman in teal t-shirt sitting beside woman in suit jacket" title="woman in teal t-shirt sitting beside woman in suit jacket" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/d3PhcEBAyEm1XSveK4KgZBiwuWqA0vFqa2HlsNIkaQx5rlMXGnKfhtXraIFj3hqc.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure>(Box 2 not covered today, but this is just a new category of tools which trainers have always shared with trainees. Some examples: instead of teaching branding and giving advice on how to make a logo, AI image generation tools can just apply those best practice and make your business a logo. Other examples: customer surveys.)<figure id="figure-qiz7wl136m1iarj74" class=""><img id="qiz7wl136m1iarj74" src="https://cdn.durable.co/blocks/38wRZFkZKQ85QzDYQBNHHadvCMo8rdD34mMBNywOaHs7jvP9fHyoH0MGts6cUwZu.jpg" alt="black pencil on white printerpaper" title="black pencil on white printerpaper" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/38wRZFkZKQ85QzDYQBNHHadvCMo8rdD34mMBNywOaHs7jvP9fHyoH0MGts6cUwZu.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure><figure id="figure-2e7xjng0ym1iarxvf" class=""><img id="2e7xjng0ym1iarxvf" src="https://cdn.durable.co/blocks/eeA8IPaqzHLccHPKlP6RmuLbRoKHqzyqBRfHvGe3BwjHUloNWK6rFy4m2UtRaEDe.jpg" alt="three people sitting in front of table laughing together" title="three people sitting in front of table laughing together" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/eeA8IPaqzHLccHPKlP6RmuLbRoKHqzyqBRfHvGe3BwjHUloNWK6rFy4m2UtRaEDe.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure><figure id="figure-mu77k4tbjm1iasdl1"><img id="mu77k4tbjm1iasdl1" src="https://cdn.durable.co/blocks/22e52uphZ3zQpgjmKvHhqZClRaSwmbuWGOTPTuH0ut7yAhRP2XOAACTBEv1FfVl0.jpg" alt="man facing green mountains during daytime" title="man facing green mountains during daytime" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/22e52uphZ3zQpgjmKvHhqZClRaSwmbuWGOTPTuH0ut7yAhRP2XOAACTBEv1FfVl0.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure><figure id="figure-g2vr77hpum1iat2s8"><img id="g2vr77hpum1iat2s8" src="https://cdn.durable.co/blocks/5khVyfgshiCGfOCaj4M9Nx9rc1kPWeox3HnJ5q11C71BoWcspThawDoN9EkuWr0z.jpg" alt="two people sitting during day" title="two people sitting during day" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/5khVyfgshiCGfOCaj4M9Nx9rc1kPWeox3HnJ5q11C71BoWcspThawDoN9EkuWr0z.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure><figure id="figure-evwkjod23nm1iatexe" class=""><img id="evwkjod23nm1iatexe" src="https://cdn.durable.co/blocks/16A6ht3EQIcI3wPDUXEQnp3xOhF3iFrgg0NFLA2MWSzTVIVi5g1lYQzcMtt6kMaz.jpg" alt="two people sitting during day" title="two people sitting during day" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/16A6ht3EQIcI3wPDUXEQnp3xOhF3iFrgg0NFLA2MWSzTVIVi5g1lYQzcMtt6kMaz.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure><figure id="figure-tjkb0ukezm1iats0m"><img id="tjkb0ukezm1iats0m" src="https://cdn.durable.co/blocks/28EQ9aP7pP2mKiHQboXBKrrGeJZjdRp56tqkcXFftB3Ncx70HFUyknkr47qvhEUb.jpg" alt="person using laptop computer" title="person using laptop computer" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/28EQ9aP7pP2mKiHQboXBKrrGeJZjdRp56tqkcXFftB3Ncx70HFUyknkr47qvhEUb.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure>This presentation focused on AI-mentor complementarities for entrepreneurial training, that seems like the most powerful combination out there. But... we do have to deal with the fact that even if we are massively successful, if we scale up human-led training by 10x, that still wouldn't reach 1/5th of MSMEs in Peru, for example. So we also need to figure out how to make entrepreneurial training with AI available for the rest, who can't receive traditional in-person training. As the work of Otis et al. shows, this is definitely possible, but has to be done carefully! <figure class=""></figure>

Presentation to the ILO SIYB Master Trainers Network

As a rule I avoid new AI tool hype, but this was an interesting feature added to Google Notebook LM- a push-button explanatory 'podcast'. This one is based on just a couple documents about MAIA. It has some outdated figures (partiuclarly around costs, which we've managed to reduce quite a bit), but does a really great job of a big-picture explanation in a user-friendly, 'planet money-esque' format, based on just a couple documents. Try it for your own project, and follow<a data-mce-href="https://drive.google.com/file/d/1kksL5Fb0sHg8YUL-DYDTECt9-nPRHBpU/view?usp=sharing" href="https://drive.google.com/file/d/1kksL5Fb0sHg8YUL-DYDTECt9-nPRHBpU/view?usp=sharing" target="_blank" rel="noopener"> this link</a> to listen to the MAIA Fodcast!

MAIA Fodcast (fake podcast)

The biggest problem we face in applying emerging academic research on AI advisory tools for MSMEs to the real world is the external validity of those results. Let me explain with an example, the best paper out there so far: "We recruited entrepreneurs into our study over the Meta ad platform in partnership with the Busara Center, a Kenyan research organization, starting in May 2023. Figure 2 and Appendix C provide additional details on the experiment timeline and sample recruitment process. Our recruitment strategy involved running ads on Facebook to invite entrepreneurs to a short paid survey (Figure A1). All entrepreneurs who responded to our ads were required to pass basic attention checks and take part in three rounds of pre-treatment surveys delivered over the course of three months, which helped us reduce post-treatment attrition and ensure valid causal inference. Appendix E describes our surveys, which asked about firm profits, revenues, and management practices, among other measures. Our final sample includes 640 Kenyan entrepreneurs who completed all three pre-treatment surveys. The median entrepreneur in this sample was 26 years old, had been running their business for one year, and held a college degree (Table A1). " Otis et al p7.This approach makes complete sense to maximize the power of the study and minimize attrition. But if we are seeking to apply the results to all Kenyan entrepreneurs, we need to ask ourselves how representative are those who would see and respond to a facebook ad of those MSME entrepreneurs and pass all those steps? Might they differ in a way that would be relevant for the impact of intervention?Facebook penetration is high in that country, as it is in Peru, but the number of registered Facebook accounts per capita doesn't necessarily translate into a high proportion of MSME owners as frequent users who would click on a facebook ad.&nbsp;Moreover, not only did they have to click on the ad, they then had to pass all of the attention checks and complete the pre-treatment surveys. It seems quite possible that this would result in a set of participants that are younger, more technologically inclined, and more motivated and engaged in terms of looking for ways to improve their business. Indeed the paper describes the median participant in that study as a 26 year old college graduate, with a higher proportion of technology-related businesses than one would expect for Kenyan MSMEs as a whole. And you can compare the usage rates of the tool in the paper to what you would expect 'in the wild' (from an average set of owners who weren't recruited and offered rewards for participation). If you think that the impact of an AI advisor on business performance might be different for this group in some important ways, you need to be very careful in applying those results to the broader population.The average bodega owner in Peru is a mid 40s mother with no post-secondary education. In evaluating distribution channels in the country we met with a local digial marketing expert, and they indicated that facebook ads here could only reach a small and particular subset of bodega owners who are on average more educated, younger, and running more sophisticated operations. When I pressed them for a percentage of the universe of bodega owners that would be reached by Facebook ads, she said 3-5%, despite the high penetration rates of Facebook here.To repeat, this was not an error of the Otis study, their approach made perfect sense for that exercise. And because theirs the only real evaluation of these tools to date, we have paid very very close attention to it and incorporated its results explicitly in our current model prompt. However, it is an open question what the impact of these tools would be on a more representative set of MSME owners. And to figure that out, we need to get to more representative samples than are possible with online advertising. We are trying to do this do this by, for example, providing the tool to all bodegas in particular areas through partnerships with universal suppliers, or our own direct on-the-ground promoters. Its quite likely that usage patterns and tool impact will look quite different for this universe of MSME owners.

Blog

Usage Patterns Update!

Voice Mode in PHL

Beyond 'chat with the manual' II

Presentation to the ILO SIYB Master Trainers Network

MAIA Fodcast (fake podcast)

Worry about external validity