Blog

At its most basic level, an AI chatbot on WhatsApp is just an easier front end to ChatGPT, with ChatGPT instructed in a particular way and supplemented with particular documents to make it more relevant and reliable for a particular group (like Peruvian microenterprise owners). It will then sit there and wait for questions. But as discussed in a <a data-mce-href="https://mypeasesoria.org/blog/beyond--chat-with-the-training-manual" href="https://mypeasesoria.org/blog/beyond--chat-with-the-training-manual" target="_blank" rel="noopener">previous post,</a> this "chat with the manual" bot isn't very useful, and won't get much use.Why? Its placing all of the burden of the user to know what questions to ask, how to use AI, and when. Small business owners are incredibly busy, they have very limited time, and limited experience with technology. Making these tools actually useful and usable isn't their job, its our job.We're only at the beginning of our experiments to learn how to do this better, but so far have found a few really useful things:1) A gradual onboarding process. One week after they first connect with Maia, we send them a list of the 3 most common questions across all users in their partiuclar sector, so they get a sense of the breadth of topics they can explore. The next week they get a guide on how to ask better questions to get better answers (provide more context, follow-up questions, etc). Two weeks later, they get a message they can forward to fellow MSME owners to share Maia, if they found it useful.&nbsp;2) The 'tip of the week'. Every week, users get a different tip built from the best practice manuals in the knowledge base. This surfaces new ideas to explore that they may not have thought of. Users can then discuss those topics further if they are relevant (with quick reply buttons, which are a must!) and very quickly get to a specific actionable plan to implement a particular improvement that they design with the help of the tool.3) Follow-up messages. Its one thing to know that we should eat well and exercise, doing it is another matter. Small business owners are busy, and many things come up that may need immediate attention but distract from important but non-urgent things like improvements. If users explored a particular topic the week before and identified some improvements they wanted to make, the following week they will get a follow-up message, to see if they implemented them (and if so, how it went and what are the next steps), and if not, an opportunity to refelct on and possibly revise the plan.

Beyond 'chat with the manual' II

Here is a presentation I gave today to the ILO's global network of SIYB master trainers. SIYB is the largest small business training program in the world, and it was a very interesting discussion!<figure class=""><img id="yrydhlkd4m1iao1v2" src="https://cdn.durable.co/blocks/18p7ZA33fXSdObi6mYMgjF2u8NDsMHxaUqK7ydtr5oP4xhOpEVufFHWggd5MqopM.jpg" alt="right arrow sign on wall" title="right arrow sign on wall" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/18p7ZA33fXSdObi6mYMgjF2u8NDsMHxaUqK7ydtr5oP4xhOpEVufFHWggd5MqopM.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure><figure id="figure-837va9m65m1iap9ed"><img id="837va9m65m1iap9ed" src="https://cdn.durable.co/blocks/73OKRA7Xb3wjJBicLj5AcpOIb4zirmY5iS8rmZ4PLUZtDxAaS3n8yDvHao8Tn7cV.jpg" alt="black smartphone near person" title="black smartphone near person" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/73OKRA7Xb3wjJBicLj5AcpOIb4zirmY5iS8rmZ4PLUZtDxAaS3n8yDvHao8Tn7cV.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure><figure id="figure-qdudzyw9erm1iappdy" class=""><img id="qdudzyw9erm1iappdy" src="https://cdn.durable.co/blocks/1bT37W1TZexY9S9Voxtx1NL24wnxfc6GZLjXrxMd4nAhKQOtYuGCfjfQu9zXNahq.jpg" alt="person holding compass selective focus photography" title="person holding compass selective focus photography" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/1bT37W1TZexY9S9Voxtx1NL24wnxfc6GZLjXrxMd4nAhKQOtYuGCfjfQu9zXNahq.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure>(Box 1 not covered today, but there is a ton of exciting ways that trainers could use AI to help them with things like lesson plans, exercises, role-playing, etc. See https://www.youtube.com/watch?v=8FnOkxj0ZuA to get started on AI for educators. The best source of ideas are the experiments of your fellow master trainers)<figure id="figure-gu1p0bki49m1iaqogu" class=""><img id="gu1p0bki49m1iaqogu" src="https://cdn.durable.co/blocks/d3PhcEBAyEm1XSveK4KgZBiwuWqA0vFqa2HlsNIkaQx5rlMXGnKfhtXraIFj3hqc.jpg" alt="woman in teal t-shirt sitting beside woman in suit jacket" title="woman in teal t-shirt sitting beside woman in suit jacket" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/d3PhcEBAyEm1XSveK4KgZBiwuWqA0vFqa2HlsNIkaQx5rlMXGnKfhtXraIFj3hqc.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure>(Box 2 not covered today, but this is just a new category of tools which trainers have always shared with trainees. Some examples: instead of teaching branding and giving advice on how to make a logo, AI image generation tools can just apply those best practice and make your business a logo. Other examples: customer surveys.)<figure id="figure-qiz7wl136m1iarj74" class=""><img id="qiz7wl136m1iarj74" src="https://cdn.durable.co/blocks/38wRZFkZKQ85QzDYQBNHHadvCMo8rdD34mMBNywOaHs7jvP9fHyoH0MGts6cUwZu.jpg" alt="black pencil on white printerpaper" title="black pencil on white printerpaper" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/38wRZFkZKQ85QzDYQBNHHadvCMo8rdD34mMBNywOaHs7jvP9fHyoH0MGts6cUwZu.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure><figure id="figure-2e7xjng0ym1iarxvf" class=""><img id="2e7xjng0ym1iarxvf" src="https://cdn.durable.co/blocks/eeA8IPaqzHLccHPKlP6RmuLbRoKHqzyqBRfHvGe3BwjHUloNWK6rFy4m2UtRaEDe.jpg" alt="three people sitting in front of table laughing together" title="three people sitting in front of table laughing together" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/eeA8IPaqzHLccHPKlP6RmuLbRoKHqzyqBRfHvGe3BwjHUloNWK6rFy4m2UtRaEDe.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure><figure id="figure-mu77k4tbjm1iasdl1"><img id="mu77k4tbjm1iasdl1" src="https://cdn.durable.co/blocks/22e52uphZ3zQpgjmKvHhqZClRaSwmbuWGOTPTuH0ut7yAhRP2XOAACTBEv1FfVl0.jpg" alt="man facing green mountains during daytime" title="man facing green mountains during daytime" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/22e52uphZ3zQpgjmKvHhqZClRaSwmbuWGOTPTuH0ut7yAhRP2XOAACTBEv1FfVl0.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure><figure id="figure-g2vr77hpum1iat2s8"><img id="g2vr77hpum1iat2s8" src="https://cdn.durable.co/blocks/5khVyfgshiCGfOCaj4M9Nx9rc1kPWeox3HnJ5q11C71BoWcspThawDoN9EkuWr0z.jpg" alt="two people sitting during day" title="two people sitting during day" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/5khVyfgshiCGfOCaj4M9Nx9rc1kPWeox3HnJ5q11C71BoWcspThawDoN9EkuWr0z.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure><figure id="figure-evwkjod23nm1iatexe" class=""><img id="evwkjod23nm1iatexe" src="https://cdn.durable.co/blocks/16A6ht3EQIcI3wPDUXEQnp3xOhF3iFrgg0NFLA2MWSzTVIVi5g1lYQzcMtt6kMaz.jpg" alt="two people sitting during day" title="two people sitting during day" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/16A6ht3EQIcI3wPDUXEQnp3xOhF3iFrgg0NFLA2MWSzTVIVi5g1lYQzcMtt6kMaz.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure><figure id="figure-tjkb0ukezm1iats0m"><img id="tjkb0ukezm1iats0m" src="https://cdn.durable.co/blocks/28EQ9aP7pP2mKiHQboXBKrrGeJZjdRp56tqkcXFftB3Ncx70HFUyknkr47qvhEUb.jpg" alt="person using laptop computer" title="person using laptop computer" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/28EQ9aP7pP2mKiHQboXBKrrGeJZjdRp56tqkcXFftB3Ncx70HFUyknkr47qvhEUb.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure>This presentation focused on AI-mentor complementarities for entrepreneurial training, that seems like the most powerful combination out there. But... we do have to deal with the fact that even if we are massively successful, if we scale up human-led training by 10x, that still wouldn't reach 1/5th of MSMEs in Peru, for example. So we also need to figure out how to make entrepreneurial training with AI available for the rest, who can't receive traditional in-person training. As the work of Otis et al. shows, this is definitely possible, but has to be done carefully! <figure class=""></figure>

Presentation to the ILO SIYB Master Trainers Network

As a rule I avoid new AI tool hype, but this was an interesting feature added to Google Notebook LM- a push-button explanatory 'podcast'. This one is based on just a couple documents about MAIA. It has some outdated figures (partiuclarly around costs, which we've managed to reduce quite a bit), but does a really great job of a big-picture explanation in a user-friendly, 'planet money-esque' format, based on just a couple documents. Try it for your own project, and follow<a data-mce-href="https://drive.google.com/file/d/1kksL5Fb0sHg8YUL-DYDTECt9-nPRHBpU/view?usp=sharing" href="https://drive.google.com/file/d/1kksL5Fb0sHg8YUL-DYDTECt9-nPRHBpU/view?usp=sharing" target="_blank" rel="noopener"> this link</a> to listen to the MAIA Fodcast!

MAIA Fodcast (fake podcast)

The biggest problem we face in applying emerging academic research on AI advisory tools for MSMEs to the real world is the external validity of those results. Let me explain with an example, the best paper out there so far: "We recruited entrepreneurs into our study over the Meta ad platform in partnership with the Busara Center, a Kenyan research organization, starting in May 2023. Figure 2 and Appendix C provide additional details on the experiment timeline and sample recruitment process. Our recruitment strategy involved running ads on Facebook to invite entrepreneurs to a short paid survey (Figure A1). All entrepreneurs who responded to our ads were required to pass basic attention checks and take part in three rounds of pre-treatment surveys delivered over the course of three months, which helped us reduce post-treatment attrition and ensure valid causal inference. Appendix E describes our surveys, which asked about firm profits, revenues, and management practices, among other measures. Our final sample includes 640 Kenyan entrepreneurs who completed all three pre-treatment surveys. The median entrepreneur in this sample was 26 years old, had been running their business for one year, and held a college degree (Table A1). " Otis et al p7.This approach makes complete sense to maximize the power of the study and minimize attrition. But if we are seeking to apply the results to all Kenyan entrepreneurs, we need to ask ourselves how representative are those who would see and respond to a facebook ad of those MSME entrepreneurs and pass all those steps? Might they differ in a way that would be relevant for the impact of intervention?Facebook penetration is high in that country, as it is in Peru, but the number of registered Facebook accounts per capita doesn't necessarily translate into a high proportion of MSME owners as frequent users who would click on a facebook ad.&nbsp;Moreover, not only did they have to click on the ad, they then had to pass all of the attention checks and complete the pre-treatment surveys. It seems quite possible that this would result in a set of participants that are younger, more technologically inclined, and more motivated and engaged in terms of looking for ways to improve their business. Indeed the paper describes the median participant in that study as a 26 year old college graduate, with a higher proportion of technology-related businesses than one would expect for Kenyan MSMEs as a whole. And you can compare the usage rates of the tool in the paper to what you would expect 'in the wild' (from an average set of owners who weren't recruited and offered rewards for participation). If you think that the impact of an AI advisor on business performance might be different for this group in some important ways, you need to be very careful in applying those results to the broader population.The average bodega owner in Peru is a mid 40s mother with no post-secondary education. In evaluating distribution channels in the country we met with a local digial marketing expert, and they indicated that facebook ads here could only reach a small and particular subset of bodega owners who are on average more educated, younger, and running more sophisticated operations. When I pressed them for a percentage of the universe of bodega owners that would be reached by Facebook ads, she said 3-5%, despite the high penetration rates of Facebook here.To repeat, this was not an error of the Otis study, their approach made perfect sense for that exercise. And because theirs the only real evaluation of these tools to date, we have paid very very close attention to it and incorporated its results explicitly in our current model prompt. However, it is an open question what the impact of these tools would be on a more representative set of MSME owners. And to figure that out, we need to get to more representative samples than are possible with online advertising. We are trying to do this do this by, for example, providing the tool to all bodegas in particular areas through partnerships with universal suppliers, or our own direct on-the-ground promoters. Its quite likely that usage patterns and tool impact will look quite different for this universe of MSME owners.

Worry about external validity

Happy 'Dia del Bodeguero'! August 12th was the day of the bodeguero in Peru. At the Peruvian Bodega Association's celebration, Peru's Minister of Production highlighted Maia, and did a live demo for the crowd. The Ministry's entrepreneurial training program Tu Empresa will incorporate Maia into its programs and offer it to participants, and Bodegueros across Peru can now use Maia thanks to some international support soon to be announced on our site. In the meantime, some news stories from print can be found <a data-mce-href="https://www.elperuano.pe/noticia/250164-herramienta-con-ia-para-bodegueros" href="https://www.elperuano.pe/noticia/250164-herramienta-con-ia-para-bodegueros" target="_blank" rel="noopener">here </a>and <a data-mce-href="https://diariouno.pe/2024/08/13/ministerio-de-la-produccion-promueve-herramienta-con-inteligencia-artificial-para-fortalecer-la-productividad-de-los-bodegueros/#google_vignette" href="https://diariouno.pe/2024/08/13/ministerio-de-la-produccion-promueve-herramienta-con-inteligencia-artificial-para-fortalecer-la-productividad-de-los-bodegueros/#google_vignette" target="_blank" rel="noopener">here </a>and from TV <a data-mce-href="https://plataforma.ipnoticias.com/Landing?i=8rjVc38Q1fmQN9n3eazhjw%3d%3d&amp;cac=rN2GE%2fYIVP99UB5qFlVYiA%3d%3d&amp;c=rD%2b4wLEDjFCxdp6PZfPQQEPMkj5kts1ILGBwH%2fiMysw%3d&amp;utm_source=alerta&amp;utm_medium=correo&amp;utm_content=video&amp;utm_campaign=videomail" href="https://plataforma.ipnoticias.com/Landing?i=8rjVc38Q1fmQN9n3eazhjw%3d%3d&amp;cac=rN2GE%2fYIVP99UB5qFlVYiA%3d%3d&amp;c=rD%2b4wLEDjFCxdp6PZfPQQEPMkj5kts1ILGBwH%2fiMysw%3d&amp;utm_source=alerta&amp;utm_medium=correo&amp;utm_content=video&amp;utm_campaign=videomail" target="_blank" rel="noopener">here</a>.

Feliz Dia del Bodeguero!

The Oits et al paper, which offered the very first and best evidence about if and how AI training could work for MSMEs, has a new version which you can find <a data-mce-href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4671369" href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4671369" target="_blank" rel="noopener">here</a>. If you have read the first version, do take a look, because the revisions are significant. There is additional work unpacking user converstaions which is quite useful, and the interpretation of the results has evolved significantly. The main conclusion of the paper is:"Our findings show that generative AI-based tools can serve as an efficient source of personalized feedback for many entrepreneurs in low- and middle-income countries. In fact, the treatment effect we find for high performers is similar to the 10% to 15% performance improvements driven by human-to-human training programs (Brooks, Donovan, and Johnson, 2018; McKenzie, 2021), but are likely delivered at a fraction of the cost."In other words, strong evidence that this can work- we can deliver at least similar (and potentailly greater) productivity impacts compared to traditional training, at 1/100th the cost, to a much greater proportion of developing country MSMEs. All we have to do is just hook them all up to ChatGPT and watch the impact roll in, right? Well, its not that simple, because the authors go on: "That being said, we also find evidence that generative AI can widen the gap between low- and high-performing businesses in these contexts. More broadly, our findings are consistent with the idea that generative AI has the potential—if designed and deployed with care—to benefit the billions of people and millions of firms in developing countries."The Otis et al AI Mentor was desgined in a specific way, you can see all the details in the paper. It generates a set of potential solutions to any particular problem, then offers greater detail on any one the user wants to investigate further. In some ways, its a bit like a brainstorming engine. Their results show that previously well-performing MSMEs improved their performance with this brainstorming engine, but previously poorly-performing MSMEs actually did slightly worse after using it. And it makes sense that high performers will be more able to filter ideas from this engine, use the tool to refine them, and do better, while poor performers may lack the judgement to select the best ideas. Interestingly, there is forthcoming work by the lead author Nick Otis using an AI mentor on a similar population with a different design and functioning, and the results are actually quite different.What does this tell us? It actually matters quite a bit how the AI tool used by MSMEs is designed- how the model is instructed, what resources it uses, how the user experience is structured. Some designs seem to work very well for some entrepreneurs, but not well for others, and they even have the potential to harm.So, its not quite a slam dunk here, there is quite a bit of work to left to do, to harness the potential and avoid the harms. These tools have to be designed and deployed with care. That is the goal of this project, to evlauate a variety of potential designs. Luckily, we don't have to start from scratch- there is actually a wealth of ideas and results in the deacdes of entrepreneurial training research to draw on. And, we can also directly use the results from Otis et al's paper, and their and others' future work, to refine tool designs to maximize impact and minimize harm. You'll see this directly the next time we post the model instructions. That shows the value of making this process open and collaborative rather than closed and proprietary.

Blog

Beyond 'chat with the manual' II

Presentation to the ILO SIYB Master Trainers Network

MAIA Fodcast (fake podcast)

Worry about external validity

Feliz Dia del Bodeguero!

Kenya AI Mentor (Otis et al) Revised