Blog

The biggest problem we face in applying emerging academic research on AI advisory tools for MSMEs to the real world is the external validity of those results. Let me explain with an example, the best paper out there so far: "We recruited entrepreneurs into our study over the Meta ad platform in partnership with the Busara Center, a Kenyan research organization, starting in May 2023. Figure 2 and Appendix C provide additional details on the experiment timeline and sample recruitment process. Our recruitment strategy involved running ads on Facebook to invite entrepreneurs to a short paid survey (Figure A1). All entrepreneurs who responded to our ads were required to pass basic attention checks and take part in three rounds of pre-treatment surveys delivered over the course of three months, which helped us reduce post-treatment attrition and ensure valid causal inference. Appendix E describes our surveys, which asked about firm profits, revenues, and management practices, among other measures. Our final sample includes 640 Kenyan entrepreneurs who completed all three pre-treatment surveys. The median entrepreneur in this sample was 26 years old, had been running their business for one year, and held a college degree (Table A1). " Otis et al p7.This approach makes complete sense to maximize the power of the study and minimize attrition. But if we are seeking to apply the results to all Kenyan entrepreneurs, we need to ask ourselves how representative are those who would see and respond to a facebook ad of those MSME entrepreneurs and pass all those steps? Might they differ in a way that would be relevant for the impact of intervention?Facebook penetration is high in that country, as it is in Peru, but the number of registered Facebook accounts per capita doesn't necessarily translate into a high proportion of MSME owners as frequent users who would click on a facebook ad.&nbsp;Moreover, not only did they have to click on the ad, they then had to pass all of the attention checks and complete the pre-treatment surveys. It seems quite possible that this would result in a set of participants that are younger, more technologically inclined, and more motivated and engaged in terms of looking for ways to improve their business. Indeed the paper describes the median participant in that study as a 26 year old college graduate, with a higher proportion of technology-related businesses than one would expect for Kenyan MSMEs as a whole. And you can compare the usage rates of the tool in the paper to what you would expect 'in the wild' (from an average set of owners who weren't recruited and offered rewards for participation). If you think that the impact of an AI advisor on business performance might be different for this group in some important ways, you need to be very careful in applying those results to the broader population.The average bodega owner in Peru is a mid 40s mother with no post-secondary education. In evaluating distribution channels in the country we met with a local digial marketing expert, and they indicated that facebook ads here could only reach a small and particular subset of bodega owners who are on average more educated, younger, and running more sophisticated operations. When I pressed them for a percentage of the universe of bodega owners that would be reached by Facebook ads, she said 3-5%, despite the high penetration rates of Facebook here.To repeat, this was not an error of the Otis study, their approach made perfect sense for that exercise. And because theirs the only real evaluation of these tools to date, we have paid very very close attention to it and incorporated its results explicitly in our current model prompt. However, it is an open question what the impact of these tools would be on a more representative set of MSME owners. And to figure that out, we need to get to more representative samples than are possible with online advertising. We are trying to do this do this by, for example, providing the tool to all bodegas in particular areas through partnerships with universal suppliers, or our own direct on-the-ground promoters. Its quite likely that usage patterns and tool impact will look quite different for this universe of MSME owners.

Worry about external validity

Happy 'Dia del Bodeguero'! August 12th was the day of the bodeguero in Peru. At the Peruvian Bodega Association's celebration, Peru's Minister of Production highlighted Maia, and did a live demo for the crowd. The Ministry's entrepreneurial training program Tu Empresa will incorporate Maia into its programs and offer it to participants, and Bodegueros across Peru can now use Maia thanks to some international support soon to be announced on our site. In the meantime, some news stories from print can be found <a data-mce-href="https://www.elperuano.pe/noticia/250164-herramienta-con-ia-para-bodegueros" href="https://www.elperuano.pe/noticia/250164-herramienta-con-ia-para-bodegueros" target="_blank" rel="noopener">here </a>and <a data-mce-href="https://diariouno.pe/2024/08/13/ministerio-de-la-produccion-promueve-herramienta-con-inteligencia-artificial-para-fortalecer-la-productividad-de-los-bodegueros/#google_vignette" href="https://diariouno.pe/2024/08/13/ministerio-de-la-produccion-promueve-herramienta-con-inteligencia-artificial-para-fortalecer-la-productividad-de-los-bodegueros/#google_vignette" target="_blank" rel="noopener">here </a>and from TV <a data-mce-href="https://plataforma.ipnoticias.com/Landing?i=8rjVc38Q1fmQN9n3eazhjw%3d%3d&amp;cac=rN2GE%2fYIVP99UB5qFlVYiA%3d%3d&amp;c=rD%2b4wLEDjFCxdp6PZfPQQEPMkj5kts1ILGBwH%2fiMysw%3d&amp;utm_source=alerta&amp;utm_medium=correo&amp;utm_content=video&amp;utm_campaign=videomail" href="https://plataforma.ipnoticias.com/Landing?i=8rjVc38Q1fmQN9n3eazhjw%3d%3d&amp;cac=rN2GE%2fYIVP99UB5qFlVYiA%3d%3d&amp;c=rD%2b4wLEDjFCxdp6PZfPQQEPMkj5kts1ILGBwH%2fiMysw%3d&amp;utm_source=alerta&amp;utm_medium=correo&amp;utm_content=video&amp;utm_campaign=videomail" target="_blank" rel="noopener">here</a>.

Feliz Dia del Bodeguero!

The Oits et al paper, which offered the very first and best evidence about if and how AI training could work for MSMEs, has a new version which you can find <a data-mce-href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4671369" href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4671369" target="_blank" rel="noopener">here</a>. If you have read the first version, do take a look, because the revisions are significant. There is additional work unpacking user converstaions which is quite useful, and the interpretation of the results has evolved significantly. The main conclusion of the paper is:"Our findings show that generative AI-based tools can serve as an efficient source of personalized feedback for many entrepreneurs in low- and middle-income countries. In fact, the treatment effect we find for high performers is similar to the 10% to 15% performance improvements driven by human-to-human training programs (Brooks, Donovan, and Johnson, 2018; McKenzie, 2021), but are likely delivered at a fraction of the cost."In other words, strong evidence that this can work- we can deliver at least similar (and potentailly greater) productivity impacts compared to traditional training, at 1/100th the cost, to a much greater proportion of developing country MSMEs. All we have to do is just hook them all up to ChatGPT and watch the impact roll in, right? Well, its not that simple, because the authors go on: "That being said, we also find evidence that generative AI can widen the gap between low- and high-performing businesses in these contexts. More broadly, our findings are consistent with the idea that generative AI has the potential—if designed and deployed with care—to benefit the billions of people and millions of firms in developing countries."The Otis et al AI Mentor was desgined in a specific way, you can see all the details in the paper. It generates a set of potential solutions to any particular problem, then offers greater detail on any one the user wants to investigate further. In some ways, its a bit like a brainstorming engine. Their results show that previously well-performing MSMEs improved their performance with this brainstorming engine, but previously poorly-performing MSMEs actually did slightly worse after using it. And it makes sense that high performers will be more able to filter ideas from this engine, use the tool to refine them, and do better, while poor performers may lack the judgement to select the best ideas. Interestingly, there is forthcoming work by the lead author Nick Otis using an AI mentor on a similar population with a different design and functioning, and the results are actually quite different.What does this tell us? It actually matters quite a bit how the AI tool used by MSMEs is designed- how the model is instructed, what resources it uses, how the user experience is structured. Some designs seem to work very well for some entrepreneurs, but not well for others, and they even have the potential to harm.So, its not quite a slam dunk here, there is quite a bit of work to left to do, to harness the potential and avoid the harms. These tools have to be designed and deployed with care. That is the goal of this project, to evlauate a variety of potential designs. Luckily, we don't have to start from scratch- there is actually a wealth of ideas and results in the deacdes of entrepreneurial training research to draw on. And, we can also directly use the results from Otis et al's paper, and their and others' future work, to refine tool designs to maximize impact and minimize harm. You'll see this directly the next time we post the model instructions. That shows the value of making this process open and collaborative rather than closed and proprietary.

Kenya AI Mentor (Otis et al) Revised

There is a great 3-part series by Eugene Yan, Bryan Bischof, Charles Frye, Hamel Husain, Jason Liu and Shreya Shankar called "What we learned from a year of building with LLMs (find it <a data-mce-href="https://www.oreilly.com/radar/what-we-learned-from-a-year-of-building-with-llms-part-i/" href="https://www.oreilly.com/radar/what-we-learned-from-a-year-of-building-with-llms-part-i/" target="_blank" rel="noopener">here</a>).&nbsp; Based on some of those insights, we have updated the prompt structure, also borrowing some principles of structuring from Lilach and Ethan Mollick. Also, in the balance in focus between providing hard information vs. building personal initaitive and motivating action, we have slightly adjusted more towards the latter, based on a recent round of user feedback. Here is is the current (Bodega) model prompt:&nbsp;&nbsp;-------------------Your Persona: You are MAIA, an opinionated, empathetic, and supportive business coach.&nbsp;Your Goal: help a microenterprise Owner in Latin America improve their small retail shop (bodega). You must help them spend less, earn more, and grow.&nbsp;The time since the Owner's last message may precede the Owner's message as [milliseconds], and a previous message to Owner not yet in the thread may also precede Owner’s message in [[[]]]. Use these for context.Follow these steps in order: 1) analyze the user message and thread history, 2) learn important context and prioritize, and then 3) offer advice and create an action plan with Owner based on that advice.STEP 1) Analyze user message and conversation history: Carefully analyze the owner's message to understand their current question, priorities, and any updates they share on actions they have taken since your last conversation. If it has been more than 24 hours (86400000 milliseconds) since their last message, look for opportunities to follow up on any actions in previous messages. If they took action, praise their effort to build their personal initiative. Help them learn from that action to identify further improvements and reinforce the lessons.STEP 2) Learn important context and prioritize before offering advice: Ask a series of clarifying questions, including follow-up questions, to gain a deeper understanding of the owner's specific challenges, needs, and goals before offering advice. Ensure each question helps gather detailed information to tailor your advice effectively. Ask those questions one at a time, and craft them to require as little typing as possible by the owner to answer. Before offering advice, direct the conversation towards the most impactful changes the owner could make based on their current situation and priorities with their limited time and resources.STEP 3) Advice and action plan: Use the owner's responses and the information you gather through your questions to tailor your advice to their unique circumstances and challenges. Base your advice on best practices from the attached training guides, which are your primary Knowledge Base (KB). Talk simply and plainly like the KB does. Prioritize file-JgMjwRKPlrCUxhMVc97QBgiA which has information specific to stores in Peru. Your advice should encourage the development of the owner's personal initiative (build self-reliance and persistence, steer the conversation towards concrete steps to improve their business, reinforce their proactive ideas, ensure suggested plans are practical and achievable, break plans down into small steps, starting with the very first one, and encourage a growth mindset by highlighting the owner's progress, however small, and emphasizing that setbacks are learning opportunities). To build personal initiative, your advice should lead to a simple, actionable plan with 1-2 specific steps they can take immediately to implement your advice. Make each step clear and achievable, defining the first step in detail, establishing a realistic timeframe for completion, and offering to follow up. Encourage the owner to provide feedback on the advice given and iterate on plans based on their experiences and results. Use this feedback to refine future advice and support.Tone and response parameters: Stay relevant to the context of Peruvian microenterprises and don’t suggest things for fancy businesses in rich countries. Acknowledge the owner's feelings and concerns and provide specific positive reinforcement and support, especially when discussing setbacks or challenges, to demonstrate empathy and understanding. Keep all your responses brief, using 100 words or less, as the conversation is happening over WhatsApp. Ask only one question at a time. Respond in the language of the owner's messages. Default to Spanish, but switch if they ask for another language. If the owner tries to discuss topics that seem unrelated to their business, first ask a clarifying question to ensure you understand their message correctly. For example, you could ask, "¿Podrías explicar un poco más cómo esto se relaciona con tu negocio?" If it is indeed unrelated, gently steer the conversation back to business topics by saying, "Estoy aquí para ayudarte con tu negocio. ¿De qué manera puedo ayudarte?" Discourage them from sharing personal details like account numbers. If they say something that sounds like “no más mensajes,” stop the conversation.Remember that your goal is to help Owner take action, which means not only providing information. You need to help them quickly identify an improvement that they can make today, and then follow-up.

Prompt update July 2024

As discussed in a previous post, the basic version of an LLM-based advisory bot for MSMEs is like some all-knowing guru that sits in your WhatsApp contact list and waits for questions. This could be based on the LLM's training data, or on top of management best practice guides: a 'chat with your training manuals' tool.This places all the burden on the entrepreneurs to know the space of potential questions and use cases. It also places the burden on them to come back to the tool despite all the demands on their time and attention. And it does not help them at all with the most important step of translating information into an action plan, and following up. If a real life business coach did that, they wouldn't be very helpful, and we can see that an LLM-based tool working like that is also not very helpful and is quickly forgotten. Since 2023, WhatsApp has been running <a data-mce-href="https://developers.facebook.com/docs/whatsapp/cloud-api/guides/experiments/" href="https://developers.facebook.com/docs/whatsapp/cloud-api/guides/experiments/" target="_blank" rel="noopener">an experiement</a> and a random selection of 1% of users can't receive any business messages unless they write first. We have a few of these accounts in our user pool, and find that they ask a few questions when they first try the tool, and then go back to life and all the other demands on their time and rarely re-engage.&nbsp;To be useful, there needs to be a two-way communication, rather than just a guru sitting on a mountain. In the other implementation of this type of tool <a data-mce-href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4671369" href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4671369" target="_blank" rel="noopener">in Kenya by the Otis et al team of academic researchers</a>, their bot sent out reminders to use the tool (note though that those participants were pre-screened for the study and offered a reward for their completion, so its pretty different from a real-world set-up). There are many other promising ways that these tools can proactively engage with MSME entrepreneurs:- Send out 'tips of the week' based on training materials- Share the most frequently asked questions by other users from the previous month- Scope out well-defined use cases where the tool can offer more direct and applied advice and even help complete tasks (eg upload a photo of your display for step-by-step imrpovements, calculate and evaluate margins on a particular product to plan a sale, get a new logo, etc)- Exercises that the entrepreneurial training research has shown impactful for MSMEs (eg a goal setting module, a kaizen-type module, personal initiative-focused module)- Personalized follow-up to build momentum (eg last week we talked about X and you said you wanted to try Y. Did you try it? How did that go?)A core focus of our work is testing these and other ideas out, and finding out from our growing set of Bodega owners which ones were must useful and impactful for their businesses, and how else they'd like the tool to help them. Becuase they want and deserve something more useful than 'chat with the training manual'.

Beyond 'chat with the training manual'

For many applications, 'user engagement' is their north star. It makes sense: users have to actually use your product, and the more they use it, the more useful it is, right? This is clearly important if you are selling ads, but what if you are offering advice. Is more advice better?I spoke with a MAIA user recently. After what looked like some useful discussions with the tool around how to improve sales (store appearance, customer access, etc), she did not re-engage for nearly a month. I took that absense as an indication that she didn't find the tool useful, and wanted to learn more. But she said quite the opposite: she loved the tool. She was in the process of implementing a handfull of changes based on her conversation, and was already seeing very positive results. Some of those changes involved physical changes (moving gates) that take some time. I asked why she wasn't using MAIA regularly to find more improvements, and she said that she will come back to it when she needs new ideas, but right now she's still implementing the first action plan. What's the point of more ideas if she doesn't have the time to actually enact them?I hate golf, but its a useful analogy. If you take a golf lesson, a bad pro will give you 20 generic tips to improve your swing, you'll remember one or two of them, and maybe you'll get a bit better. A good pro, however, will give you just the one tip that will fix your most critical problem at that point in your evoluation as a player, for maximum impact. The same holds for business advice. Better consultants don't give more recommendations, they give the smallest set of key improvements that have maximum impact.So we will look at and report on engagement. But beyond some point, more usage isn't necessarly better, it could actually be worse. So business performance and improvement will be our north star, not engagement, and we'll figure out what engagement patterns are associated with maximum business impact.[PS- the day after posting this, I attended a presentation by Nick Otis, the lead author of that interesting Kenya paper mentioned earlier. He mentioned that in his recent research, they've found some negative relationships between intensity of AI mentor usage and business performance. His explanation for this was that chatting with the tool could at some point substitute for talking with real people, which has some benefits that AI doesn't eg introductions and networking. This is analogous to what researchers find about why screen time is a problem for young kids]

Blog

Worry about external validity

Feliz Dia del Bodeguero!

Kenya AI Mentor (Otis et al) Revised

Prompt update July 2024

Beyond 'chat with the training manual'

Engagement, More Advice & Better Advice