Blog

In the first class of my first semester as an undergraduate student, Professor Larry Boland sat in front of the class and just said ‘questions?’ over and over. After a few sessions we got used to his unique Socratic method, and most ended up enjoying the class. But it was an extremely awkward way to get started, particularly for a room full of new students.An LLM is a lot like that- its an open text box with a blinking cursor, awkwardly waiting for questions. It has access to a mind-boggling amount of knowledge, but its up to you to come to it and ask the right question in the right way. Its not obvious how to make this powerful tool useful.When you give a microenterprise owner in a developing country access to an LLM for the first time, they ask a few questions to test its knowledge and are impressed with the results, but similarly its not clear to them how to make it useful over time. What we’ve seen is that unaided, their pattern of usage looks very much like most people when they first try ChatGPT- they ask a few questions to test it out and are amazed with the results, but then move on with other priorities. There are some power users that will come back to it again on their own, but that is the minority, and you can see similar patterns in other LLM studies like <a data-mce-href="https://www.hbs.edu/faculty/Pages/download.aspx?name=24-042.pdf" href="https://www.hbs.edu/faculty/Pages/download.aspx?name=24-042.pdf" target="_blank" rel="noopener">this one for entrepreneurs in Kenya</a> and <a data-mce-href="https://arxiv.org/pdf/2310.02982.pdf" href="https://arxiv.org/pdf/2310.02982.pdf" target="_blank" rel="noopener">this one for teachers in Sierra Leone</a>, even with reminders and incentives.Many traditional entrepreneurial training programs are happy to find that minority of power users and focus on them, that is why <a data-mce-href="https://www.argidius.com/en/learning/how-to-fulfill-the-potential-of-business-development-services-using-scale" href="https://www.argidius.com/en/learning/how-to-fulfill-the-potential-of-business-development-services-using-scale" target="_blank" rel="noopener">selection </a>is a core part of how they work. But, to empower the majority of MSMEs, you need to make it useful for them, not just sit back and wait for questions. Our initial launch and focus group testing showed that they don’t want a guru sitting back and waiting for questions: after helping with their first set of priorities, they want a tool that proactively offers advice and ideas that they can engage with and interrogate when they want, and are free to ignore if they are busy.So that’s the approach we have taken so far. Every week the tool sends out a ‘tip of the week’, as well as a second message that is a rotating mix of system tips, poll questions, or special guided exercises like long-term goal setting or identifying ‘quick wins’.What we’ve observed so far with this approach is that for every 10 MSME owners who get access to MYPE Asesor IA, there is 1 power user who uses the tool proactively and engages with almost every message, there is 1 inactive user who maybe reads the message but never really engages with the tool, and the remaining 8 engage about half the time. Messages from the system have about a 50% response rate, but its not the same 50% of users week to week.Last week we put out a poll to see what users thought of this approach, asking if they wanted more, the same, or fewer messages from the system. Here are the results:<figure id="figure-8s3j6sd139ls9amtu0"><img id="8s3j6sd139ls9amtu0" src="https://cdn.durable.co/blocks/32c44xj6ym0KViHIBQnvrUsC2rc1NfG2Yda45qNw3fLhRZQrWcnCUkkz4mQ88HEs.jpg" alt="woman holding brown umbrella" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/32c44xj6ym0KViHIBQnvrUsC2rc1NfG2Yda45qNw3fLhRZQrWcnCUkkz4mQ88HEs.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure>So no one finds the system-initiated messages annoying enough to click the ‘I want fewer messages’ response in the poll. And among the half that did respond, they either like the current approach or want more. Eventually the system should adjust to those preferences- there isn’t any reason why everyone should get the same number of messages, or even the same tips of the week. &nbsp;The benefit of AI-based training is that the tool should adjust its approach for each small business owner. But its clear that the next experiment for A/B testing will be in the direction of more, rather than fewer, messages from the system.

What does early engagement look like?

Early research trying to test if entrepreneurial training improved micro, small, and medium-sized enterprise (MSME) performance suffered from a commom problem: what type of training? Training could be via lectures in a workshop, or individualized mentorship and consulting. It could focus on hard business skills, or on psychological traits like personal initiative. It could be a few hours with limited attendance, or an intensive in-depth process. After billions was spend training millions of entrepreneurs, best practices emerged, and it turned out that <a data-mce-href="https://voxdev.org/voxdevlit/training-entrepreneurs-issue-3" href="https://voxdev.org/voxdevlit/training-entrepreneurs-issue-3" target="_blank" rel="noopener">these design differences mattered quite a bit for impact</a>.&nbsp;Similarly, if we want to know if AI-enabled training, mentoring and consulting can improve MSME performance, the first question is what kind? An LLM-based tool could actually wear many hats. It could be the expert on a hill that entrepeneurs come to with their own questions, and which dispenses detailed information based on best practies. It could be more like a personal trainer, working with the entrepreneur to identify an improvement they can implement immediately for a quick win, and motivating them towards action. It could be like a career coach, taking the entrepreneur through a long-term goal setting exercise and building a top-down plan from there. It could follow the entpreneur's priorities, or use training materials and other users' queries to guide towards a broader set of topics. Becuase AI-based entrepreneurial training is so new, its not clear what hat the LLM should wear.&nbsp;In our early results, we're finding that rather than picking one approach, different entrepreneurs in different situations prefer a different mix of these approaches. And it is a mix rather than a single one- most of our early users have some business priorities of their own they want to address first, and then appreciate suggestions of new areas of inquiry. And after an initial testing of the system with detailed substantive questions, the key seems less about transferring knowledge and more about helping put knowledge into practice. Given how quickly AI capabilities are evolving, there are probably many new hats we haven't even considered yet. That is why our focus this year is offering and testing a variety of approaches and tools before focusing on impact evaluation and scale-up, and we're keen to share our ideas and to hear from others, so we can all figure out what works.&nbsp;

What Hat Should the LLM Wear?

Its been a busy month! Here is a summary of the key updates to the system:1) Time since last message is now included in the model prompt, so that the bot can distinguish between an ongoing conversation and a follow-up query a week later, for example.2) We've added in a persistent summary about the business owner that is also inserted into model prompts. OpenAI's threads in the Assistants API aren't the clearest about how long they save conversation history. As best as I can understand it after reading the documentation, particular messages are saved for 30 days and threads are deleted after 60 days without use. So we generate a user history based on conversation to date, and will set that to update regularly. This is obviously done by GPT4 as well, with the following prompt: {This spreadsheet contains the full record of a conversation between an AI business coach chatbot and a business owner. Generate a one paragraph summary about the owner. This summary will be used to give the chatbot context when providing advice to them in the future, so focus on details that would be useful context for a chatbot working as a business coach for this person. This should include a description of their business/sector, note their key business problems, and list what business topics have already been discussed with them. This description will be inserted in future model queries so write efficiently to save tokens. Don't include evidence for your summary (like quotes from the chat) or any other language that won't help customize future model queries.} Hopefully as the Assistants API improves, we'll be able to make the threads persistent, but this is a work-around for now.3) We've migrated from Zapier to Make.com for middleware. Zapier is easy but expensive, make.com requires a bit more effort but works well and is much lower cost, which also allows for building much more logic into the system (such as #1 and #2 above). Thankfully, with the help of ChatGPT even a civilian like me can bang out some JSON payloads with ease. Here is an overview of the scenario<figure id="figure-ke8f11xl8wlr297tfd"><img id="ke8f11xl8wlr297tfd" src="https://cdn.durable.co/blocks/12zotS4XICNiHjP7eVN3r7VbvBoOyzroaYZEqPyCcMMt6VozMeaLBuNodFyq8TS7.jpg" alt="" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="make.com scenario overview" data-positionx="undefined" data-positiony="undefined" data-mce-src="https://cdn.durable.co/blocks/12zotS4XICNiHjP7eVN3r7VbvBoOyzroaYZEqPyCcMMt6VozMeaLBuNodFyq8TS7.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"><figcaption class="body-normal text-gray-500 mt-2">make.com scenario overview</figcaption></figure>Next up is a customization for a new sector. We are curretly testing with Bodega (small retail shop) owners in Peru, of which there are almost 500,000. We have started on a new customied model for a pretty different sector: coffee growers. There are over 200,000 small-scale coffee growers in Peru, and a truly disorienting number of training manuals and best practices guides.&nbsp;

Updates January 2024

We've been looking for other projects similar to ours, and hadn't found much. There are private-sector products delivering AI chatbots via WhatsApp. And we've been reading a lot and studing the excelent work being done in the education area, in particular by the Khan Academy and their Khanmigo AI tutor (they have put together fantastic resources for educators about AI, benefits and risks, etc, and we are using them as a guide, particularly when it comes to openness and safety, check out more about Khanmigo <a data-mce-href="https://www.khanacademy.org/khan-labs" href="https://www.khanacademy.org/khan-labs" target="_blank" rel="noopener">here</a>). But I hadn't seen anything specifically for developing country entrepreneurs, until <a data-mce-href="https://osf.io/preprints/osf/hdjpk" href="https://osf.io/preprints/osf/hdjpk" target="_blank" rel="noopener">this great paper preprint was released yesterday</a>. From all my searching, these are the first people to do it, which is impressive in its own right, but they also have done it in as very clean and rigorous impact evaluation. It is great work!The findings are really interesting, and show that its not as simple as just hooking up deleloping country entrepreneurs to ChatGPT and then watch all the good things happen. The advice from their AI mentor seems to have had a notable positive impact on businesses that were already doing well, but actually may have hurt businesses that were already struggling. That may just be a temporary result, as it could take longer to turn around a struggling business than to improve an already successful business, with an initial dip in earnings as changes are made (compared to control firms who were struggling but may not be taking corrective action spured on by the mentor) - we'll have to see as they (hopefully) continue to collect outcomes data over time.It is also important to note that their AI Mentor has a particular approach, focus, and structure designed into it, and as a result the logs in the paper do show a dialogue that sounds different from ours. And 'ours' also act and sound very different from one another, depending on which flavor is being tested: structured or unstructured diagnostic, hard-skills or PI-focused, expert consutlant or collaborative coach, generalist or sector (eg bodega)-specific, etc. There are lots of approaches to instruct the LLM, materials to use for context, and ways to structure the user interaction. This paper shows that not only may some be more effective than others generally, but some could be better than others depending on the situation facing the business. More work to be done, but a fantastic first study!

We are not alone!

We are very early in the process of learning how AI can best be used to support emerging market entrepreneurs, and are actively testing many different approaches and gathering feedback. So any system snapshot will be out of date the day after its posted, but this information is important so every user can know exactly what model they are interacting with and what unerlies its behavior. It will also be useful to other economists and development practicitioners, particularly those who (like me) aren't natural-born developers and are looking for easy and efficient ways to work with AI tools.<h2>System Overview</h2>We use WhatsApp at the front and <a data-mce-href="https://platform.openai.com/docs/assistants/overview" href="https://platform.openai.com/docs/assistants/overview">OpenAI's Assistants API</a> running GPT4 at the back, with <a data-mce-href="zapier.com" href="zapier.com" target="_blank" rel="noopener">Zapier </a>in between. Zapier is expensive so I'm in the process of migrating to new middleware, but its a really easy way to get started. For WhatsApp, you need a business provider. Most seem to use <a data-mce-href="https://www.googleadservices.com/pagead/aclk?sa=L&amp;ai=DChcSEwi6k-KhhJKDAxXlFq0GHfxVAVIYABAAGgJwdg&amp;ase=2&amp;gclid=Cj0KCQiAj_CrBhD-ARIsAIiMxT-fbryoiDFL6TBOMWq9jr8xtbThZIgBhpmABauNcH44RA341QL4ZlYaAhawEALw_wcB&amp;ohost=www.google.com&amp;cid=CAESVuD2Fcrb2Xjq-eeUhtQZP4K0BO8ovPdkGO7UbIvBKyXdoxgo0Y-ouL1bAdC_85kCIBrafgFTQlCRw66idUXRkgYlYXK2WtKV_K09CI2CPkTkCwnOpZM7&amp;sig=AOD64_3aIYJtq4byXCt9dqXhyUzjqPJvtw&amp;q&amp;nis=4&amp;adurl&amp;ved=2ahUKEwistNuhhJKDAxU6OTQIHbQYC3EQ0Qx6BAgHEAE" href="https://www.googleadservices.com/pagead/aclk?sa=L&amp;ai=DChcSEwi6k-KhhJKDAxXlFq0GHfxVAVIYABAAGgJwdg&amp;ase=2&amp;gclid=Cj0KCQiAj_CrBhD-ARIsAIiMxT-fbryoiDFL6TBOMWq9jr8xtbThZIgBhpmABauNcH44RA341QL4ZlYaAhawEALw_wcB&amp;ohost=www.google.com&amp;cid=CAESVuD2Fcrb2Xjq-eeUhtQZP4K0BO8ovPdkGO7UbIvBKyXdoxgo0Y-ouL1bAdC_85kCIBrafgFTQlCRw66idUXRkgYlYXK2WtKV_K09CI2CPkTkCwnOpZM7&amp;sig=AOD64_3aIYJtq4byXCt9dqXhyUzjqPJvtw&amp;q&amp;nis=4&amp;adurl&amp;ved=2ahUKEwistNuhhJKDAxU6OTQIHbQYC3EQ0Qx6BAgHEAE" target="_blank" rel="noopener">Twilio</a>, but I used <a data-mce-href="woztell.com" href="woztell.com" target="_blank" rel="noopener">Woztell</a>, because they don't charge per message for user-initiated conversations, which is most of what we do, and they are helpful with the whole WhatsApp verification process, which is a bit of a pain (because META rightly wants to keep spam off the platform). The Assistant's API is a great way to get started as well, because it manages retreival (getting the right information from all the training manuals) and context (the conversation history for that particular user), though over time you'll want more control over those functions, that's our direction.<h2>Base LLM</h2>GPT4 is quite overpowered for this use case, but its cheap enough for piloting and is clearly the best model available, so the best place to start. OpenAI's prices have gone down a ton over 2023, and we'll see what happens in 2024 to detmine our direction. The open-source models like Llama and Mistral that are already showing GPT3.5-level performance are interesting, and over time it could make sense to tune a model to this particular use case, though so far we are getting qutie a bit of mileage from using retreival and context in a careful way.<h2>LLM Instructions</h2>We are experimenting with a number of different flavors here. The research suggests that personal initiative (PI) training can have similar or larger impacts than hard skills training, so some versions focus more on PI than others. There is also a tradeoff between making the interaction more structured (to help users who aren't used to interacting with LLMs) versus more flexible (following the user's own questions and business priorities). But here are the model instructions for the version that is being piloted today.{You are a business coach hired to help a microenterprise owner who runs a small store in Peru and hasn’t had much school. Your advice should be in the style and tone of the uploaded guides. Speak very simply, like the guides do.Your conversation with the microenterprise owner should encourage the development of Personal Initiative (PI) skills while also providing substantive, specific advice based on the guides when relevant. Be proactive in using information from the guides to suggest actionable steps.1) Check the Owner's first message. If it contains a specific question or problem, respond directly to that. If the message is a generic greeting or doesn’t contain a specific question, use the opening statement "Do you have a business problem you want help with, or can I start with a question to figure out how I can help you?".2) If Owner asks you to start, you should begin by understanding the business’s current performance, the problems faced, and the owner's aspirations for the future.3) Encourage self-starting behavior and provide specific advice, using the guides if relevant.4) Promote persistence by finding lessons from these steps and helping the owner learn from them.5) Suggest specific, actionable steps the Owner could take to improve their business, drawing upon the guides.6) Provide positive reinforcement for proactive ideas, and give useful feedback on Owners plans so that they are realistic and actionable.Remember to use the principles of Personal Initiative in your coaching, promoting self-starting, future-oriented, and persistent behavior in the context of Owner’s business. Stay relevant to the context of microenterprises in Peru, avoid suggesting ideas suitable for bigger stores in rich countries, and use the guides when relevant.Ask them only one question at a time about their business, limit all your messages to 100 words, and give simple replies followed by another simple question when appropriate. Give advice like a business mentor to help them sell more, spend less, and do better. When you arrive at a specific actionable step together that they will try out, end the conversation by asking them to write you in a few days to report how it went.Start in Spanish, but switch to English if the user asks for it. If the owner asks about topics completely unrelated to their business, reply “I am here to help you with your business, but I can’t help with other things”. If they say somthing that sounds like “no más mensajes”, stop the conversation.}Those of you who have played around with LLM prompting will see many common strategies here. I don't say "use the guides" so many times out of carelesness: GPT often needs reminidng of what is important. The knowledge base also keeps the conversation on topic and reduces risks of bad advice from hallucinations or just from a bias towards the developed country context that LLMs are largely built on. Finally, we've really found that wording instructions in simpler wording and vocabulary help the LLM speak that way to the user. LLMs are weird and behave in weird ways, and prompting them is an art that is just developing. Though many prompting tricks are also becoming less relevant as the models improve, particularly GPT4.&nbsp; &nbsp;<h2>Knowledge Base</h2>Because our first pilot is a narrowly-defined segment, the knowldge base for what is currently fielded is limited to the Bodegueros Best Practices guide which can be seen <a data-mce-href="https://www.indecopi.gob.pe/documents/1902049/3749566/MANUAL+DE+BUENAS+PR%C3%81CTICAS+BODEGUERAS.pdf/9e4d3e11-7ce1-4b06-3485-d55196b6d20f" href="https://www.indecopi.gob.pe/documents/1902049/3749566/MANUAL+DE+BUENAS+PR%C3%81CTICAS+BODEGUERAS.pdf/9e4d3e11-7ce1-4b06-3485-d55196b6d20f" target="_blank" rel="noopener">here</a> which I've optimized for RAG, along with some materials from the most widely-used entrepreneurship training programs, as summarized in McKenzie &amp; Woodruff 2023: the ILO's SIYB program, CEFE, and Empretec.<h2>Process Flow</h2>This is what it looks like:<figure id="figure-41ik60x4eolq6ywux3"><img id="41ik60x4eolq6ywux3" src="https://cdn.durable.co/covers/2c1tMNPjAo6Yv8Jx5hMvj2OBl4mjJXnI9tGjoBoGKzRsTLZlKneAPkZtACLzgBJA.jpg" alt="" class="object-cover bg-gray-100 " style="aspect-ratio: 3/2; object-position: undefined undefined;" data-corners="no-rounded" data-aspect="3:2" data-caption="" data-positionx="undefined" data-positiony="undefined" data-mce-src="https://cdn.durable.co/covers/2c1tMNPjAo6Yv8Jx5hMvj2OBl4mjJXnI9tGjoBoGKzRsTLZlKneAPkZtACLzgBJA.jpg" data-mce-style="aspect-ratio: 3/2; object-position: undefined undefined;"></figure>

System Snapshot

This is where we will post snapshots of all the details of our system including its architecture, what models we use, and how they are instructed. Our goal is to provide a playbook that others looking to do something similar can learn from. Starting a blog right when the internet is taken over by AI-generated content probably isn't the best move, but here we are. This is entirely human-written (though that's also what a sneaky AI would say).

Blog

What does early engagement look like?

What Hat Should the LLM Wear?

Updates January 2024

We are not alone!

System Snapshot

Welcome