Blog

A super interesting study <a data-mce-href="https://twitter.com/emollick/status/1770618237782307075" href="https://twitter.com/emollick/status/1770618237782307075" target="_blank" rel="noopener">as summarized by Ethan Mollick</a>: "Bloomberg trained a GPT-3.5 class AI on their own financial data last year… …only to find that GPT-4 8k, without specialized finance training, beat it on almost all finance tasks. Hard to beat the frontier models."Many organizations are applying AI in International Development, and have sensible arguments for fine-tuning their own models. A particular country's language may be poorly represented in LLM training data, the application may be very sepcialized (eg agricultural extension), there are government rules around where data goes, and a fine-tuned open source model may look much more attractive when making financial projections out to millions and millions of users.&nbsp;There are many valid reasons, but we should also be on guard for the risk of some motivated reasoning here: non-profits still have incentives to 'build a moat' to survive and thrive. An NGO working to be a leader in AI and _ will, just like many AI start-ups, not want to look like a thin GPT "wrapper", and one way to do that is to fine-tune your own model instead of using off-the-shelf models.&nbsp;We are taking the opposite approach, and trying to be the thinnest wrapper possible that achieves significant impact for MSMEs. Much like unconditional cash transfers give an impact benchmark that more highly-engineered development interventions must beat, our results give an impact benchmark that proprietary models should beat, and which any institution can quickly implement on their own and achieve. Its important to avoid spending donor money to train another BloombergGPT.----------An aside on the other issues raised above:COST: Yes its true that GPT4 is expensive now, GPT4V has a low rate limit, etc. But look at the prices of GPT 3.5 a year ago compared to now, and project prices forward the 12-18mos when a solution is fully tested out and optimized, before which it isn't ready for massive scale-up.&nbsp;LANGUAGE: To figure out the right intervention model and UX we're starting with what is a very easy language (Spanish) that still gives access to millions and millions of low-income MSMEs needing help.&nbsp;SPECIALIZATION: The services sector absorbs a massive amount of labor in low-income countries, with a generic production process that is relatively easy for LLMs to address, and persistent productivity gaps, so its the obvious place to start. But we will test how specalized a production process has to get before the simple approach of a frontier model supplemented with basic RAG doesn't outperform a less capable fine-tuned model, potentially in the coffee sector.

BloombergGPT, Fine Tuning & Moat building

Below is the system prompt as of March 2024. A couple things to note:- In general, we don't use prompting tricks (like offering a tip) because I've found their impact is either small, or becomes small pretty quickly as the model providers adjust. <a data-mce-href="https://www.oneusefulthing.org/p/captains-log-the-irreducible-weirdness" href="https://www.oneusefulthing.org/p/captains-log-the-irreducible-weirdness" target="_blank" rel="noopener">This advice of Ethan Mollick</a> seems right on: "This stuff does work, but not all the time, and in ways that are hard to anticipate. Sometimes it backfires. You probably don’t need to use magic words frequently."- I do continue to find that using simpler vocabulary in the prompt (eg 'fancy businesses in rich countries') is more effective at getting the model to use simple language in its completions than just asking it to do so.&nbsp;- I've found LLMs to be really really bad at prompt writing. Every attempt I've made to improve our prompt with the help of ChatGPT made it perform worse. But the research noted in Ethan's blog post (which created a crazy star trek prompt to solve math problems) did that in a more systematic way so maybe it can work if done carefully. But at first pass I'd never use LLMs for prompt writing.&nbsp;- Someone from Khan Academy who worked on their AI tutor Khanmigo told me they found better performance when they wrote system prompts in the same language as the completion they wanted. So next up we will try converting the prompt to Spanish and see its impact. To be fair, Khan Academy was working in less common languages than Spanish, so that lesson may not apply. PROMPTYou are MAIA, an opinionated and supportive business coach helping a microenterprise Owner of a small store who hasn’t had much school. Your goal is to have a focused exchange identifying specific contextualized advice, encouraging Personal Initiative (PI), and following-up.Your advice should be in the style and tone of the attached training guides, which are your primary Knowledge Base (KB). Talk simply and plainly like the KB does. Be proactive in offering specific, actionable advice from the KB, and prioritize file-JgMjwRKPlrCUxhMVc97QBgiA which has information specific to stores in Peru. Always rephrase KB content in simple words. Give advice and provide examples based on details you have learned about Owner’s business.You should encourage the development of PI, by 1) encouraging self-reliance and building persistence, 2) steering the conversation towards specific steps for business improvement, 3) reinforcing proactive ideas, 4) making sure plans are practical and achievable, and 5) breaking plans down into small steps and starting with the first step, so it's not overwhelming.Owner’s questions and priorities should guide the discussion, but give your opinion about what their priorities should be. Owner has limited time, effort and money, and you must help them spend those on the most important things. Suggest other areas of business improvement if they ask you for topics to discuss (which might be a message that only says ‘you’ or ‘something new’). Ask short focused questions to diagnose problems and contextualize solutions. Guide Owner to specific actionable steps that they will try out, and then offer to discuss other business topics.&nbsp;The time since Owner's last message may precede the Owner's message as [milliseconds], use this for context. Find opportunities to ask about actions that the Owner identified in past chats (from more than 24 hours ago), to 1) determine if they acted and, if so, to praise that effort to build their personal initiative, 2) to help them learn from that action to identify better advice, further improvements, and reinforce what was learned.Stay relevant to the context of Peruvian microenterprises: don’t suggest things for fancy businesses in rich countries. Your conversation is through whatsapp, so keep every response short, 100 words or less, and ask only one question at a time. Start in Spanish, but switch to another language if the user asks for it. If the owner asks about topics completely unrelated to running a business, reply “I am here to help you with your business, but I can’t help with other things”. Don't ask for personal information like account numbers, and discourage Owner from providing that. If they say something that sounds like “no más mensajes”, stop the conversation. Additional context may be provided in CHARACTER_HIDDEN.

System Prompt Update

Last week, we launched image analysis. Now in addition to text messages, users can send a picture of the interior or exterior of their Bodega and get specific advice on potential improvements.&nbsp;Technical DetailsWe are using GPT4V, released by OpenAI a few months ago. We instruct that model to base its advice on best practices in our Knowledge Base, in particular the practices in <a data-mce-href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3685514" href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3685514" target="_blank" rel="noopener">Anderson-Macdonald et al 2020</a>. That paper implemented a randomized control trial with small retail shops across Mexico, and found that some simple improvements in exterior features of the shops (eg improvements in lighting, product organization, store layout) increased monthly sales by 15-20%. The model prompt is shown below. The output of this model is then fed into the general model to further contextualize the advice in the conversation with the Owner (and because GPT4V isn't available in the Assistants API yet) .&nbsp;Multi-Modal Inputs &amp; Knowledge DiffusionThe results of Anderson-Macdonald have been published for years and are available online for free. The knowledge of how to increase sales 15-20% is just sitting there, but it is inaccessible to your typical Bodega owner because it is in an online academic paper. These and other best practices can be found in more accessible guides distributed by governments and NGOs helping small retailers, but even still the knowledge remains largely inaccessible. It is too much to ask Bodegueros to absorb hundreds of pages of best practices and perfectly recall them and figure out how to apply them to particular actions at a particular time. The text-based AI coach helps solve this problem by making this information instantly available, interactive, and customized, but that still is relying on the owner to realize they could make this particular improvement and ask the bot about it (or have the bot suggest it generically in one of our 'tips of the week'). Now Bodegueros can just take a picture and the model figures that out for them and offers an immediate solution. That is very powerful in reducing persistent barriers to knowledge diffusion, and so far our users are loving it.GPT4V Prompt{You are a business coach hired to help a microenterprise owner who runs a small store in Peru and hasn’t had much school. You are an expert in marketing and in techniques like the “external modernization structures” listed in FILEREF. Owner has sent you this picture of their store. You need to tell them one specific improvement, based on this picture. First, specifically state what from the picture you see as a potential shortcoming. Then tell them step by step, how to fix it, specifically referencing what to do where based on the picture. Finally, tell them why this improvement will help. Speak in informal Peruvian Spanish. Limit all messages to 100 words.}

Multimodal Powered Knowledge Diffusion

In the first class of my first semester as an undergraduate student, Professor Larry Boland sat in front of the class and just said ‘questions?’ over and over. After a few sessions we got used to his unique Socratic method, and most ended up enjoying the class. But it was an extremely awkward way to get started, particularly for a room full of new students.An LLM is a lot like that- its an open text box with a blinking cursor, awkwardly waiting for questions. It has access to a mind-boggling amount of knowledge, but its up to you to come to it and ask the right question in the right way. Its not obvious how to make this powerful tool useful.When you give a microenterprise owner in a developing country access to an LLM for the first time, they ask a few questions to test its knowledge and are impressed with the results, but similarly its not clear to them how to make it useful over time. What we’ve seen is that unaided, their pattern of usage looks very much like most people when they first try ChatGPT- they ask a few questions to test it out and are amazed with the results, but then move on with other priorities. There are some power users that will come back to it again on their own, but that is the minority, and you can see similar patterns in other LLM studies like <a data-mce-href="https://www.hbs.edu/faculty/Pages/download.aspx?name=24-042.pdf" href="https://www.hbs.edu/faculty/Pages/download.aspx?name=24-042.pdf" target="_blank" rel="noopener">this one for entrepreneurs in Kenya</a> and <a data-mce-href="https://arxiv.org/pdf/2310.02982.pdf" href="https://arxiv.org/pdf/2310.02982.pdf" target="_blank" rel="noopener">this one for teachers in Sierra Leone</a>, even with reminders and incentives.Many traditional entrepreneurial training programs are happy to find that minority of power users and focus on them, that is why <a data-mce-href="https://www.argidius.com/en/learning/how-to-fulfill-the-potential-of-business-development-services-using-scale" href="https://www.argidius.com/en/learning/how-to-fulfill-the-potential-of-business-development-services-using-scale" target="_blank" rel="noopener">selection </a>is a core part of how they work. But, to empower the majority of MSMEs, you need to make it useful for them, not just sit back and wait for questions. Our initial launch and focus group testing showed that they don’t want a guru sitting back and waiting for questions: after helping with their first set of priorities, they want a tool that proactively offers advice and ideas that they can engage with and interrogate when they want, and are free to ignore if they are busy.So that’s the approach we have taken so far. Every week the tool sends out a ‘tip of the week’, as well as a second message that is a rotating mix of system tips, poll questions, or special guided exercises like long-term goal setting or identifying ‘quick wins’.What we’ve observed so far with this approach is that for every 10 MSME owners who get access to MYPE Asesor IA, there is 1 power user who uses the tool proactively and engages with almost every message, there is 1 inactive user who maybe reads the message but never really engages with the tool, and the remaining 8 engage about half the time. Messages from the system have about a 50% response rate, but its not the same 50% of users week to week.Last week we put out a poll to see what users thought of this approach, asking if they wanted more, the same, or fewer messages from the system. Here are the results:<figure id="figure-8s3j6sd139ls9amtu0"><img id="8s3j6sd139ls9amtu0" src="https://cdn.durable.co/blocks/32c44xj6ym0KViHIBQnvrUsC2rc1NfG2Yda45qNw3fLhRZQrWcnCUkkz4mQ88HEs.jpg" alt="woman holding brown umbrella" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/32c44xj6ym0KViHIBQnvrUsC2rc1NfG2Yda45qNw3fLhRZQrWcnCUkkz4mQ88HEs.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure>So no one finds the system-initiated messages annoying enough to click the ‘I want fewer messages’ response in the poll. And among the half that did respond, they either like the current approach or want more. Eventually the system should adjust to those preferences- there isn’t any reason why everyone should get the same number of messages, or even the same tips of the week. &nbsp;The benefit of AI-based training is that the tool should adjust its approach for each small business owner. But its clear that the next experiment for A/B testing will be in the direction of more, rather than fewer, messages from the system.

What does early engagement look like?

Early research trying to test if entrepreneurial training improved micro, small, and medium-sized enterprise (MSME) performance suffered from a commom problem: what type of training? Training could be via lectures in a workshop, or individualized mentorship and consulting. It could focus on hard business skills, or on psychological traits like personal initiative. It could be a few hours with limited attendance, or an intensive in-depth process. After billions was spend training millions of entrepreneurs, best practices emerged, and it turned out that <a data-mce-href="https://voxdev.org/voxdevlit/training-entrepreneurs-issue-3" href="https://voxdev.org/voxdevlit/training-entrepreneurs-issue-3" target="_blank" rel="noopener">these design differences mattered quite a bit for impact</a>.&nbsp;Similarly, if we want to know if AI-enabled training, mentoring and consulting can improve MSME performance, the first question is what kind? An LLM-based tool could actually wear many hats. It could be the expert on a hill that entrepeneurs come to with their own questions, and which dispenses detailed information based on best practies. It could be more like a personal trainer, working with the entrepreneur to identify an improvement they can implement immediately for a quick win, and motivating them towards action. It could be like a career coach, taking the entrepreneur through a long-term goal setting exercise and building a top-down plan from there. It could follow the entpreneur's priorities, or use training materials and other users' queries to guide towards a broader set of topics. Becuase AI-based entrepreneurial training is so new, its not clear what hat the LLM should wear.&nbsp;In our early results, we're finding that rather than picking one approach, different entrepreneurs in different situations prefer a different mix of these approaches. And it is a mix rather than a single one- most of our early users have some business priorities of their own they want to address first, and then appreciate suggestions of new areas of inquiry. And after an initial testing of the system with detailed substantive questions, the key seems less about transferring knowledge and more about helping put knowledge into practice. Given how quickly AI capabilities are evolving, there are probably many new hats we haven't even considered yet. That is why our focus this year is offering and testing a variety of approaches and tools before focusing on impact evaluation and scale-up, and we're keen to share our ideas and to hear from others, so we can all figure out what works.&nbsp;

What Hat Should the LLM Wear?

Its been a busy month! Here is a summary of the key updates to the system:1) Time since last message is now included in the model prompt, so that the bot can distinguish between an ongoing conversation and a follow-up query a week later, for example.2) We've added in a persistent summary about the business owner that is also inserted into model prompts. OpenAI's threads in the Assistants API aren't the clearest about how long they save conversation history. As best as I can understand it after reading the documentation, particular messages are saved for 30 days and threads are deleted after 60 days without use. So we generate a user history based on conversation to date, and will set that to update regularly. This is obviously done by GPT4 as well, with the following prompt: {This spreadsheet contains the full record of a conversation between an AI business coach chatbot and a business owner. Generate a one paragraph summary about the owner. This summary will be used to give the chatbot context when providing advice to them in the future, so focus on details that would be useful context for a chatbot working as a business coach for this person. This should include a description of their business/sector, note their key business problems, and list what business topics have already been discussed with them. This description will be inserted in future model queries so write efficiently to save tokens. Don't include evidence for your summary (like quotes from the chat) or any other language that won't help customize future model queries.} Hopefully as the Assistants API improves, we'll be able to make the threads persistent, but this is a work-around for now.3) We've migrated from Zapier to Make.com for middleware. Zapier is easy but expensive, make.com requires a bit more effort but works well and is much lower cost, which also allows for building much more logic into the system (such as #1 and #2 above). Thankfully, with the help of ChatGPT even a civilian like me can bang out some JSON payloads with ease. Here is an overview of the scenario<figure id="figure-ke8f11xl8wlr297tfd"><img id="ke8f11xl8wlr297tfd" src="https://cdn.durable.co/blocks/12zotS4XICNiHjP7eVN3r7VbvBoOyzroaYZEqPyCcMMt6VozMeaLBuNodFyq8TS7.jpg" alt="" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="make.com scenario overview" data-positionx="undefined" data-positiony="undefined" data-mce-src="https://cdn.durable.co/blocks/12zotS4XICNiHjP7eVN3r7VbvBoOyzroaYZEqPyCcMMt6VozMeaLBuNodFyq8TS7.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"><figcaption class="body-normal text-gray-500 mt-2">make.com scenario overview</figcaption></figure>Next up is a customization for a new sector. We are curretly testing with Bodega (small retail shop) owners in Peru, of which there are almost 500,000. We have started on a new customied model for a pretty different sector: coffee growers. There are over 200,000 small-scale coffee growers in Peru, and a truly disorienting number of training manuals and best practices guides.&nbsp;

Blog

BloombergGPT, Fine Tuning & Moat building

System Prompt Update

Multimodal Powered Knowledge Diffusion

What does early engagement look like?

What Hat Should the LLM Wear?

Updates January 2024