Blog

<h3>Why evaluation matters for MAIA </h3>Every week, hundreds of small business owners use Maia to improve their business. And Maia is not a finished tool, we are constantly tinkering and experimenting to make it better. To keep improving—and to reassure partners and users that the tool is both safe and effective—we need a systematic way to test whether MAIA is doing what we think it’s doing. Over the past month have posted information about our testing &amp; evaluation framework, take-up patterns, and experiment results.A timely reference point to tie all this together is the “AI Evaluation Framework for the Development Sector” published by the Center for Global Development (CGD) and written by leaders from the Agency Fund. The authors outline four incremental levels of evaluation—from technical correctness all the way to real-world impact. Importantly, this framwork adds sorely needed nuance to the previous model in development of simply 'intervention+RCT', and highlights asking the right questions at the right stage.Below, we map MAIA’s current (and planned) evaluation activities to each level.<h3>Level 1 - Model Evaluation</h3>Does Maia behave the way we want it to? If the tool can't accurately give users information about their tax deadline or best practices in sales and marketing, there is no need to wait for an RCT. Moreover, we need to make frequent decisions about updating the prompt, or updating to new LLMs as they are released. This is why we use <a data-mce-href="https://mypeasesoria.org/blog/testing---evaluation-framework" href="https://mypeasesoria.org/blog/testing---evaluation-framework" target="_blank" rel="noopener">an extensive testing and evaluation framework</a>, which is partially automated but also uses some subjective response rating, to evaluate responses on a regular sechedule as well as with any major update.&nbsp;<h3>Level 2 - Product Evaluation</h3>Are users engaging with the tool and solving real-world problems? One important feature of our T&amp;E process is manual review. Bailey dedicates up to 2 hours every day manually reviewing chat logs to evaluate all aspects of the product, identify bugs and improvements, etc. In addition, <a data-mce-href="https://mypeasesoria.org/blog/usage-patterns-update" href="https://mypeasesoria.org/blog/usage-patterns-update">we closely monitor usage patterns</a> (Monthly Active Users and Weekly Active Users), and use these as <a data-mce-href="https://mypeasesoria.org/blog/designing-better-first-interactions" href="https://mypeasesoria.org/blog/designing-better-first-interactions" target="_blank" rel="noopener">early indicators for product improvements </a>to iterate fast.<h3>Level 3 - User Evaluation</h3>First, we directly ask users if the tool is helping them and their business, <a data-mce-href="https://mypeasesoria.org/blog/has-maia-helped-users-improve-their-businesses" href="https://mypeasesoria.org/blog/has-maia-helped-users-improve-their-businesses">see results here</a>. We are also working in partnership with the ILO to test some cognitive outcomes among small-scale coffee farmers using the tool, specifically surveying their knowlege of specific best practices for production, worker safety, and formalization. Finally, with our new and improved follow-up messages we will be able to measure how often users self-report implementing their action plans from the previous week, and study why is that higher for some users than others.<h3>Level 4 - Independent Impact Evaluation</h3>Our ultimate goal is for Maia to directly and causally improve business outcomes for users, so that is what we want to measure. I have published a quasi-experimental impact evaluation (regression discontinuity) of entrepreneurial training programs and collaborated extensively with organizations like JPAL and IPA on other work. Our goal is to get to as good of a product that we can using Levels 1-3, as well as to the nescessary user volumes, and then collaborate on RCTs of the tool. We seek to follow in the footsteps of Otis et al and go beyond the simple up-front question 'what is the impact of this AI tool' and further unpack the more complicated question of what types of AI assistance are useful (or harmful) for what kinds of entrepreneurs facing what situations.

Evaluating Maia

A core goal of Maia is to serve as an on-ramp to AI for traditional micro- and small-business owners in emerging markets. One barrier we encountered is well known in HCI literature: blank-page syndrome. When the first thing users see in an AI tool is an empty text box—“Cuéntame tu problema”—many fall off the onboarding process. But immediately directing users to specific use cases can hide one of AI's key benefits over traditional e-learning tools (it range) and will also cause new user fall-off if the specific examples are irrelevant for them.Weekly Active Users as a % of Cohort<figure id="figure-xrjbpce92m9zs2ym2"><img id="xrjbpce92m9zs2ym2" src="https://cdn.durable.co/blocks/1dvSOMDiF9DegNvKpAHdpfkFYe7D2dYeTOz9mLCgiy7cj0VvlpP9JsEvJwQRgIcw.png" alt="" title="" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/1dvSOMDiF9DegNvKpAHdpfkFYe7D2dYeTOz9mLCgiy7cj0VvlpP9JsEvJwQRgIcw.png" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure>One approach we tried was to deliver specific use cases gradually, the first and second week after a user first connected. The idea was to have a very open interaction first, to experience Maia's versatility, and then give more specific examples of how other owners are using it. Seemed like a good idea, but one week was just too late, with too many users dropping off up-front from blank-page syndrome. Even moving the examples to just 23 hours after their first connection was too late. If a busy business owner doesn't see value right away, they are out.The other approach is to give the examples up-front. We first did this with an optional button (press 💡 if you'd like some examples of how other entrepreneurs have used Maia), but even the small friction of the button reduced use, so we modified the flow to quickly give example questions to everyone (eg 'Send a photo of your menu or your signature dish, and I'll give you ideas to make them look irresistible' for Maia restaurants, 'how can I improve my presence on social media' for artisans).&nbsp;The winning approach is starting more specific and then exposing the users to the broad range over time, rather than starting broad and teaching specfics over time. Quick business wins with minimal friction are essential, and because MSMEs share common needs, we can offer use cases that are both specific and broadly relevant.

Designing Better First Interactions

In their book ‘Abundance’, Thompson and Klein discuss how innovation often isn't a single 'eureka!' moment, but rather a patient process of tinkering and accumulating many 'micro-inventions'. This really resonates with our journey building Maia, and I wanted to share one recent example: the evolution of our weekly follow-up messages.Initially, Maia functioned like an <a data-mce-href="https://mypeasesoria.org/blog/beyond--chat-with-the-manual--ii" href="https://mypeasesoria.org/blog/beyond--chat-with-the-manual--ii" target="_blank" rel="noopener">‘ask an expert’, ‘chat with the manual’ type tool</a>, enhanced by proactive 'Tips of the Week' introducing best practices. But it was clear that users needed more to bridge the gap between knowing and doing. So the next step was adding a Monday morning follow-up feature targeting users who, guided by Maia, had defined a specific action plan during the previous week. Maia would remind them of their plan and ask if they'd implemented it. This served as a simple behavioral nudge – a prompt designed to increase follow-through and reduce the common intention-action gap faced by busy entrepreneurs.However, analyzing chat logs and talking with users revealed that many engaged with a topic but didn't quite reach the action plan stage – demands of entrepreneurial life often intervened! To support that group, we introduced a second follow-up version. This one revisited the topic discussed the previous week and prompted users to take the next step to define a concrete action now.This second version proved to be a significant unlock. It reached a broader group of entrepreneurs beyond the initially highly motivated ones who finalized action plans immediately. Many were intrigued by a Tip of the Week, started a conversation, but got pulled away. This gentle Monday nudge successfully guided a large portion of this group towards creating an action plan, effectively widening Maia's reach and impact – <a data-mce-href="https://mypeasesoria.org/blog/worry-about-external-validity" href="https://mypeasesoria.org/blog/worry-about-external-validity" target="_blank" rel="noopener">a key challenge for many digital tools aiming for broad external validity</a>.But the tinkering doesn't stop there. During a recent review of these follow-ups, we saw an opportunity to move beyond simple check-ins towards fostering deeper reflective practice. Encouraging users to reflect on their actions – what worked, what didn’t, and why – is crucial for building agency, problem-solving skills, and ultimately achieving greater impact than just skill transmission alone. <a data-mce-href="https://voxdev.org/voxdevlit/training-entrepreneurs" href="https://voxdev.org/voxdevlit/training-entrepreneurs" target="_blank" rel="noopener">Impact evaluations of entrepreneurial training</a> shows this clearly. This aligns with concepts like Kolb's experiential learning cycle, where reflection transforms experience into learning.Interestingly, I used the latest frontier models (GPT 4.5 and Gemini 2.5) as thought partners during this refinement process, which was incredibly insightful compared to what the frontier models of 6-12 months ago brough to the table. Together, we redesigned the action plan follow-up to specifically prompt this reflection. Instead of just asking if the action was done, the message now asks how it went, using Quick Reply buttons like Sí, pero con dificultades ("Yes, but with problems") and Aún no, mejoremos plan ("Not yet, let's improve the plan").Early results are promising! We're already seeing users engage more thoughtfully when they encounter obstacles, using these prompts to analyze challenges and adjust their strategies with Maia's help. It’s a small change, but it fosters resilience and encourages users to engage in metacognition – thinking about their own actions and learning processes. This whole evolution is a nice example of how innovating with AI to help microenterprise owners in developing countries doesn’t come from a single idea, and instead is build through persistent tinkering, learning from users, and accumulating many 'micro-innovations'.

Tinkering and Micro-innovations

This week we asked some users in Peru "Has MAIA helped you improve your business?" Here are the responses:<figure id="figure-np07lix5bm85zcl87"><img id="np07lix5bm85zcl87" src="https://cdn.durable.co/covers/27ohYz9tQJ3cBACTW38qxxR3ejlvq0wqD49dNe27YAGSO7vsIYpTlShi18MlNcTl.jpg" alt="" title="" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/covers/27ohYz9tQJ3cBACTW38qxxR3ejlvq0wqD49dNe27YAGSO7vsIYpTlShi18MlNcTl.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure>We're very happy with these results! In marketing terms we could translate that into an approximate Net Promoter Score of 74, which is excellent. Though I was surprised to learn that NPS ignores non-responses, and to me the response rate seems very important. Our response rate on the above question was 46%. This isn't our field, but according to ChatGPT, response rates for customer satisfaction surveys are usually 10-30% via email and 20-40% in-app. So our 46% response rate is telling. Double win!What we hope to do are financial impact measurements from a survey as part of a third-party RCT, to get to a robust dollar value of impact. We will pursue that in a future phase of this project, after our current round of A/B testing and tool refinements. That plan has been complicated by the unfortunate closing of USAID's DIV, one of our main funders, but we soldier on.We really want to thank our users for trusting us and using our tool. These entrepreneurs in Peru are facing major challenges, particularly relating to crime and input costs, yet continue to work and thrive, they are an inspiration. We're humbled to be part of their team.

Has Maia helped users improve their businesses?

For anyone looking to make their own AI-based MSME support tool with a WhatsApp front-end, here are sequence and activity diagrams for both the basic components (responding to user messages) and additional components (system-generated tips and follow-up) of Maia. I've provided the mermaid code, just paste this text into any free tool like mermaid.live to see the diagrams, and you can adapt them to your own case.Core Components - Sequence DiagramsequenceDiagram&nbsp; &nbsp; participant User&nbsp; &nbsp; participant WhatsAppProvider as WhatsApp Business Provider&nbsp; &nbsp; participant BackEnd as Back-End Automation&nbsp; &nbsp; participant ThreadManagement as Thread Management&nbsp; &nbsp; participant RAG as Retrieval Augmented Generation&nbsp; &nbsp; participant LLM as Large Language Model&nbsp; &nbsp; participant DataStorage as Data Storage &nbsp; &nbsp; Note over User,WhatsAppProvider: WhatsApp Front-end&nbsp; &nbsp; Note over WhatsAppProvider,BackEnd: Provider-Specific Integration&nbsp; &nbsp; Note over BackEnd: Automation &amp; Orchestration&nbsp; &nbsp; Note over ThreadManagement: Manage Conversations &amp; Threads&nbsp; &nbsp; Note over RAG: Retrieve &amp; Augment Context&nbsp; &nbsp; Note over LLM: Generate Responses&nbsp; &nbsp; Note over DataStorage: Data Storage &nbsp; &nbsp; User-&gt;&gt;WhatsAppProvider: Sends message&nbsp; &nbsp; WhatsAppProvider-&gt;&gt;BackEnd: Sends message &nbsp; &nbsp; alt First-time user&nbsp; &nbsp; &nbsp; &nbsp; BackEnd-&gt;&gt;User: Sends onboarding questions&nbsp; &nbsp; &nbsp; &nbsp; User-&gt;&gt;WhatsAppProvider: Sends responses&nbsp; &nbsp; &nbsp; &nbsp; WhatsAppProvider-&gt;&gt;BackEnd: Sends responses&nbsp; &nbsp; &nbsp; &nbsp; BackEnd-&gt;&gt;BackEnd: Initialize user&nbsp; &nbsp; else Existing user&nbsp; &nbsp; &nbsp; &nbsp; BackEnd-&gt;&gt;BackEnd: Sanitize message&nbsp; &nbsp; &nbsp; &nbsp; BackEnd-&gt;&gt;ThreadManagement: Add message to thread&nbsp; &nbsp; &nbsp; &nbsp; ThreadManagement--&gt;&gt;BackEnd: Confirmation&nbsp; &nbsp; &nbsp; &nbsp; BackEnd-&gt;&gt;RAG: Request context&nbsp; &nbsp; &nbsp; &nbsp; RAG--&gt;&gt;BackEnd: Requested context&nbsp; &nbsp; &nbsp; &nbsp; BackEnd-&gt;&gt;LLM: Generate response&nbsp; &nbsp; &nbsp; &nbsp; LLM--&gt;&gt;BackEnd: Response&nbsp; &nbsp; &nbsp; &nbsp; BackEnd-&gt;&gt;BackEnd: Sanitize response&nbsp; &nbsp; &nbsp; &nbsp; BackEnd-&gt;&gt;WhatsAppProvider: Send response&nbsp; &nbsp; &nbsp; &nbsp; WhatsAppProvider-&gt;&gt;User: Sends response&nbsp; &nbsp; end&nbsp; &nbsp; BackEnd-&gt;&gt;DataStorage: Stream all data Additional Components - Sequence DiagramsequenceDiagram&nbsp; &nbsp; participant User&nbsp; &nbsp; participant WhatsAppProvider as WhatsApp Business Provider&nbsp; &nbsp; participant BackEnd as Back-End Automation&nbsp; &nbsp; participant RAG as Retrieval Augmented Generation&nbsp; &nbsp; participant LLM as Large Language Model&nbsp; &nbsp; participant DataStorage as Data Storage &nbsp; &nbsp; opt Tip Message Flow – Time Trigger initiates tip generation&nbsp; &nbsp; &nbsp; BackEnd-&gt;&gt;RAG: Request tip context/resources&nbsp; &nbsp; &nbsp; RAG--&gt;&gt;BackEnd: Provide tip context/resources&nbsp; &nbsp; &nbsp; BackEnd-&gt;&gt;LLM: Generate tip using context&nbsp; &nbsp; &nbsp; LLM--&gt;&gt;BackEnd: Return generated tip&nbsp; &nbsp; &nbsp; BackEnd-&gt;&gt;WhatsAppProvider: Send tip message to user&nbsp; &nbsp; &nbsp; WhatsAppProvider-&gt;&gt;User: Deliver tip message&nbsp; &nbsp; end &nbsp; &nbsp; opt Follow-Up Message Flow – Time Trigger initiates follow-up generation&nbsp; &nbsp; &nbsp; BackEnd-&gt;&gt;DataStorage: Query user's chat history&nbsp; &nbsp; &nbsp; DataStorage--&gt;&gt;BackEnd: Return chat history&nbsp; &nbsp; &nbsp; BackEnd-&gt;&gt;LLM: Generate follow-up message using chat history&nbsp; &nbsp; &nbsp; LLM--&gt;&gt;BackEnd: Return follow-up message&nbsp; &nbsp; &nbsp; BackEnd-&gt;&gt;WhatsAppProvider: Send follow-up message to user&nbsp; &nbsp; &nbsp; WhatsAppProvider-&gt;&gt;User: Deliver follow-up message&nbsp; &nbsp; end &nbsp; &nbsp; BackEnd-&gt;&gt;DataStorage: Stream all data New User - Activity Diagramflowchart TD&nbsp; &nbsp; A([Start])&nbsp;&nbsp; &nbsp; B[User sends message via WhatsApp]&nbsp; &nbsp; C[WhatsApp Business Provider receives message]&nbsp; &nbsp; D[Back-End receives message]&nbsp; &nbsp; E{Is user new?}&nbsp; &nbsp;&nbsp;&nbsp; &nbsp; %% Branch for existing users&nbsp; &nbsp; F[Existing user – proceed with normal flow]&nbsp; &nbsp;&nbsp;&nbsp; &nbsp; %% Branch for new user onboarding&nbsp; &nbsp; G[Determine user's country from phone number]&nbsp; &nbsp; H["Ask user for economic activity&lt;br&gt;(options based on country)"]&nbsp; &nbsp; I[User replies with economic activity]&nbsp; &nbsp; J["Initialize user profile&lt;br&gt;(store country, activity, system variables)"]&nbsp; &nbsp; K[Send country-sector specific welcome messages]&nbsp; &nbsp;&nbsp;&nbsp; &nbsp; L([End])&nbsp; &nbsp;&nbsp;&nbsp; &nbsp; A --&gt; B&nbsp; &nbsp; B --&gt; C&nbsp; &nbsp; C --&gt; D&nbsp; &nbsp; D --&gt; E&nbsp; &nbsp; E -- "No" --&gt; F&nbsp; &nbsp; F --&gt; L&nbsp; &nbsp; E -- "Yes" --&gt; G&nbsp; &nbsp; G --&gt; H&nbsp; &nbsp; H --&gt; I&nbsp; &nbsp; I --&gt; J&nbsp; &nbsp; J --&gt; K&nbsp; &nbsp; K --&gt; L Returning User - Activity Diagramflowchart TD &nbsp; Start(["Start: Existing User&lt;br&gt;Message Received"]) &nbsp; A["User sends message"] &nbsp; B["WhatsApp Provider&lt;br&gt;forwards message&lt;br&gt;to Back-End"] &nbsp; X{"Can user receive&lt;br&gt;response?"} &nbsp; Y["Send invalid user&lt;br&gt;template"] &nbsp; C{"Is message audio?"} &nbsp; D["Transcribe audio&lt;br&gt;message"] &nbsp; E["Sanitize&lt;br&gt;transcript"] &nbsp; F{"Is message image?"} &nbsp; G["Fetch image and&lt;br&gt;accompanying text"] &nbsp; H["Sanitize accompanying&lt;br&gt;text"] &nbsp; I["Sanitize text (or button&lt;br&gt;press)"] &nbsp; J["Determine sanitized&lt;br&gt;message content"] &nbsp; K["Check if user has an&lt;br&gt;existing thread"] &nbsp; L{"Thread exists?"} &nbsp; M["If not, create new&lt;br&gt;thread"] &nbsp; N1["Assign LLM instructions&lt;br&gt;and RAG resources&lt;br&gt;based on country and sector"] &nbsp; N["Add sanitized content,&lt;br&gt;image, RAG, and other&lt;br&gt;context to thread"] &nbsp; O["Thread Management&lt;br&gt;confirms thread update"] &nbsp; P["Request context&lt;br&gt;from RAG"] &nbsp; Q["Receive context&lt;br&gt;from RAG"] &nbsp; R["Run LLM on thread&lt;br&gt;with RAG context"] &nbsp; S["Poll model run and&lt;br&gt;fetch response"] &nbsp; T["Sanitize generated&lt;br&gt;response"] &nbsp; U["Send response via&lt;br&gt;WhatsApp Provider"] &nbsp; V["User receives&lt;br&gt;response"] &nbsp; End(["End"]) &nbsp; Start --&gt; A&nbsp; A --&gt; B&nbsp; B --&gt; X&nbsp; X -- "Yes" --&gt; C&nbsp; X -- "No" --&gt; Y&nbsp; Y --&gt; End&nbsp; C -- "Yes" --&gt; D&nbsp; D --&gt; E&nbsp; E --&gt; J&nbsp; C -- "No" --&gt; F&nbsp; F -- "Yes" --&gt; G&nbsp; G --&gt; H&nbsp; H --&gt; J&nbsp; F -- "No" --&gt; I&nbsp; I --&gt; J&nbsp; J --&gt; K&nbsp; K --&gt; L&nbsp; L -- "Yes" --&gt; N1&nbsp; L -- "No" --&gt; M&nbsp; M --&gt; N1&nbsp; N1 --&gt; N&nbsp; N --&gt; O&nbsp; O --&gt; P&nbsp; P --&gt; Q&nbsp; Q --&gt; R&nbsp; R --&gt; S&nbsp; S --&gt; T&nbsp; T --&gt; U&nbsp; U --&gt; V&nbsp; V --&gt; End Weekly Tips - Activity Diagramflowchart TD&nbsp; &nbsp; A[Start Tip Message Process: Time Trigger]&nbsp; &nbsp; B[Request tip context/resources from RAG]&nbsp; &nbsp; C[Receive tip context/resources from RAG]&nbsp; &nbsp; D[Generate tip using LLM]&nbsp; &nbsp; E[Send tip message to WhatsApp Provider]&nbsp; &nbsp; F[Deliver tip message to User]&nbsp; &nbsp; G{Did user reply?}&nbsp; &nbsp; H[Include tip as additional context]&nbsp; &nbsp; I[Begin existing user reply process]&nbsp; &nbsp; J[End Process] &nbsp; &nbsp; A --&gt; B&nbsp; &nbsp; B --&gt; C&nbsp; &nbsp; C --&gt; D&nbsp; &nbsp; D --&gt; E&nbsp; &nbsp; E --&gt; F&nbsp; &nbsp; F --&gt; G&nbsp; &nbsp; G -- Yes --&gt; H&nbsp; &nbsp; H --&gt; I&nbsp; &nbsp; G -- No --&gt; J Follow-up Messages - Activity Diagram flowchart TD&nbsp; &nbsp; A[Start Follow-Up Message Process: Time Trigger]&nbsp; &nbsp; B[Query user's chat history from Data Storage]&nbsp; &nbsp; C[Receive user's chat history]&nbsp; &nbsp; D[Generate follow-up message using LLM with chat history]&nbsp; &nbsp; E[Send follow-up message to WhatsApp Provider]&nbsp; &nbsp; F[Deliver follow-up message to User]&nbsp; &nbsp; G{Did user reply?}&nbsp; &nbsp; H[Include follow-up as additional context]&nbsp; &nbsp; I[Begin existing user reply process]&nbsp; &nbsp; J[End Process] &nbsp; &nbsp; A --&gt; B&nbsp; &nbsp; B --&gt; C&nbsp; &nbsp; C --&gt; D&nbsp; &nbsp; D --&gt; E&nbsp; &nbsp; E --&gt; F&nbsp; &nbsp; F --&gt; G&nbsp; &nbsp; G -- Yes --&gt; H&nbsp; &nbsp; H --&gt; I&nbsp; &nbsp; G -- No --&gt; J

Maia Sequence & Activity Diagrams

One interesting thing we've noticed is that often, someone with very little online experience actually can have an easier time getting started with an LLM than someone with some technical experience, particularly when the front-end is WhatsApp. A user with no technical experience talks to the LLM as if it were a person, which is a good way to get started. But someone with some technical experience usually talks to the LLM as if it were a Google search, which is a good way to get disappointed. LLM's have a knowledge cutoff date and can hallucinate very specific details like addresses and phone numbers.Given MAIA is the on-ramp to AI for most of our users, we've seen it as our job to teach users the difference, often through experience, akin to how we've developed through experience knowing when something is a ChatGPT question vs. a Perplexity question. But teaching this is hard, and loses us a lot of first-time users. That's why ChatGPT has integrated search, and we've been working for a while to do the same.Seems like it should be easy, but its taken some work, at least for us. The search capability of ChatGPT isn't available when using their models via the API, <a data-mce-href="https://community.openai.com/t/searchgpt-availability-in-api/881067" href="https://community.openai.com/t/searchgpt-availability-in-api/881067" target="_blank" rel="noopener">seems to be for legal reasons</a>. As discussed in that same thread, you can recreate it yourself with function calling, but it seems to me that this not only requires a lot of work but also would give pretty weak performance at reasonable levels of complexity. Google seems to have an out of the box option for <a data-mce-href="https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/ground-gemini" href="https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/ground-gemini">grounding completions by Gemini with google search</a>, built in with their API, but not OpenAI.We decided on giving our microenterprise users the same workflow we use ourselves: GPT 4o + Perplexity. The first attempt at this was to add instructions to the main model to trigger background research by Perplexity if the situation called for it. But this didn't work well- as with other experiments, trying to get our main model prompt to do too many things degrades its performance at the key thing its supposed to do best. So instead, we added a first check of the user message and chat history with GPT 4o mini, its only job is to determine if additional context from Perplexity is needed. The cases where it is needed come from real user chats over the past year where the LLM either can't give the user the information our users want, or in some cases, hallucinated. This is a combination of things that are after the knowledge cutoff date (eg today's spot price for coffee, the largest POS providers in Peru this year), and things too specific to be reliably reterived by an LLM from its training data (eg the address and phone number of a small supplier in a midsized Peruvian town).&nbsp;In most cases, that mini model determines that additional context isn't needed and the user message continues to a response from the main LLM. When additional context is needed, that minimodel generates a self-contained research question, which we then submit to Perplexity via their API, and provide the result as additional context for the responding LLM.Seems simple enough, but its taking a lot of adjusting to get right, and we're still not quite there. This is for two main reasons:1) Compared to adjusting one model prompt that answers user questions, adjusting a prompt in a chain of models is trickier. Things just seem to cascade and amplify when a chain of multiple models are talking to each other, this is probably why agents are so hard to get right. Here you need to get your mini model to trigger in the right situation, and formulate the right kind of question to Perplexity. You need your Perplexity prompt to get the key information for the main model and only that. And you need your main model prompt to use the right information in the right way and ignore the rest.&nbsp;2) The context of Peruvian microenterprises is a small small corner of GPT 4o's training data, but you can prettly reliably limit the models' responses to the relevant corner of its galaxy brain. But, in my experience, this can't be as reliably done for LLMs incorporating realtime info from the web. For example, our main model prompt will do a great job giving advice relevant to a typical small neighborhood restaurant in Peru. But a model responding with grounded search about average 'menu' prices, even when prompted explicitly, can't limit itself to what's relevant for small neighborhood restaurants in Peru. It usually can't get there at all, and is swamped by what makes up 99% of search results about Peruvian restaurant prices: tourist restaurants. This is the case for what seem to be the best of the best- both Perplexity and Gemini Grounded in Google Search. They aren't great at staying relevant for microenterprises in devleoping countries.&nbsp;This is still a work in progress, because its tough to get all these dials tuned just right, but it is improving the user experience in our tests and getting better with each adjustment. If anyone has any other experiences grounding LLM responses in search in a similar context and would like to swap notes, please do reach out to me at <a href="mailto:bailey@hacerperu.pe" data-mce-href="mailto:bailey@hacerperu.pe">bailey@hacerperu.pe</a>Here are the model instructions for those first two steps (our main model prompt can be seen <a data-mce-href="https://mypeasesoria.org/blog/prompt-update-2" href="https://mypeasesoria.org/blog/prompt-update-2" target="_blank" rel="noopener">here</a>, the results from perplexity are added to the context field in [[[]]] there). Mini model evaluating the message:&nbsp; &nbsp;Un LLM va a responder al siguiente mensaje de un microempresario peruano. Pero primero, tu trabajo es determinar si ese LLM necesita información específica o actualizada de Internet para responder al mensaje.&nbsp; &nbsp; La mayoría de los mensajes no necesitan búsqueda web, ya que el LLM es capaz. Pero los casos en que se necesita información específica o actualizada de una búsqueda web son:&nbsp; &nbsp; &nbsp; - Si el usuario necesita una dirección específica, número de teléfono o dirección de sitio web &nbsp; &nbsp; &nbsp; - Si el usuario necesita una empresa en particular para software, insumos, servicios, etc., que podrían estar desactualizados en un LLM &nbsp; &nbsp; &nbsp; - Si el usuario pregunta sobre precios, volúmenes o valores actuales que podrían estar desactualizados en un LLM &nbsp; &nbsp; &nbsp; - Cualquier otra cosa que se beneficie de la información en tiempo real&nbsp; &nbsp; Tu única respuesta debe ser 'LLM' si no se necesita búsqueda web.&nbsp; &nbsp; Si se necesita una búsqueda web, debes responder 'websearch' seguido de una pregunta de investigación en línea clara, concisa y autónoma para obtener la información útil. Esta no es necesariamente la pregunta del usuario; esta es una pregunta para obtener la información actualizada de Internet, que el LLM luego usará para responder la pregunta del usuario. Debes formular esta pregunta de investigación basándote no solo en la pregunta del usuario, sino también en los mensajes anteriores como contexto, incluidos los detalles sobre la actividad comercial del usuario. Esta pregunta de investigación debe contener toda la información para que el motor de búsqueda recopile la información necesaria. Escribe tu pregunta de investigación en español.&nbsp; &nbsp; Aquí está el mensaje del usuario encerrado entre [[[ ]]]: [[[ @contact.message ]]]&nbsp; &nbsp; Aquí está el historial de chat del usuario, un unparsed thread de OpenAI, encerrado entre [[]]: [[ @getthread_response ]]------------------The request to Perplexity: &nbsp; &nbsp; &nbsp;{ &nbsp; &nbsp; &nbsp; &nbsp; "model": "sonar", &nbsp; &nbsp; &nbsp; &nbsp; "messages": [ &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; { &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "role": "system", &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "content": "Eres un motor de búsqueda para otro LLM que responderá a una pregunta de una microempresa peruana. Ese LLM es tu usuario, y necesita contexto actualizado y completo de Internet para responder a la pregunta de la microempresa, por lo que te ha dado esta pregunta de investigación. Responde a esta pregunta de investigación para el LLM y proporciona un contexto útil de manera eficiente, incluye enlaces directamente en el texto. Recuerda que esto es para una microempresa peruana, por lo que las preguntas sobre cosas como proveedores deben considerar empresas que estén basadas en Perú o que indiquen explícitamente en sus sitios web que venden a Perú. Recuerda, el LLM responderá a la pregunta de la microempresa, tú eres solo el motor de búsqueda que le da contexto a ese LLM." &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }, &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; { &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "role": "user", &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "content": "Pregunta de investigación: @contact.message" &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; } &nbsp; &nbsp; &nbsp; &nbsp; ] &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; } &nbsp;

Blog

Evaluating Maia

Designing Better First Interactions

Tinkering and Micro-innovations

Has Maia helped users improve their businesses?

Maia Sequence & Activity Diagrams

Grounding GPT 4o Responses with Perplexity