Blog

Benchmarking LLM's has evolved from (increasingly gamed) single leaderbords to more useful task-specific benchmarks. One new super-interesting one is Vending Bench by Andon Labs, where they compare different models' ability to independently and profitably run a vending machine.Their goal (testing agentic capabilities) is pretty different from what we are up to at maia, but their results are fascinating and are really useful for our efforts in a number of ways that we'll write about in the future. But two immediate top-line take-aways:1) Similar to the findings in Otis et al (discussed <a data-mce-href="https://mypeasesoria.org/blog/kenya-ai-mentor--otis-et-al--revised" href="https://mypeasesoria.org/blog/kenya-ai-mentor--otis-et-al--revised" target="_blank" rel="noopener">here</a>), AI for MSMEs in developing countries isn't as simple as letting ChatGPT take the wheel. If you just put an LLM in charge of a vending machine, Vend shows most actually go out of business, and this is a much more straightforward business than those of our users. LLMs are amazing and can massively improve MSME productivity, but their capabilities are a '<a data-mce-href="https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the-jagged" href="https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the-jagged" target="_blank" rel="noopener">jagged frontier</a>': they succeed at some very hard tasks but can also fail at some very simple tasks. Teaching MSMEs about that jagged frontier and bulilding scaffolding around tools like Maia to adjust for it is a core part of the work.2) Behind most buzzy AI hype areas is a kernel of truth, and that's true with 'context engineering'. Figuring out what information to put into an LLM significantly impacts the value of what comes out. If you look at how LLMs fail in running a vending machine, they succeed steadily for a while, but then enter a sharp death loop where their results crater, and the Lab's paper shows this isn't because they are hitting the technical context limits for the models. Often well within those limits, an LLM can just go off the rails, as anyone who has used an LLM for programming and debugging can attest, and recovering performance reuqires a context refresh. This is one of the reasons why we are hard at work rolling out our new backend for Maia which features quite a few evolutions in managing chat history, to ensure key buisness context is used to customize advice, but the bloat of what can be long multi-turn conversations doesn't degrade performance.Also really interesting stuff there about what business tasks LLMs perform better than others, but we'll leave that topic for another day. If you are interested in learning more, Anthropic did a really entertaining post about their Vend test of Claud in their office, which you can find <a data-mce-href="https://www.anthropic.com/research/project-vend-1" href="https://www.anthropic.com/research/project-vend-1" target="_blank" rel="noopener">here</a> and you can find Andon Lab's live Vending Bench results <a data-mce-href="https://andonlabs.com/evals/vending-bench" href="https://andonlabs.com/evals/vending-bench" target="_blank" rel="noopener">here</a>.

Can an LLM run a Vending Machine?

Maia doesn't just sit back and wait for questions. Every week we send out a sector-specific 'tip of the week' to expose users to a broader set of topics, drawn from leading entrepreneurial training programs and manuals. Now that MAIA users number in the thousands, we can look at the data to see what makes for a useful tip via WhatsApp. Ideally this would be based on business impact, our north star, but that data is only starting to fill in. Yet it is pretty clear from the chat data if a user 1) doesn't even read a tip, 2) reads it but doesn't ask any follow-up questions, or 3) engages in a conversation to learn more about the topic and how it applies to their situation. That is the outcome we use here.Our Data-Driven Rules for High-Impact Tips After analyzing thousands of user interactions across more than a hundred tip-week pairs, we've distilled our findings into a few key principles.1. Always End with a Question. This is our strongest result, and it’s not even close. The single biggest factor in driving replies is ending the message with a direct, open-ended question. It turns a monologue into a dialogue.The Impact: Tips that end with an invitation like "Want help with this? Ask me!" see up to 6 times more follow on topic engagement than tips that simply offer advice without a question. 2. Solve Today's Problem, Not Next Year's. Specificity is key. Tips that focus on a concrete, immediate challenge consistently perform better than those about broad, long-term goals.The Impact: Tips focused on a challenge that can be tackled this week (like "how to price against a new competitor") consistently double the rate that entrepreneurs will engage with the topic. In contrast, tips on broader abstract topics (like "how to build a brand") are more often ignored. But broader topics like branding are important and can't be ignored! The challenge is to break them into steps that are more concrete and achievable, to help entrepreneurs build progress. 3. Explain How You Can Help. We tested tips that explicitly stated, "Maia can help you by..." followed by a list of how Maia can not just give advice, but also be a tool in implementing the solution.The Impact: early results here were actually not that impressive, and near baseline. But mixing a content and form test together can muddy the waters, so we're re-testing to better isolate the impact of this form change with a proper A/B test, which we can now more easily do thanks to user volume and improvements to our whole system and data pipeline. 4. Make It Easy on the Eyes. While harder to quantify with a single number, the visual trend is clear. Our best-performing tips are concise, break up text into small paragraphs, and use 1-2 relevant emojis (like 🧐 or 📈) to grab attention, but no more. Long walls of text get ignored, and users don't want an overload of emojis and exclamation points. In other words: more Millenial less Gen Z.&nbsp; 5. Make it Easy on the Hands. Quick reply buttons are the single best tool to make it easy for users to start exploring a topic, and kick off in-depth conversations. This was such a clear boost that we implemented it way back, even before setting up the pipeline used in our current data-driven analysis.&nbsp; We'll keep updating on new results and experiment findings, so that everyone can improve the quality and usefulness of their WhatsApp-based tips.&nbsp;

What makes for a useful tip over WhatsApp?

<h3>Why evaluation matters for MAIA </h3>Every week, hundreds of small business owners use Maia to improve their business. And Maia is not a finished tool, we are constantly tinkering and experimenting to make it better. To keep improving—and to reassure partners and users that the tool is both safe and effective—we need a systematic way to test whether MAIA is doing what we think it’s doing. Over the past month have posted information about our testing &amp; evaluation framework, take-up patterns, and experiment results.A timely reference point to tie all this together is the “AI Evaluation Framework for the Development Sector” published by the Center for Global Development (CGD) and written by leaders from the Agency Fund. The authors outline four incremental levels of evaluation—from technical correctness all the way to real-world impact. Importantly, this framwork adds sorely needed nuance to the previous model in development of simply 'intervention+RCT', and highlights asking the right questions at the right stage.Below, we map MAIA’s current (and planned) evaluation activities to each level.<h3>Level 1 - Model Evaluation</h3>Does Maia behave the way we want it to? If the tool can't accurately give users information about their tax deadline or best practices in sales and marketing, there is no need to wait for an RCT. Moreover, we need to make frequent decisions about updating the prompt, or updating to new LLMs as they are released. This is why we use <a data-mce-href="https://mypeasesoria.org/blog/testing---evaluation-framework" href="https://mypeasesoria.org/blog/testing---evaluation-framework" target="_blank" rel="noopener">an extensive testing and evaluation framework</a>, which is partially automated but also uses some subjective response rating, to evaluate responses on a regular sechedule as well as with any major update.&nbsp;<h3>Level 2 - Product Evaluation</h3>Are users engaging with the tool and solving real-world problems? One important feature of our T&amp;E process is manual review. Bailey dedicates up to 2 hours every day manually reviewing chat logs to evaluate all aspects of the product, identify bugs and improvements, etc. In addition, <a data-mce-href="https://mypeasesoria.org/blog/usage-patterns-update" href="https://mypeasesoria.org/blog/usage-patterns-update">we closely monitor usage patterns</a> (Monthly Active Users and Weekly Active Users), and use these as <a data-mce-href="https://mypeasesoria.org/blog/designing-better-first-interactions" href="https://mypeasesoria.org/blog/designing-better-first-interactions" target="_blank" rel="noopener">early indicators for product improvements </a>to iterate fast.<h3>Level 3 - User Evaluation</h3>First, we directly ask users if the tool is helping them and their business, <a data-mce-href="https://mypeasesoria.org/blog/has-maia-helped-users-improve-their-businesses" href="https://mypeasesoria.org/blog/has-maia-helped-users-improve-their-businesses">see results here</a>. We are also working in partnership with the ILO to test some cognitive outcomes among small-scale coffee farmers using the tool, specifically surveying their knowlege of specific best practices for production, worker safety, and formalization. Finally, with our new and improved follow-up messages we will be able to measure how often users self-report implementing their action plans from the previous week, and study why is that higher for some users than others.<h3>Level 4 - Independent Impact Evaluation</h3>Our ultimate goal is for Maia to directly and causally improve business outcomes for users, so that is what we want to measure. I have published a quasi-experimental impact evaluation (regression discontinuity) of entrepreneurial training programs and collaborated extensively with organizations like JPAL and IPA on other work. Our goal is to get to as good of a product that we can using Levels 1-3, as well as to the nescessary user volumes, and then collaborate on RCTs of the tool. We seek to follow in the footsteps of Otis et al and go beyond the simple up-front question 'what is the impact of this AI tool' and further unpack the more complicated question of what types of AI assistance are useful (or harmful) for what kinds of entrepreneurs facing what situations.

Evaluating Maia

A core goal of Maia is to serve as an on-ramp to AI for traditional micro- and small-business owners in emerging markets. One barrier we encountered is well known in HCI literature: blank-page syndrome. When the first thing users see in an AI tool is an empty text box—“Cuéntame tu problema”—many fall off the onboarding process. But immediately directing users to specific use cases can hide one of AI's key benefits over traditional e-learning tools (it range) and will also cause new user fall-off if the specific examples are irrelevant for them.Weekly Active Users as a % of Cohort<figure id="figure-xrjbpce92m9zs2ym2"><img id="xrjbpce92m9zs2ym2" src="https://cdn.durable.co/blocks/1dvSOMDiF9DegNvKpAHdpfkFYe7D2dYeTOz9mLCgiy7cj0VvlpP9JsEvJwQRgIcw.png" alt="" title="" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/1dvSOMDiF9DegNvKpAHdpfkFYe7D2dYeTOz9mLCgiy7cj0VvlpP9JsEvJwQRgIcw.png" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure>One approach we tried was to deliver specific use cases gradually, the first and second week after a user first connected. The idea was to have a very open interaction first, to experience Maia's versatility, and then give more specific examples of how other owners are using it. Seemed like a good idea, but one week was just too late, with too many users dropping off up-front from blank-page syndrome. Even moving the examples to just 23 hours after their first connection was too late. If a busy business owner doesn't see value right away, they are out.The other approach is to give the examples up-front. We first did this with an optional button (press 💡 if you'd like some examples of how other entrepreneurs have used Maia), but even the small friction of the button reduced use, so we modified the flow to quickly give example questions to everyone (eg 'Send a photo of your menu or your signature dish, and I'll give you ideas to make them look irresistible' for Maia restaurants, 'how can I improve my presence on social media' for artisans).&nbsp;The winning approach is starting more specific and then exposing the users to the broad range over time, rather than starting broad and teaching specfics over time. Quick business wins with minimal friction are essential, and because MSMEs share common needs, we can offer use cases that are both specific and broadly relevant.

Designing Better First Interactions

In their book ‘Abundance’, Thompson and Klein discuss how innovation often isn't a single 'eureka!' moment, but rather a patient process of tinkering and accumulating many 'micro-inventions'. This really resonates with our journey building Maia, and I wanted to share one recent example: the evolution of our weekly follow-up messages.Initially, Maia functioned like an <a data-mce-href="https://mypeasesoria.org/blog/beyond--chat-with-the-manual--ii" href="https://mypeasesoria.org/blog/beyond--chat-with-the-manual--ii" target="_blank" rel="noopener">‘ask an expert’, ‘chat with the manual’ type tool</a>, enhanced by proactive 'Tips of the Week' introducing best practices. But it was clear that users needed more to bridge the gap between knowing and doing. So the next step was adding a Monday morning follow-up feature targeting users who, guided by Maia, had defined a specific action plan during the previous week. Maia would remind them of their plan and ask if they'd implemented it. This served as a simple behavioral nudge – a prompt designed to increase follow-through and reduce the common intention-action gap faced by busy entrepreneurs.However, analyzing chat logs and talking with users revealed that many engaged with a topic but didn't quite reach the action plan stage – demands of entrepreneurial life often intervened! To support that group, we introduced a second follow-up version. This one revisited the topic discussed the previous week and prompted users to take the next step to define a concrete action now.This second version proved to be a significant unlock. It reached a broader group of entrepreneurs beyond the initially highly motivated ones who finalized action plans immediately. Many were intrigued by a Tip of the Week, started a conversation, but got pulled away. This gentle Monday nudge successfully guided a large portion of this group towards creating an action plan, effectively widening Maia's reach and impact – <a data-mce-href="https://mypeasesoria.org/blog/worry-about-external-validity" href="https://mypeasesoria.org/blog/worry-about-external-validity" target="_blank" rel="noopener">a key challenge for many digital tools aiming for broad external validity</a>.But the tinkering doesn't stop there. During a recent review of these follow-ups, we saw an opportunity to move beyond simple check-ins towards fostering deeper reflective practice. Encouraging users to reflect on their actions – what worked, what didn’t, and why – is crucial for building agency, problem-solving skills, and ultimately achieving greater impact than just skill transmission alone. <a data-mce-href="https://voxdev.org/voxdevlit/training-entrepreneurs" href="https://voxdev.org/voxdevlit/training-entrepreneurs" target="_blank" rel="noopener">Impact evaluations of entrepreneurial training</a> shows this clearly. This aligns with concepts like Kolb's experiential learning cycle, where reflection transforms experience into learning.Interestingly, I used the latest frontier models (GPT 4.5 and Gemini 2.5) as thought partners during this refinement process, which was incredibly insightful compared to what the frontier models of 6-12 months ago brough to the table. Together, we redesigned the action plan follow-up to specifically prompt this reflection. Instead of just asking if the action was done, the message now asks how it went, using Quick Reply buttons like Sí, pero con dificultades ("Yes, but with problems") and Aún no, mejoremos plan ("Not yet, let's improve the plan").Early results are promising! We're already seeing users engage more thoughtfully when they encounter obstacles, using these prompts to analyze challenges and adjust their strategies with Maia's help. It’s a small change, but it fosters resilience and encourages users to engage in metacognition – thinking about their own actions and learning processes. This whole evolution is a nice example of how innovating with AI to help microenterprise owners in developing countries doesn’t come from a single idea, and instead is build through persistent tinkering, learning from users, and accumulating many 'micro-innovations'.

Tinkering and Micro-innovations

This week we asked some users in Peru "Has MAIA helped you improve your business?" Here are the responses:<figure id="figure-np07lix5bm85zcl87"><img id="np07lix5bm85zcl87" src="https://cdn.durable.co/covers/27ohYz9tQJ3cBACTW38qxxR3ejlvq0wqD49dNe27YAGSO7vsIYpTlShi18MlNcTl.jpg" alt="" title="" class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: center center;" data-corners="default" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/covers/27ohYz9tQJ3cBACTW38qxxR3ejlvq0wqD49dNe27YAGSO7vsIYpTlShi18MlNcTl.jpg" data-mce-style="aspect-ratio: 16/9; object-position: center center;"></figure>We're very happy with these results! In marketing terms we could translate that into an approximate Net Promoter Score of 74, which is excellent. Though I was surprised to learn that NPS ignores non-responses, and to me the response rate seems very important. Our response rate on the above question was 46%. This isn't our field, but according to ChatGPT, response rates for customer satisfaction surveys are usually 10-30% via email and 20-40% in-app. So our 46% response rate is telling. Double win!What we hope to do are financial impact measurements from a survey as part of a third-party RCT, to get to a robust dollar value of impact. We will pursue that in a future phase of this project, after our current round of A/B testing and tool refinements. That plan has been complicated by the unfortunate closing of USAID's DIV, one of our main funders, but we soldier on.We really want to thank our users for trusting us and using our tool. These entrepreneurs in Peru are facing major challenges, particularly relating to crime and input costs, yet continue to work and thrive, they are an inspiration. We're humbled to be part of their team.

Blog

Can an LLM run a Vending Machine?

What makes for a useful tip over WhatsApp?

Evaluating Maia

Designing Better First Interactions

Tinkering and Micro-innovations

Has Maia helped users improve their businesses?