Backup models upgraded early, national events acknowledged, and new checks on coaching replies
We moved MAIA's onboarding and backup models off a model Google retires on November 19, with two small regressions we accepted and one we are still fixing. Proactive messages now acknowledge major events in a country, after our Panama messages missed the earthquake. And we defined three new problems a coaching reply should avoid, with guidance and review tools to find them.
- 204 real production failures
- 302 sampled real conversations
- 34 derived from real conversations
- 74 hand-written edge cases
Highlights
Proactive messages now acknowledge major national events
Panama had a 7.7 earthquake on October 9. Our weekly coaching messages to Panama went out three days later with no awareness of it, and six owners wrote back about the quake. There is now a short per-country note for major events, drafted in our daily conversation review and approved by a person before it goes live. While it is live, the weekly coaching message, the 23-hour check-in and SABY's weekly message all read it. For the first two days MAIA asks how the owner and the business are doing and offers help. After that it acknowledges the event in one line and still offers a small business step, without pushing sales or promotions that would clash with the moment. The note expires on its own after five days.
Onboarding and backup models upgraded, five weeks before retirement
Google is retiring Gemini 3.6 Flash on November 19. MAIA uses it to onboard new users and as the backup for several jobs if our primary provider (OpenAI) goes down. We moved all of those roles to Gemini 3.8 Flash this week rather than waiting, so there is time to catch problems before the deadline. The backup router now runs with higher reasoning, which makes it slower (about 13 to 19 seconds per call against 3 to 8 for the primary). That is acceptable because it only runs during an outage. Cost was not a factor in the choice.
Three new checks on what makes a good coaching reply
From reading real conversations we defined three problems to stamp out: a reply that gives useful information but leaves the owner without a clear next step; building a plan on top of a price without first checking it against what similar products sell for locally; and repeating back what was just said. The coaching guidance now covers all three. We also built two review tools. A screener points our reviewers to suspect replies with a quoted reason; a person makes the call, because on unseen cases it is not yet reliable enough to decide alone. A fact checker tests domain claims (doses, rules, figures) against our sector knowledge base and official sources.
Fixes and improvements
New, in testing
- Pulso, a tool for public officials to ask questions about what small businesses are living through, answered from de-identified MAIA conversations. Our privacy policy allows research use of anonymised data only, so Pulso never touches anything else, even internally. This week we built the de-identification pipeline and ran it on Peru's full history (790,000 messages from 7,700 owners). A pattern pass masks phones, ID numbers, emails, links and addresses, then GPT-5.4 mini and two independent Claude Haiku passes find person and business names, which are masked across all of that owner's messages. On a 5,659-message test set the final version leaked no person names. An independent Claude Opus audit of 3,000 random rows still found a leftover fragment in 0.6% of messages, mostly a single mention of a shop name that all three passes missed. Across the corpus that would be roughly 4,700 messages, which is why the app adds a second line of defence: officials never see conversations, quotes are capped at 25 words, and any count under 10 is withheld. The app is being tested internally and is not open to anyone yet.
Coaching quality
- When MAIA takes a second look and changes a provisional answer, it now presents the result as updated guidance instead of apologising for a mistake it did not make. On 64 replies across 8 test cases where MAIA was not at fault, false "I was wrong" replies fell from 8 to 3, all 3 on one case that was also 3 of 8 before. On the separate test set where MAIA really was wrong, it still owns the error.
- A new user who opens with a photo (for example "write a paragraph about this image") now gets an answer about the photo. Before, onboarding could not see images and asked them to resend it, and at least one owner left. About 1 in 100 new users open this way.
- The arithmetic check that runs on every reply now reads Spanish number formats correctly (30 × 0,124 = 3,7 and $5.150.000), removing false alarms on correct math.
Staying focused on the business
- MAIA had been making some purely personal pictures on request. Requests like that are now declined up front, while business images (product shots, flyers, catalog photos) are unaffected. Replayed on 8 real incident cases: 7 declined, up from 1. No change on 85 business and borderline tests.
- The short note MAIA adds when someone is mostly using it for personal tasks now fires less often and follows the same rule in text replies as in image replies.
Opportunity Radar
- Radar links that had expired or were close to expiring are renewed before they are sent, so an owner never taps a dead link.
- Published a blog post on how the Radar works and what it has taught us: The Opportunity Radar.
Reliability and data quality
- A tap on "Otra idea" (another idea) under the weekly coaching message was being recorded as the owner dropping their goal: 10 of 13 such records in one day came seconds after that tap. Nothing users see read that record, and it is no longer written for these owners.
- Our hourly health check now tells apart a single error that our WhatsApp provider automatically retries from a real failure, so alerts point at things that need attention. One such error on October 14 was traced end to end; no messages were lost.
- The record that a personal-use note was sent is now saved together with the message itself, so a failed send can no longer be counted as delivered.
Evaluation
- New regression tests from this week's conversation reviews: revised advice after a second look, a coffee dosing pushback, numeric choices, and personal image requests.
- Evaluation runs now record token usage, and the backup-router tests replay the exact request production sends, so a passing test means the same thing in production.
Known issues we are still working on
- If OpenAI goes down, the backup model that writes SABY's 23-hour check-ins now skips about 45% of them, against about 12% for the primary and about 17% for the old backup. Gemini 3.8 and Claude Haiku both read the skip rules more literally than the models the prompt was written for. A prompt fix is being prepared and will be re-tested before release.
- When an owner states a unit cost that is clearly implausible, MAIA can still build a plan on it instead of questioning it. This week's guidance on checking prices did not fix that case. It stays in the regression suite as a known failure while we test a check in code rather than in the prompt.
- In a real Panama conversation from July, the owner answered "Sí" seven times and MAIA repeated nearly the same list about five times. That case still fails on the current guidance and stays in the suite.
- The weekly coaching messages and 23-hour check-ins launched on October 6, and the memory behind them, are checked by replaying live samples. They do not yet have a fixed regression set like replies do.
Also this week
- Tested, not adopted
- Claude Haiku 5.5 and GPT-6 luna for onboarding (they launched correctly only 50% and 83% of the time, against about 90% for Gemini 3.8), and Claude Haiku as the backup router (it sent too many questions it should have answered to the stronger model).
- Rolled back
- Nothing this week.
- Experiments
- No A/B experiments in the field this week. The last two ended on October 4; the descriptive close-out of the second (an Opportunity Radar option at onboarding) is due October 18.
- Coming up
- Gemini 3.6 Flash will be removed from our model gateway before November 19, once traffic confirms nothing still calls it.
Model and prompt changes
Changes to the AI models MAIA uses or to the guidance they follow. Primary models unchanged: OpenAI GPT-6 luna and GPT-5.6 luna for routing and coaching replies, and GPT-6 sol for the hardest questions.
| Date | Change | Where | Evidence | Rollback |
|---|---|---|---|---|
| Oct 13 | Coaching guidance: clear next step, check prices against the local market, no repeating back | PeruPanamaColombiaBrazilOther countries | 451 replies per version, 0 consistent changes | Rebuild prompts from the previous version |
| Oct 13 | Router: personal picture requests declined up front | All countries | 8 real incidents: 7 declined (was 1); 0 changes on 85 business tests | Redeploy previous worker |
| Oct 14 | Gemini 3.6 Flash to Gemini 3.8 Flash for onboarding and all backup roles; backup router at higher reasoning | All countries | Per-role replays; forced failover passed; two small regressions and one known issue (see above) | Redeploy previous versions of the 8 services |
| Oct 16 | Coaching guidance: revised answers framed as updated guidance | PeruPanamaColombiaBrazilOther countries | False apologies 8 to 3 of 64; real errors still owned | Rebuild prompts from the previous version |
Service: No outages. One transient error traced, nothing lost.