Kenya AI Mentor (Otis et al) Revised

The Oits et al paper, which offered the very first and best evidence about if and how AI training could work for MSMEs, has a new version which you can find here. If you have read the first version, do take a look, because the revisions are significant. There is additional work unpacking user converstaions which is quite useful, and the interpretation of the results has evolved significantly. The main conclusion of the paper is:

"Our findings show that generative AI-based tools can serve as an efficient source of personalized feedback for many entrepreneurs in low- and middle-income countries. In fact, the treatment effect we find for high performers is similar to the 10% to 15% performance improvements driven by human-to-human training programs (Brooks, Donovan, and Johnson, 2018; McKenzie, 2021), but are likely delivered at a fraction of the cost."

In other words, strong evidence that this can work- we can deliver at least similar (and potentailly greater) productivity impacts compared to traditional training, at 1/100th the cost, to a much greater proportion of developing country MSMEs. All we have to do is just hook them all up to ChatGPT and watch the impact roll in, right? Well, its not that simple, because the authors go on:

"That being said, we also find evidence that generative AI can widen the gap between low- and high-performing businesses in these contexts. More broadly, our findings are consistent with the idea that generative AI has the potential—if designed and deployed with care—to benefit the billions of people and millions of firms in developing countries."

The Otis et al AI Mentor was desgined in a specific way, you can see all the details in the paper. It generates a set of potential solutions to any particular problem, then offers greater detail on any one the user wants to investigate further. In some ways, its a bit like a brainstorming engine. Their results show that previously well-performing MSMEs improved their performance with this brainstorming engine, but previously poorly-performing MSMEs actually did slightly worse after using it. And it makes sense that high performers will be more able to filter ideas from this engine, use the tool to refine them, and do better, while poor performers may lack the judgement to select the best ideas. Interestingly, there is forthcoming work by the lead author Nick Otis using an AI mentor on a similar population with a different design and functioning, and the results are actually quite different.

What does this tell us? It actually matters quite a bit how the AI tool used by MSMEs is designed- how the model is instructed, what resources it uses, how the user experience is structured. Some designs seem to work very well for some entrepreneurs, but not well for others, and they even have the potential to harm.

So, its not quite a slam dunk here, there is quite a bit of work to left to do, to harness the potential and avoid the harms. These tools have to be designed and deployed with care. That is the goal of this project, to evlauate a variety of potential designs. Luckily, we don't have to start from scratch- there is actually a wealth of ideas and results in the deacdes of entrepreneurial training research to draw on. And, we can also directly use the results from Otis et al's paper, and their and others' future work, to refine tool designs to maximize impact and minimize harm. You'll see this directly the next time we post the model instructions. That shows the value of making this process open and collaborative rather than closed and proprietary.