Worry about external validity

The biggest problem we face in applying emerging academic research on AI advisory tools for MSMEs to the real world is the external validity of those results. Let me explain with an example, the best paper out there so far:

"We recruited entrepreneurs into our study over the Meta ad platform in partnership with the Busara Center, a Kenyan research organization, starting in May 2023. Figure 2 and Appendix C provide additional details on the experiment timeline and sample recruitment process. Our recruitment strategy involved running ads on Facebook to invite entrepreneurs to a short paid survey (Figure A1). All entrepreneurs who responded to our ads were required to pass basic attention checks and take part in three rounds of pre-treatment surveys delivered over the course of three months, which helped us reduce post-treatment attrition and ensure valid causal inference. Appendix E describes our surveys, which asked about firm profits, revenues, and management practices, among other measures. Our final sample includes 640 Kenyan entrepreneurs who completed all three pre-treatment surveys. The median entrepreneur in this sample was 26 years old, had been running their business for one year, and held a college degree (Table A1). " Otis et al p7.

This approach makes complete sense to maximize the power of the study and minimize attrition. But if we are seeking to apply the results to all Kenyan entrepreneurs, we need to ask ourselves how representative are those who would see and respond to a facebook ad of those MSME entrepreneurs and pass all those steps? Might they differ in a way that would be relevant for the impact of intervention?

Facebook penetration is high in that country, as it is in Peru, but the number of registered Facebook accounts per capita doesn't necessarily translate into a high proportion of MSME owners as frequent users who would click on a facebook ad. Moreover, not only did they have to click on the ad, they then had to pass all of the attention checks and complete the pre-treatment surveys.

It seems quite possible that this would result in a set of participants that are younger, more technologically inclined, and more motivated and engaged in terms of looking for ways to improve their business. Indeed the paper describes the median participant in that study as a 26 year old college graduate, with a higher proportion of technology-related businesses than one would expect for Kenyan MSMEs as a whole. And you can compare the usage rates of the tool in the paper to what you would expect 'in the wild' (from an average set of owners who weren't recruited and offered rewards for participation). If you think that the impact of an AI advisor on business performance might be different for this group in some important ways, you need to be very careful in applying those results to the broader population.

The average bodega owner in Peru is a mid 40s mother with no post-secondary education. In evaluating distribution channels in the country we met with a local digial marketing expert, and they indicated that facebook ads here could only reach a small and particular subset of bodega owners who are on average more educated, younger, and running more sophisticated operations. When I pressed them for a percentage of the universe of bodega owners that would be reached by Facebook ads, she said 3-5%, despite the high penetration rates of Facebook here.

To repeat, this was not an error of the Otis study, their approach made perfect sense for that exercise. And because theirs the only real evaluation of these tools to date, we have paid very very close attention to it and incorporated its results explicitly in our current model prompt. However, it is an open question what the impact of these tools would be on a more representative set of MSME owners. And to figure that out, we need to get to more representative samples than are possible with online advertising. We are trying to do this do this by, for example, providing the tool to all bodegas in particular areas through partnerships with universal suppliers, or our own direct on-the-ground promoters. Its quite likely that usage patterns and tool impact will look quite different for this universe of MSME owners.