Photo by Patricia Serna on Unsplash
What is it?
- NBER Working paper. Working papers from the National Bureau of Economic Research are hugely influential and widely quoted. They’re often released before peer review, like this one, to boost impact.
- Two experiments. A randomized trial with 163 social enterprises in India, and a survey experiment with 223 early-stage investors.
- Finding. Firms that got hands-on help measuring impact grew revenue and raised capital at roughly twice the rate of controls. In a companion experiment, investors responded strongly to quantified impact metrics and not significantly to generic impact language.
Should you care?
Yes, if you are considering impact measurement, this evidence that it can make a difference, with speculative guidance why and how to do.it.
The experiments
The treatment for the randomized control trial was a hands-on impact measurement support (framework, indicators, stakeholder surveys and a personalized impact report) to half of 163 social enterprise with access to an impact measurement course throufgh India’s largest incubator of social enterprises. Treated firms grew revenue by about USD 20,000 more than controls and raised about USD 43,000 more in outside capital. The effect shows up in the median firm, not only in a few outliers. The sample is self-selected, since these firms chose to join a measurement program, and the outcome window runs from a FY2021-22 baseline to a 2023 endline. There was also a complementary experiment, whereby investors rated 15 hypothetical startup profiles on a 1 to 7 interest scale. Adding quantitative customer surplus data moved interest scores up or down by about 0.4 points while qualitative impact statements had no statistically significant effect.
Why impact measurement works
The paper proposes two mechanisms for these effects. Firstly, applying measurement approaches prompts internal learning, and forces firms to figure out which stakeholders benefit from its product, and which customers get the most value, and this can also overlap with customer discovery. Secondly, quantified impact reporting boosts external credibility in the marketplace, and with investors in particular.
Open questions
- Are the mechanisms valid? The authors pose the two mechanisms above on the basis of theory alone, there is no empirical work like interviews or process data to validate these mechanisms or suggest others that might be at play.
- Is there respondent bias? All outcome data is survey based, and the authors do not consider whether firms might be incentivized to over-report outcomes, or how that might be managed analytically.
- Does measurement of growth miss the point? Stakeholder surveys were collected at endline, but were too noisy to use. So the study measures growth and capital, not social impact. For many working in this field, that is the most important question, and would definitely need some quantitative data to evaluate.
What it might mean for using data and AI in impact measurement
These experiments suggest and create incentives to monitor and quantify impact. This is familiar to the development and civic technology, which has a bumpy history of indicators becoming targets, and prioritizing what can be counted. What this research reinforces, is that dangerously slippery tendency to ignore how quantification hides strategic design choices, both in terms of programmatic work on social impact, and analytical explication in this working paper itself. Quantitative impact is clean and often convincing, but relies on a ton of modelling choices that are determinate but invisible.
This problem gets worse with the use of generative AI in program design, monitoring and reporting for impact. Vague and exuberant expectations about efficiency are driving the uptake of AI tools across the social impact sector, without corresponding investment in the capacities needed to design defensible workflows that will inform how they are used. Using AI tools to monitor impact likely needs some of the same quality assurance measures as does research more generally (see my previous post) but these findings also drive home the point that to the casual investor/supporter/donor, safeguards might matter less immediately than numerical clarity.

When thinking about how to use AI to demonstrate transparency and accountability impacts, the lesson might be about conceptualising these two mechanisms as a kind of Maslovian iceberg. Headline quantiative impact data is clean and compelling, but only responsible and sustainable if it rests on a solid and often hidden foundation of that first mechanism of learning, whereby impact measurement frameworks, data and processes are developed in close integration with strategy and operations. For AI tools, this means identifyng how tools and bespoke workflows interact project design and implementation, where substantive human expertise guides and safeguards automation, and how this is all recorded for declarations that funders might never see beneath the shiny headline numbers.

