Impact Investing and Measuring Impact: Why the Industry is Misinforming Itself and Its Supporters (And How to Get it Right)
Bruce Wydick and Nolan Sherwood, University of San Francisco
An online tour of prominent impact investing firms reveals some truly spectacular websites with truly spectacular claims. Beautiful infographics herald transformative results. One fund proudly displays their portfolio’s GIIRS (Global Impact Investing Rating System) rating of 4.8 stars alongside talk of Platinum Ratings and testimonials from grateful beneficiaries. Another presents SRI (Social Return on Investment) calculations claiming a 3.8:1 return on social capital. A pharmaceutical firm claiming social responsibility boasts to have reached 550 million people worldwide with their work, launching their social impact metrics sky high. In every case, each metric was precisely aligned with impact investing best practices.
But buried beneath the sophisticated metrics and compelling narratives is an uncomfortable truth: they presented no concrete evidence of whether any of these millions of individuals were actually better off because of their work. This is because beneath each of these statistics lies a claim made without genuine evidence of impact.
The world of impact investing reports a plethora of statistics that often measure everything except the one thing that matters: whether an intervention has significantly improved people’s lives.
The Only Question That Matters
We are development economists, one a researcher and professor with three decades of experience researching the impact of development programs, the other of us a graduate student training to do the same. And there is one thing that everyone in the economics profession understands today: when it comes to social impact, an intervention’s causal effect is the only thing that matters. Everything else—no matter how sophisticated the metrics, no matter how compelling the story—is window dressing.
The frameworks impact well-intentioned investing firms use to measure “impact” excel at measuring inputs, good intentions, processes, numbers of clients, outputs, sometimes even participant outcomes—the building blocks of impact. But don’t measure impact. In fact they systematically avoid the hardest question and key ingredient to actually understanding impact: What would have happened in the absence of the intervention? This is called the counterfactual, and Impact = Outcome – Counterfactual.
This is the only acceptable way today in which the word impact should be used, especially in the world of impact investing. This isn’t a fine academic nuance or small technical detail—it’s the difference between measuring well-intentioned activity and measuring change. They are not the same thing.
Estimating causal effects is difficult. But estimating the profits of a Fortune 500 firm is also difficult, yet it can be done quite well with the right kind of training. Today the training that economists receive to estimate causal effects is based on the innovations that were the subjects of the 2019 and 2021 Nobel Prizes in Economic Sciences. The former was given for innovations in experimental methods and the latter for the quasi-experimental methods. These are the tools economists now use routinely to understand if Program X is helping people (and if so, how much, and in what ways). And they can be acquired and employed to good use just like modern accounting skills. The problem is that the impact investing world has largely ignored them, perhaps believing they are too hard to learn or implement. Instead it substitutes metrics that measure well-intentioned activity but not impact. The result is a proliferation of impact investing “results” that are like a profit estimate of a Fortune 500 company carried out by somebody without training in accounting—in other words, not dependable.
This isn’t some academic obsession with methodological purity. This is about basic intellectual honesty in a field that claims to be successfully addressing the world’s most pressing problems. We demand honesty and accuracy in our financial accounting, but to this point we don’t demand it in our social impact accounting. Let’s take the metaphor of the Fortune 500 firm a little farther. Imagine one of these massive firms never had to set up an accounting division; they were allowed to focus on simply selling widgets all over the globe. Then investors and the government asked for an audit to assess the firms profits. What an overwhelming task that would seem to the firm, darn near impossible. “We are built to sell widgets, not carry out intricate and costly accounting procedures,” the firm might respond. But in reality it’s not darn near impossible–it’s required–and every firm, no matter what their business, has an accounting division. Social impact organizations need to think the same way: Like profit-seeking firms owe investors and governments an accurate accounting of their profit, social impact firms owe donors and beneficiaries an accurate accounting of their social impact. And estimating the causal effects of social impact interventions is no harder than auditing a Fortune 500 firm. Like accounting, you just need some training to learn how to do it.
Think about it on an individual level: suppose you have been diagnosed with cancer and you were told about a new medicine that had been “used by” 10,000 people, would you take it? Of course not—you’d want to know if the medicine actually worked. You would want your doctor to cite evidence from randomized controlled trials comparing health outcomes for people who were randomly assigned to take the medication compared to a valid control group. Of course you would because your life would depend on it. Yet somehow, when it comes to social interventions, the impact investing world has convinced itself that counting activities equals evidence of impact. An entire industry norm has emerged around the assumption that serving people is the same as helping them.
The False Precision Problem
Perhaps most troubling is how the measurement frameworks used in impact investing create an illusion of precision that masks the uncomfortable fact that these statistics are not measuring impact. When an SROI calculation reports a return of 3.83:1, it suggests a mathematical certainty about social value creation. When a GIIRS rating gives a company 4.5 stars, it implies comprehensive assessment of impact performance. When IRIS+ reports “reach” of 2.3 million people, it sounds like concrete evidence of scaled effectiveness.
This false precision is particularly troublesome because it stops inquiry precisely where it should begin. It fakes out people’s judgement by leading them to a conclusion that appears to have been derived rigorously. Thanks to innovations in econometrics during the last couple of decades, we now understand how to employ rigorous methods to estimate the effects of programs. But this is something the commonly used methods of impact investing definitively do not do. Precision is not the same as accuracy. In fact, it can be an effective smokescreen to mask inaccuracy. Numbers may be calculated correctly by their respective methodologies, but they tell us nothing about the key question: Does this social impact program make people better off than they would have been without it?
Making the Wrong Assumptions
Don’t get us wrong—the impact investing community is a community of good will and has made progress in many areas, including the standardization of outcomes. IRIS+ claims to provide a comprehensive system for measuring and managing impact, offering standardized metrics that allow for comparison across investments and sectors. The Global Impact Investing Rating System and B Impact Assessment create sophisticated scoring systems that evaluate companies across governance, workers, community, environment, and customers. Social Return on Investment analysis attempts to quantify social and environmental value in monetary terms.
These frameworks represent important advances in bringing comparability to measurement. This effort is well-intentioned, and these good intentions are laudable. But outcomes are not impacts. And there is a deeper problem: these frameworks don’t capture causal effects—they assume them. Imagine a world in which accountants could just assume profits were made by companies based on a collection of random production statistics. What firm couldn’t report impressive profits under those conditions?
Consider a typical Social Return on Investment analysis of a microfinance institution. The calculation might assign $500 in social value for each borrower who receives a loan, based on an assumption that microcredit increases household income by that amount. But what if rigorous impact evaluation were to reveal that microcredit has statistically insignificant average effects on household incomes of borrowers? (Which numerous studies across multiple countries have revealed.) The calculation becomes not just wrong, but actively misleading, maybe $500 per borrower misleading.
When impact investing firms report that an education program reached 50,000 students, we implicitly assume those students learned more than they would have otherwise. When they report that a certain health program “reached” 25,000 patients, we assume their resulting health was significantly better than it would have been otherwise. These assumptions might sound reasonable, but they’re often wrong, spectacularly, embarrassingly wrong. Outreach does not equal impact. What impact investing is really doing with these metrics is measuring motion instead of progress, counting inputs, outputs, and processes while communicating nothing about actual impact.
Theories of Change Don’t Always Pan Out
Many impact analyses in the impact investing space likewise rely heavily on theories of change. Theories of change are wonderful tools, but theories of change often fail in practice because in practice interventions often don’t change anything. Providing shoes to needy children sometimes makes them more aid dependent. Computer technologies that are intended to improve learning often do nothing of the sort. Job training programs that should increase employment often crowd out the employment of other workers who are not part of the program. You cannot trust theories of change to provide evidence of impact any more than you can trust a guy with a nice business plan to make long-term profit.
The harsh reality is that good intentions, logical frameworks, and high-quality implementation don’t guarantee positive impact. The world is complicated, human behavior is unpredictable, and false attribution is everywhere. Indeed, it is actually fairly surprising and exciting when today’s high-quality evaluation methods reveal an intervention that really and truly does what it claims. Two of these wonderful surprises are here and here.
The Training Imperative (No Shortcuts Allowed)
Faced with pressure to demonstrate impact, many organizations look for quick fixes. They upgrade their IRIS+ reporting, pursue higher GIIRS ratings, or commission SROI analyses. These efforts create the superficial appearance of ramped-up rigor without addressing the fundamental challenge of offering credible evidence that they are helping people.
Here we need to be blunt: there is no ethical shortcut around the patient work of training staff to understand and implement rigorous impact evaluation. IRIS+, GIIRS, and SROI are valuable tools for standardization and benchmarking, but they are not even remote substitutes for using modern causal methods.
Like learning proper accounting for a Fortune 500 company, carrying out a medical procedure, or designing a building that won’t fall down, measuring the effects of programs requires patient training. It means knowing when randomized controlled trials are appropriate and when they’re not. It means understanding how to read scientific studies, how to design quasi-experimental studies, how to generate counterfactuals to treatment, how to account for selection bias, and perhaps most importantly, how to interpret results in ways that convey truth about impact.
This training can’t be learned in six-week bootcamps any more than specialized skills in accounting, medicine, or engineering can be learned in six-week bootcamps. Firms need to invest in this training, or at least invest in people with this training, and then build in the practices of the discipline into their normal operations.
Building the Infrastructure for Truth
The solution isn’t to completely abandon existing measurement frameworks—it’s to pair them with modern causal analysis, with learning how to measure causal effects. This is exactly what we’re building at the Collaborative for Econometrics and Integrated Development Studies (CEIDS) at the University of Notre Dame in partnership with the University of San Francisco. We’re academics and research-oriented practitioners working with organizations to integrate impact evaluation directly into program operations. Our desire is to create systems that preserve the benefits of standardized measurement while adding the methodological rigor needed to establish causality.
For organizations ready to make this transition, the Social Impact Analytics program at the University of San Francisco provides comprehensive training in both standard measurement frameworks and rigorous causal analysis. The 11-month program trains organizations’ own staff in modern causal impact evaluation methods and how to implement systems allowing impact investors to (genuinely) measure their impacts. This is one of the best investments impact investors can make–training staff so that they can begin to build impact evaluation methods into an impact investing firm’s routine operations.
Impact Investing and Measuring Genuine Causal Impacts
There is an astronomical cost to poor impact evaluation: Every high GIIRS rating based on assumed impact represents resources that could have been directed toward demonstrated impact. Every IRIS+ report that counts activities without establishing impact is a missed opportunity to learn what actually works.
In a world of unlimited need and limited resources, especially today when the aid sector is being starved of funding, impact investing cannot afford measurement systems that substitute assumptions for evidence. The people impact investors want to help deserve better than sophisticated measurement theater. They deserve interventions that actually work.
The impact investing industry has reached an inflection point. It can continue refining measurement frameworks that document good intentions while failing to report actual impact. Or it can choose the harder but more honest path of pairing its standardized metrics with the modern impact evaluation methods that have become widely adopted outside the impact investing community, but not within.
The impact evaluation tools exist. The training is available. The resources are there. The only question is whether the industry has the will to invest and hold organizations accountable to valid impact measurement. Because at the end of the day, no amount of superficial sophistication can substitute for the simple question: how much have we helped people?
Bruce Wydick is Professor of Economics and International Studies at the University of San Francisco, Adjunct Professor at the University of California at Davis, and Distinguished Research Affiliate with the University of Notre Dame. Nolan Sherwood is a graduate student in International and Development Economics and graduate assistant for the program in Social Impact Analytics at the University of San Francisco. You can reach either of them at sia@usfca.edu.