One way to assess attribution is to have a “control group,” which in the Freedom Fund’s case would be a group of vulnerable community members not being served by our programs, but similar in all other respects to the community being served. If the situation of the group being served improved more than the control group, that would be a strong indicator of the positive impact of the program.*
Rigorous prevalence studies are expensive and time-consuming, so we also measure activities that are strongly tied to a reduction in vulnerability to modern slavery—e.g., the number of at-risk children helped to return to school, the number of people provided with access to social and legal services, the number of micro-enterprises started, and the number of legal cases supported. More broadly, we also look at overall lives impacted by Freedom Fund–supported programs to understand our reach in comparison to funds spent.
The Freedom Fund’s method of measuring impact is appropriate for our size, approach, and resources, but it won’t be a fit for every organization or issue area. We have found success in tracking both outputs and outcomes over time. We’ve found that different audiences tend to seize on specific indicators that they deem important, and it has taken some effort to educate donors about why effective prevention efforts should eventually result in lower numbers of people liberated from slavery (because there will be fewer people left to liberate). But with a range of metrics that indicate lower levels of vulnerability to exploitation, complemented by long-term prevalence data, we are able to paint a much richer and more complex picture of the Freedom Fund’s impact.
Think tanks face a very different challenge. Their difficulty in measuring impact often leads them to measure what they can count, not what matters. Think tanks will invariably count how many reports or opinion pieces they have published, or how many conferences they have convened. They may track how often their experts appear in the media, and how much traffic their website receives. These are all straightforward to count. And they are certainly a way to track productivity and profile. But they don’t really capture impact.
This is because think tanks rarely see their primary purpose as publishing reports. Rather, their purpose is usually defined as generating ideas and evidence and influencing policymakers. As Ken Roth, the former CEO of global human rights organization Human Rights Watch, put it, his priority was “[e]nsuring that our programmatic work is always impact-oriented—that publications are not ends in themselves but tools to change governmental behavior.”14
Counting reports or conferences won’t tell you that. It won’t tell you if those reports informed policy change or if they were even read. Measuring influence can be particularly challenging for think tanks and advocacy organizations because there are no direct measures. Policymakers rarely come out and say, “We took X action because we were persuaded by Y’s report.” However, that is not a reason for not trying to measure impact. In these cases, the best way to measure is via proxies for influence.
3. When You Can’t Measure Directly, Find Proxies
When it is not possible or feasible to measure the impact of your work directly, then you must find proxies that give some clarity, and measure those rigorously, tracking your trajectory.15 Perhaps the change you are looking to make is quite broad, and you need to find a specific metric that can be tracked relatively easily. For example, many international public health organizations looking to measure a community’s access to healthcare use the percentage of children who have received the third dose of the diphtheria, tetanus, and pertussis vaccine as a proxy. These organizations figured out that since the vaccine requires three doses and therefore three separate interactions with the healthcare system over time, the third dose serves as a fairly accurate indicator of how strong and accessible the system is.16 In some low-income countries, especially when economic data is sparse, nonprofits use the replacement of thatched roofs with more expensive corrugated metal ones as a proxy for poverty reduction.
CASE STUDY
How to Measure the Impact of a Think Tank
Faced with an economic recession, Arthur C. Brooks, the new president of the American Enterprise Institute (AEI), a free-markets think tank in Washington, DC, was under pressure to articulate the organization’s impact more convincingly. But since much of what think tanks do is indirectly influence policy, a clear link between AEI’s reports and its intended target, the thoughts and actions of American leaders, was hard to identify. Traditional evaluations like public opinion polls or legislative votes were not helpful in evaluating how demand for AEI’s ideas measured up with those of other organizations. So Brooks turned to a set of proxy metrics that, when combined, allowed AEI to assess how well their ideas were being taken up. Over time, they tracked metrics like how many op-eds they landed in the most competitive media outlets (such as the New York Times, the Wall Street Journal, and the Washington Post) and how often they were invited to participate in congressional testimonies as compared to other think tanks. Each of these proxies was of limited use on its own but, when combined, Brooks could argue that AEI was living up to its definition of success: the influence and uptake of the ideas it generated. These metrics weren’t only helpful for persuading donors; they also helped AEI leadership better assess how its programs were contributing to its ultimate goal.17
4. Indirect Impact and Systems Change
Organizations that aim to tackle thorny, deep-rooted social issues like poverty, gender inequality, and violence know that change doesn’t happen through one type of intervention; many contributory factors need to be addressed. You don’t sustainably reduce domestic violence just by providing shelter for victims or tackle poverty simply by providing cash handouts, even though these may be important interventions as part of a package of measures. Many factors are at play in these situations, and sustainable change requires addressing a number of them at the same time.
Complex systems—be they economic, political, social, or some combination of the three—call for complex solutions. As this understanding has taken hold in the nonprofit sector, the term “systems change” has emerged as a buzz-phrase. Though definitions can vary widely (Does it mean addressing root causes? Adapting to complexity? Bringing in “social entrepreneurs” who catalyze change across sectors?18), those of us focused on systems change can all agree that it is difficult to achieve and equally difficult to measure. If we are seeking to address these complex systems, we have to find ways to measure how our efforts and resources contribute to overall change, despite the challenges. This will often start with mapping out the various actors and issues that contribute to the problem, working out which of those actors and issues you can influence, and estimating how that might impact the system as a whole. As noted, none of this is easy, but if you don’t try then everyone invested in your work is being asked to take it on trust that you are making a difference.
CASE STUDY
Measuring Changes to Communities’ Vulnerability to Slavery
At the Freedom Fund, our work initially focused on direct support to those in slavery or at high risk of it. It was relatively straightforward to measure this impact in terms of individuals receiving services (such as victims liberated, individuals accessing social and legal services, at-risk children in school). But we soon became more ambitious and began to explore how our programs could address the systems that led to, and kept people in, slavery. The priority was not just getting individuals out of places where they were already being exploited but ensuring that those at risk didn’t end up in slavery in the first place. This meant making sure that laws were properly enforced and government officials did their jobs (instead of turning a blind eye to exploitation); it meant helping vulnerable individuals get access to government benefits and subsidies they were entitled to, so they didn’t get pushed into economic desperation; it meant changing norms, for example, by running information campaigns so people didn’t just assume this was always the way things were done, and that nothing could or should change.
Our partners were engaging with governments and businesses, helping strengthen local social protection systems and running social and behavioral change campaigns. They were having striking success with these joined-up efforts, and we recognized this would have long-term, sustained impact far beyond the life cycle of our programs. When a law is changed, a legal precedent is set, or a business changes its sourcing practices, the effects are felt by populations far larger than those with whom our partners directly interact. But our impact measurement framework was set up to collect direct impact figures (e.g., numbers of individuals reached by specific services, numbers of people sustainably liberated from modern slavery), and other key proxies (e.g., numbers of media stories, prosecutions, and children returned to school). We had no way of counting the impact on those we couldn’t see. We weren’t measuring our impact on the system.
So we turned to our research team and gave them a challenge: experiment with a framework for measuring “indirect impact.” (We define “indirect impact” as “beneficiaries who do not come into immediate contact with our grassroots partners and program activities, but nevertheless gain from the systems change that our program has contributed towards.”19) We also set a new organizational target of reaching ten million individuals by 2025 by focusing on and measuring our contribution to systems change. This target was driven partly by interest from our board, as well as an observation of the great potential for systems-level shifts in our program areas and our desire to capture it.
Our team began by analyzing various forms of systems change and coming up with a three-level system of categorization: protective environment, resilient communities, and empowered movement. Each of these categories describes a different level of impact, ranging from the broadest population affected by changes like national legislation and public attitudes (“protective environment”) down to the more micro shifts experienced by stronger effectiveness and collaboration between anti-slavery leaders and organizations (“empowered movement”). For each change that we have either observed or are seeking to achieve in our programs, we place it in one of these three categories and, using available data, estimate the size of the population that would benefit. By the end of 2022, we estimated we had indirectly impacted over seven million people.
We’ve added guardrails to avoid overcounting, keeping estimates conservative and accounting for variance in the status of implementation. This model of measurement is far from perfect, and would be very expensive to validate externally, but it has allowed us to find a way to evaluate our contributions to systems change and to compare the potential impact of various activities. In our experience, the ability to experiment with new forms of impact measurement has been extremely valuable, leading us to new ways to talk about our work and new strategic insights.
WHAT IF MEASURING IMPACT IS JUST TOO HARD?
It can be challenging for nonprofit leaders juggling limited resources, multiple demands, and an ambitious mission to find the resources necessary to measure impact robustly. You might feel frustrated when told what you should be measuring (by this book, or by multiple journal articles), with the implication you are failing if your impact measurement isn’t best-in-class.
The key takeaway should not be that you are failing unless you are conducting randomized control studies, baseline and endpoint prevalence surveys, or using sophisticated proxies. The most important thing to remember is that knowing what impact your efforts are having (or not) will help you ensure that your organization’s actions align with its purpose. Investing time and thought into how your organization can improve is invariably a worthwhile investment, and one that will look different for everyone.
Regardless of where you are on this journey, take a hard look at how your organization understands its own impact. Then ask yourself if that process could be improved. It usually makes sense to measure your activities as part of your efforts to track impact. You can start by embedding basic systems of measurement within how you design, carry out, and evaluate a program. That way, you aren’t left scrambling to track your impact retroactively. It particularly helps to measure your trajectory over time, as that can be a proxy for performance. Also, look at whether you can improve data collection. Can you better define the things you are measuring? Are there other particulars you can measure that will give a rounder picture of your work? Ask others for their input, particularly those you serve, and don’t just ask your fans for input, but also those who are more skeptical of your work. How do peer organizations approach this?
All nonprofit leaders should strive to ensure their organizations have the greatest impact, whatever that may look like, using the resources available. A mindset focused on impact and better (if not perfect) ways of measuring it is the first step to achieving maximum impact. This mindset is also key to developing a powerful strategy, which is what we will look at in the next chapter.
IMPACT ACTION POINTS
Identify and Measure Change
Work out the key inputs and activities of your nonprofit. • Then identify the change (outcomes) your organization is seeking to bring about or contribute to.
Assess how your organization is measuring its contribution to that change.
Determine whether the measures being used can be sharpened, clarified, or expanded.
If you can’t rigorously measure your organization’s impact, then explore whether you can improve the way you measure your activities and consider whether there are suitable proxies for impact that you can measure.
Ensure you are making the best use of all the information your organization is generating.
Regularly review and refine these efforts.
* In the Freedom Fund’s case, in four years we saw a reduction of over 80 percent in the overall levels of slavery in the communities in which we were working in northern India. See “Unlocking What Works: How Community-Based Interventions Are Ending Bonded Labour in India,” September 2019, Freedom Fund website at freedomfund.org/wp-content/uploads/Freedom-Fund-Evidence-in-Practice-Paper-Unlocking-what-works.pdf. The Freedom Fund stopped working in India in January 2023 due to the Indian government placing restrictions on the ability of international organizations to fund community-based organizations in the country.
14a While the terms “activities” and “outputs” can be used interchangeably, I’ll use “activities” throughout to avoid confusion.
* For more background on March of Dimes, see Georgette Baghdady and Joanne M. Maddock, “Marching to a Different Mission,” Stanford Social Innovation Review, Spring 2008.
* A randomized control trial (RCT) is a study that randomly assigns participants to either receive a service or program or to be in a control group, in order to compare the two groups’ outcomes. It’s regarded as a particularly rigorous way of measuring impact.
* Notably, when Russia’s invasion of Ukraine resulted in millions of Ukrainians fleeing for their lives, the US and UK established community-based schemes that mobilized hundreds of thousands of ordinary citizens as hosts and sponsors. Subsequent research on the experiences of hosts and sponsors also helped influence decisions in both countries to extend these schemes, such as through the Welcome Corps initiative that US Secretary of State Anthony Blinken announced in January 2023. More in Common’s strategy of providing robust quantitative and qualitative evidence of public attitudes shows that even on highly contested issues, there are pathways to influencing policymakers when you speak to their priorities (in this case, managing public opinion on a very sensitive issue). Matthew La Corte, L. J. Wolfgang Keppley, “New Poll Reveals Refugee Sponsorship Increases Support, Reduces Opposition to Resettlement,” Niskanen Center website, March 1, 2021, https://www.niskanencenter.org/new-poll-reveals-refugee-sponsorship-increases-support-reduces-opposition-to-resettlement/; “Welcoming Ukrainians: The Hosts’ Perspective,” More in Common website, March 13, 2023, https://www.moreincommon.org.uk/our-work/research/welcoming-ukrainians/.
