Does Measurement = Randomized Control Trials?
Posted on 27. Oct, 2009 in Social Impact Measurement
There is a debate going on about how to identify and invest in the highest-performing nonprofits – with a great emphasis on randomized control trial (RCT) studies. The RCT, a research methodology that involves randomly selecting subjects from a larger test group to receive an experimental product or service, is undoubtedly a rigorous way of determining whether a cause and effect relationship exists between a given service and a desired outcome. The current interest in RCTs is an encouraging sign of the growing momentum for linking nonprofit funding to proven results and investing in what works, particularly if we can also disseminate the information far and wide. At the same time, RCTs raise concerns for me, as they could end up stifling rather than encouraging social innovation.
The trouble with randomized control trials is that they are extremely expensive to carry out and so only the more developed and/or the best able to access resources participate. Most nonprofits struggle with finding the funds for general operations, let alone conducting an RCT.
So, we are stuck between encouraging social innovation and the strong desire to invest in what works. I believe we need to develop a spectrum that links an organization’s stage of development, with measurement requirements, to levels of investment. The stages could look something like this:
The Social Impact Measurement Spectrum
- Stage 1, Start-up: At the earliest stage, start-ups with promising ideas could receive minimal support in exchange for a commitment to developing and tracking an initial set of measures that align with standards in the organization’s field.
- Stage 2, Proof of concept: At the next stage, the organization would have a simple, internally driven performance measurement system, which would drive internal continuous improvements while generating initial data on the impact of the organization’s model. Funding would be increased to match the increased rigor of the organization’s measurement, but it would still be far less than that given to organizations at later stages.
- Stage 3, Promising, not proven: At this stage, organizations would be required to operate a full performance measurement system that is integrated with their business models and publicly share the results. They may also do an external evaluation at this stage to further elucidate their impact.
- Stage 4, Spread of proven model: This final stage would occur when an organization has conducted an RCT study with positive results. They would be able to receive substantial investment in exchange for a commitment to share the details of their model so it may spread and others can learn and adopt the organization’s successful practices. The Coalition for Evidence Based Policy is building a site that will identify those social interventions shown in rigorous studies to produce sizable, sustained benefits to participants and/or society.
A spectrum like this would do a few important things. It would provide a framework that would inform how a social innovation might spread, what kind of measurement system an organization should develop overtime, and, lastly, the size of funding an organization might expect as an innovation moves towards greater proven social impact.
| Share | | Previous Post | Next Post |
8 Responses to “Does Measurement = Randomized Control Trials?”
Leave a Reply
-
What are YOUR ideas? Let Andrew Wolk take them to D.C.
-
Why the Social Innovation Fund is on the right track
- Does Measurement = Randomized Control Trials?
-
Michelle Rhee, Public Innovator
-
A Historic Day - Are you 'all in'?
-
Mark: I only wish I had a nickel for every time someone ...
-
Lawrence Baker: Social Innovation? What a concept! What we need in...
-
JBRootCause: Hi Rhesa, You can find the interview here: ...
-
Rhesa J: Looking for updated link to an interview mentioned...
-
E Favorite: Hi, Tom - if you're talking about a teacher f...
Debra Natenshon
28. Oct, 2009
Andrew, this is a great model. The most promising element being that there is FUNDING tied to each level of development directly allocated for measurement. For too long, the need for the measurement has been clear, but there was a fundamental disconnect between that knowledge and its implementation The only issue now becomes convincing donors and public and private foundations that this is not only worthwhile but necessary if they want to see results!
I am hopeful that through your work, blogs like this and our soon-to-be-released Foundation Effectiveness Toolkit, the abyss will seem a little more tangible and measurement and performance management will become the norm.
Jeff Mason
29. Oct, 2009
Andrew, great post! I couldn’t agree more. Your model makes a lot of sense and syncs with the work of David Hunter and Steve Butz in their Guide to Effective Social Investment and the related Social Value Risk Assessment tool. Both can be found on the Alliance for Effective Social Investing website (http://www.alleffective.org).
I like to think of high-performing orgs as those that have clear goals, established indicators to monitor their progress toward achieving goals, collect quality data that relates their efforts to desired outcomes so they can understand what works and what doesn’t, and then make appropriate adjustments to continuously improve. This is an essential part of the daily work an org must do to stand a significant chance of achieving impact.
It’s a high-impact org that has been shown via RCT to have been the cause of the social change. But even high-impact orgs need to be high-performing on a daily basis since impact that may have been found via the RCT is in the past. We need to invest in the future. What will you do tomorrow? This is all about an org ability to manage their performance.
Katya Fels Smyth
29. Oct, 2009
Andrew: I like your suggestion of tiering “proven-ness.” I would like to suggest, however, that the problems with overreliance on RCTs are far greaster than cost and time. There is no magic bullet for addressing poverty; there is no magic bullet for evaluation either. RCTs are an important tool, but they are absolutely not the gold standard. For example, RCTs homogenize populations, when many of our interventions work with people who come to us with a variety of constellations of issues and challenges; They are limited in attending to differential “dosing,” dealing with models that change over time, and attending to change at the individual and policy levels. They seek to demonstrate context-neutral interventions, when most effective interventions are place based. I spare you more (for more see: http://www.hks.harvard.edu/socpol/A_Lot_to_Lose.pdf.)
Overreliance on RCTs distorts social policy and further marginalizes already marginalized people.
Again, I agree we in the field need to be accountable—often more so than we have been. We need to allow experts to pick from among a tool box to ensure that evaluation is both rigorous and relevant.
Lisa Jackson
30. Oct, 2009
Andrew – I like the model you put forth conceptually for how investors/funders should think about when and what to support regarding assessment. It might be helpful to line up this model with a lifecycle model of nonprofit organization development. This would help the nonprofits and investors think about where they should be in terms of assessment based on where there organization is developmentally (relative to things like infrastructure and capacity, program development and implementation). Just because funding is available for assessment doesn't mean a nonprofit is ready or has the capacity to engage with it at the desired level.
The Investing in Innovation (i3) Fund | Andrew Wolk
30. Oct, 2009
[...] takes a similar approach to the social impact measurement spectrum I recently wrote about in my post on randomized control trials (RCT). The fund offers three levels of award categories: up to $5 million (“development”), up [...]
Ginny Deerin
30. Oct, 2009
As a practitioner, these 4 blocks make sense to me. Hopefully, once a program gets through block 3 and begins to head into block 4, philanthropic investment will come along. Non-profit programs that have been willing to go this far should be rewarded and provided with funds to push into block 4 while continuing to make social impact.
Stephen J. Gill
30. Oct, 2009
Randomized control trials work well when you can truly randomize selection of subjects, when you can be sure that the only difference between groups is the social intervention being tested, that there is no treatment contamination, and observation of the groups does not affect the outcomes. These circumstances are extremely rare. RCT works well when measuring effects on seeds and mice; not so well measuring social impact on people, organizations, and communities. And establishing causal relationships between social interventions and outcomes is beyond this research method. The most we can say is that we are fairly confident the treatment contrbuted in some substantial way to the outcomes but other factors contributed as well. This is not to discredit your stages of investment in evaluation. It's just that I don't think RCT is the gold standard of evaluation. It all depends on what questions are being asked and if RCT can answer those questions. Other methods can determine if a social intervention contributed substantially to important social outcomes. I suggest removing RCT from Stage 4 and instead say something like "measure of impact of social intervention on people, organizations, and communities."
Performance Measurement – It’s Now or Never! | Andrew Wolk
04. Nov, 2009
[...] out there ranging from advice on how best to report overhead ratios to how to put out an RFP for a randomized control trial, it can be challenging for nonprofits to figure out where what they need to do and how to get [...]