Comments on: Does Measurement = Randomized Control Trials? https://andrewwolk.com/2009/10/27/does-measurement-randomized-control-trials/ Advancing Social Innovation - Investing in What Works Fri, 30 Oct 2009 20:02:26 -0500 http://wordpress.org/?v=2.8.4 hourly 1 By: The Investing in Innovation (i3) Fund | Andrew Wolk https://andrewwolk.com/2009/10/27/does-measurement-randomized-control-trials/comment-page-1/#comment-4449 The Investing in Innovation (i3) Fund | Andrew Wolk Fri, 30 Oct 2009 20:02:26 +0000 https://andrewwolk.com/?p=444#comment-4449 [...] takes a similar approach to the social impact measurement spectrum I recently wrote about in my post on randomized control trials (RCT). The fund offers three levels of award categories: up to $5 million (“development”), up [...] [...] takes a similar approach to the social impact measurement spectrum I recently wrote about in my post on randomized control trials (RCT). The fund offers three levels of award categories: up to $5 million (“development”), up [...]

]]> By: Lisa Jackson https://andrewwolk.com/2009/10/27/does-measurement-randomized-control-trials/comment-page-1/#comment-4427 Lisa Jackson Fri, 30 Oct 2009 13:20:35 +0000 https://andrewwolk.com/?p=444#comment-4427 Andrew - I like the model you put forth conceptually for how investors/funders should think about when and what to support regarding assessment. It might be helpful to line up this model with a lifecycle model of nonprofit organization development. This would help the nonprofits and investors think about where they should be in terms of assessment based on where there organization is developmentally (relative to things like infrastructure and capacity, program development and implementation). Just because funding is available for assessment doesn't mean a nonprofit is ready or has the capacity to engage with it at the desired level. Andrew – I like the model you put forth conceptually for how investors/funders should think about when and what to support regarding assessment. It might be helpful to line up this model with a lifecycle model of nonprofit organization development. This would help the nonprofits and investors think about where they should be in terms of assessment based on where there organization is developmentally (relative to things like infrastructure and capacity, program development and implementation). Just because funding is available for assessment doesn't mean a nonprofit is ready or has the capacity to engage with it at the desired level.

]]> By: Katya Fels Smyth https://andrewwolk.com/2009/10/27/does-measurement-randomized-control-trials/comment-page-1/#comment-4330 Katya Fels Smyth Thu, 29 Oct 2009 21:50:52 +0000 https://andrewwolk.com/?p=444#comment-4330 Andrew: I like your suggestion of tiering “proven-ness.” I would like to suggest, however, that the problems with overreliance on RCTs are far greaster than cost and time. There is no magic bullet for addressing poverty; there is no magic bullet for evaluation either. RCTs are an important tool, but they are absolutely not the gold standard. For example, RCTs homogenize populations, when many of our interventions work with people who come to us with a variety of constellations of issues and challenges; They are limited in attending to differential “dosing,” dealing with models that change over time, and attending to change at the individual and policy levels. They seek to demonstrate context-neutral interventions, when most effective interventions are place based. I spare you more (for more see: <a href="http://www.hks.harvard.edu/socpol/A_Lot_to_Lose.pdf.)" target="_blank">www.hks.harvard.edu/socpol/A_Lot_to_Lose.pdf.)</a> Overreliance on RCTs distorts social policy and further marginalizes already marginalized people. Again, I agree we in the field need to be accountable—often more so than we have been. We need to allow experts to pick from among a tool box to ensure that evaluation is both rigorous and relevant. Andrew: I like your suggestion of tiering “proven-ness.” I would like to suggest, however, that the problems with overreliance on RCTs are far greaster than cost and time. There is no magic bullet for addressing poverty; there is no magic bullet for evaluation either. RCTs are an important tool, but they are absolutely not the gold standard. For example, RCTs homogenize populations, when many of our interventions work with people who come to us with a variety of constellations of issues and challenges; They are limited in attending to differential “dosing,” dealing with models that change over time, and attending to change at the individual and policy levels. They seek to demonstrate context-neutral interventions, when most effective interventions are place based. I spare you more (for more see: http://www.hks.harvard.edu/socpol/A_Lot_to_Lose.pdf.)
Overreliance on RCTs distorts social policy and further marginalizes already marginalized people.
Again, I agree we in the field need to be accountable—often more so than we have been. We need to allow experts to pick from among a tool box to ensure that evaluation is both rigorous and relevant.

]]> By: Jeff Mason https://andrewwolk.com/2009/10/27/does-measurement-randomized-control-trials/comment-page-1/#comment-4302 Jeff Mason Thu, 29 Oct 2009 14:49:35 +0000 https://andrewwolk.com/?p=444#comment-4302 Andrew, great post! I couldn’t agree more. Your model makes a lot of sense and syncs with the work of David Hunter and Steve Butz in their Guide to Effective Social Investment and the related Social Value Risk Assessment tool. Both can be found on the Alliance for Effective Social Investing website (<a href="http://www.alleffective.org)." target="_blank">http://www.alleffective.org).</a> I like to think of high-performing orgs as those that have clear goals, established indicators to monitor their progress toward achieving goals, collect quality data that relates their efforts to desired outcomes so they can understand what works and what doesn’t, and then make appropriate adjustments to continuously improve. This is an essential part of the daily work an org must do to stand a significant chance of achieving impact. It’s a high-impact org that has been shown via RCT to have been the cause of the social change. But even high-impact orgs need to be high-performing on a daily basis since impact that may have been found via the RCT is in the past. We need to invest in the future. What will you do tomorrow? This is all about an org ability to manage their performance. Andrew, great post! I couldn’t agree more. Your model makes a lot of sense and syncs with the work of David Hunter and Steve Butz in their Guide to Effective Social Investment and the related Social Value Risk Assessment tool. Both can be found on the Alliance for Effective Social Investing website (http://www.alleffective.org).

I like to think of high-performing orgs as those that have clear goals, established indicators to monitor their progress toward achieving goals, collect quality data that relates their efforts to desired outcomes so they can understand what works and what doesn’t, and then make appropriate adjustments to continuously improve. This is an essential part of the daily work an org must do to stand a significant chance of achieving impact.

It’s a high-impact org that has been shown via RCT to have been the cause of the social change. But even high-impact orgs need to be high-performing on a daily basis since impact that may have been found via the RCT is in the past. We need to invest in the future. What will you do tomorrow? This is all about an org ability to manage their performance.

]]> By: Debra Natenshon https://andrewwolk.com/2009/10/27/does-measurement-randomized-control-trials/comment-page-1/#comment-4224 Debra Natenshon Wed, 28 Oct 2009 12:38:01 +0000 https://andrewwolk.com/?p=444#comment-4224 Andrew, this is a great model. The most promising element being that there is FUNDING tied to each level of development directly allocated for measurement. For too long, the need for the measurement has been clear, but there was a fundamental disconnect between that knowledge and its implementation The only issue now becomes convincing donors and public and private foundations that this is not only worthwhile but necessary if they want to see results! I am hopeful that through your work, blogs like this and our soon-to-be-released Foundation Effectiveness Toolkit, the abyss will seem a little more tangible and measurement and performance management will become the norm. Andrew, this is a great model. The most promising element being that there is FUNDING tied to each level of development directly allocated for measurement. For too long, the need for the measurement has been clear, but there was a fundamental disconnect between that knowledge and its implementation The only issue now becomes convincing donors and public and private foundations that this is not only worthwhile but necessary if they want to see results!

I am hopeful that through your work, blogs like this and our soon-to-be-released Foundation Effectiveness Toolkit, the abyss will seem a little more tangible and measurement and performance management will become the norm.

]]>