This post is an expanded commentary on today’s posting on the World Bank’s Impact Evaluation blog about IEs and Sachs’ Millennium villages.
Their discussion nicely avoids a much more basic problem in real-existing impact evaluations, namely that the comparisons are to no-treatment cases rather than the best-alternatives using comparable resources. If the Millennium villages, where $XXXX is spent per person, passed all the impact evaluation tests compared to no-treatment comparison villages, that would still be of little significance since it uses the no-treatment pseudo-counterfactual.
The real question is the best alternative use of scarce development aid, and that is best investigated by fostering parallel experimentation of different locally-sponsored approaches and then comparisons or benchmarking between them.[1]
Without parallel experiments where comparable resources were spent per person, even “successful” impact evaluations of the MVs would only be employing the ultimate low-hurdle of showing that spending a lot of money is better than doing nothing.
Hence we see this pseudo-debate between the World Bank, the Vatican of social engineering, and the rogue high-priest in that church and world’s most successfully self-promoting social engineer, Jeff Sachs, about the details of “scientific” impact evalutions and randomized testing, while the underlying flaws in this approach to development assistance go unaddressed by either side.
Thinking impact evaluators acknowledge that IEs only measure the “benefit side” and that one would have to do a number of IEs in other projects to get a meaningful comparison of benefits and costs. But projects are not designed that way since the whole social engineering mentality wants to invest all the scare development resources in the “best designed project.” Why not the “best” for our clients? Hence there is “no data” on alternatives where comparable resources were spent. Thus while the pseudo-counterfactual of no treatment is theoretically acknowledged as dealing only with the “benefit side,” that is how all or almost all real-existing impact evaluations take place, i.e., using only the ultimate low hurdle of “better than nothing.” Gee, I wonder if that ultimate low hurdle aspect of impact evaluations has anything to do with their rising popularity with World Bank project managers??
See Figure 1: Arianna Legovini, Development Impact Evaluation Initiative: A World Bank-Wide Strategic Approach to Enhance Developmental Effectiveness, Washington DC: World Bank, 2010, p. 7. Downloadable at: http://go.worldbank.org/9R255TPZQ0