Is the debate about evaluation asking the wrong questions?
There are such strong debates about evaluations in the field of economic development that it sometimes seems like a civil war. But the debates about evaluation may be to some extent ill-posed. We might conceptualize the practice of development assistance as the “helpers” (development agencies, funding sources, and North NGOs) trying to give some assistance to the “doers” (people, organizations, and governments in developing countries). The assumed model is that those helpers sponsoring (and funding) a development program will want to do evaluations to see “what works and what doesn’t” so they will know how to better structure programs in the future. Then the debate rages about which evaluations are “best.”
Lately the debate has raged about Randomized Controlled Tests (RCT), a statistical methodology that dates back at least to Ronald A. Fisher (of the F-test fame) and his agricultural research work at the Rothamsted Experimental Station. When researching on plants, one can control the experimental design to include a wide variety of soils, fertilizers, irrigation levels, and the like. The drive to use this RCT experimental design methodology in development evaluations is part of the effort to be more “scientific” in some (rather crude) sense.
The attempt to apply this statistical methodology to human medical experimentation has been difficult from the beginning due to the obvious ethical constraints. And the attempt to shoehorn evaluations of large-scale social programs into the RCT model has required substantial feats of imagination. But what alternative is there? In spite of the problems in treating social programs like Fisher’s “potted plants,” how else can the helpers learn “what works and what doesn’t”?
I would like to suggest that there is a rather different way to approach this problem that renders the debate about evaluation rather secondary and ill-posed. As described in some detail in my book, Helping People Help Themselves, the important point is not the learning “what works and what doesn’t” by the helpers but by the doers themselves. Even if the helpers could begin to overcome all their incredibly strong organizational biases and preconceived notions about “what works,” such knowledge can rarely be “transmitted” to the doers in a way that is “owned” by the doers and can be the basis for sustained change.
The important point should be to foster direct and practical social learning by the doers about “what works and what doesn’t.” Such practical knowledge is hardly amenable to the RCT methodology that has academic cachet in the North and with the helper organizations adopting “scientistic” ideas about methodology in the human sciences. The best sources of such knowledge are other doers in roughly comparable situations who are trying to solve similar social problems. Thus the real alternative to some imagined “scientific evaluations” to foster social learning is peer-to-peer networks of the doers facing similar problems and searching for solutions. Instead of trying to be the “teachers” (of that “knowledge” supposedly gained from “scientific evaluations”), the North-helpers should focus on being the facilitators of South-to-South social learning networks.
Donald Schön and Everett Rogers on Decentralized Social Learning
Broadly speaking, there are two rather different paradigms of social learning. The standard model has the Center acquiring the knowledge (e.g., by supposedly surveying and evaluating “what works and what doesn’t”) and then transmitting “the knowledge” to the periphery perhaps along with funding to implement “the knowledge”—all to be followed up with more evaluations of the projects implemented in the periphery (e.g., in the American War on Poverty).
[The standard approach] treats government as center, the rest of society as periphery. Central has responsibility for the formation of new policy and for its imposition on localities at the periphery. Central attempts to ‘train’ agencies at the periphery. In spite of the language of experimentation, government-initiated learning tends to be confined to efforts to induce localities to behave in conformity with central policy. [Schön, Donald A. 1971. Beyond the Stable State. New York: Norton. p. 177]
But social learning can take place in a decentralized bottom-up manner perhaps with centralized facilitation. In large multi-plant companies, innovation may take the form of new ways of socially organizing and structuring productive processes, e.g., quality circles or self-managed work teams. Separate plants might perform pilot experiments to find out “what works and what doesn’t.” The headquarters office frames the experiments, detects the successes, and plays the knowledge-broker to help other plants cross-learn from the successful ones. In the Japanese system of continuous improvement, there is local problem-solving by teams, benchmarking between teams, and continuous improvement ratcheting up the performance of the teams.
Charles Sabel and his colleagues have recently made powerful applications of these ideas in social learning. But the ideas go back at least to Donald Schön who described a similar process involving the government and the periphery of local units trying to carry out a certain social reform.
Government cannot play the role of ‘experimenter for the nation’, seeking first to identify the correct solution, then to train society at large in its adaptation. The opportunity for learning is primarily in discovered systems at the periphery, not in the nexus of official policies at the center. Central’s role is to detect significant shifts at the periphery, to pay explicit attention to the emergence of ideas in good currency, and to derive themes of policy by induction. The movement of learning is as much from periphery to periphery, or periphery to center, as from center to periphery. Central comes to function as facilitator of society’s learning, rather than as society’s trainer. [Schön, Ibid., 177-8]
Decentralized parallel experimentation with centrally-sponsored framing and benchmarking followed by peer-to-peer cross-learning in the periphery is a more appropriate model than research at a central facility followed by the teaching-dissemination of the results.
In Everett Rogers’ early work on the diffusion of innovations he focused on the classical hub-and-spokes or center-periphery model of diffusion.
In this classical diffusion model, an innovation originates from some expert source (often an R&D organization). This source then diffuses the innovation as a uniform package to potential adopters who accept or reject the innovation. The role of the adopter of the innovation is that of a passive accepter. [Rogers, Everett 1983. Diffusion of Innovations. New York: Free Press, p. 333]
Spurred on by Schön’s work, he became aware of decentralized learning systems with horizontal diffusion between peers (which might involve partial re-invention of the model) rather than vertical transmission from expert-helpers to doers.
During the late 1970s I gradually became aware of diffusion systems that did not operate at all like the relatively centralized diffusion systems that I had described in my previous books. Instead of coming out of formal R&D systems, innovations often bubbled up from the operational levels of a system, with the inventing done by certain users. Then the new ideas spread horizontally via peer networks, with a high degree of re-invention occurring as the innovations are modified by users to fit their particular conditions. …
Gradually, I began to realize that the centralized diffusion model was not the only wheel in town. [Rogers, Ibid., p. 334]
Perhaps the best example of a parallel system of decentralized innovation and diffusion in a developing country is in China over the last three decades. The Chinese recognized local reform models which could be in a region, county, commune, or even brigade, and could be in any sector or area such as administration, health, education, or industry. The center would recognize a “model” which could then be visited by groups from all over China who want to make a similar reform in their locality.
The diffusion of innovations in China is distinctive in that it is (1) more horizontal in nature, (2) less dependent upon scientific and technical expertise, and (3) more flexible in allowing re-invention of the innovation as it is implemented by local units. These aspects of decentralized diffusion are facilitated by China’s use of such diffusion strategies as models and on-the-spot conferences. The “learning from others” approach to decentralized diffusion in China was adopted officially as a national policy in the national constitution in 1978. [Rogers, Ibid., 340-1]
The same period marks the beginning of China’s historic record of growth and development at the end of the twentieth century, perhaps the most remarkable growth episode in history.