What do the Homelessness Partnerships Strategy, the Forest Communities Program and the Aboriginal Leadership Strategy have in common? They are all federal initiatives that address stubbornly persistent issues by engaging in a collaborative decision-making and planning approach within a place. Whether they focus on a neighbourhood, a community or an ecosystem, such “place-based” approaches leverage resources, build common objectives and solutions, and put into place a co-ordinated response across multiple sectors. They involve many actors working in partnership and have unpredictable paths to success - a complex dynamic that traditional evaluation has a hard time reconciling.
Habits, Needs and the Drive for Change
Place-based initiatives embrace difference, responding to the unique context of each place. Assessing their impacts appears to be problematic using traditional evaluation practices, which generally require pre-determined outputs and outcomes. This can be particularly true within the public sector, where an historic focus on performance management and accountability has made it more difficult to tailor national programs to on the ground realities. Top-down evaluation approaches have been identified by some as detrimental to the collaborative benefits of place-based approaches. For example, evaluation expert John Mayne has encouraged governments to embrace a new view of accountability that acknowledges partnerships and complexity.
This would also mean responding to the multi-dimensional evaluation needs of place-based approaches, which include performance management, evaluation for accountability and evaluation for learning that strive to be meaningful to multiple partners with different perspectives and needs. There is an emerging question in the evaluation community as to whether it is possible to achieve both accountability and learning within the same evaluation. While this is an open question, a potential hurdle could exist if performance management, evaluation for accountability and evaluation for learning do not inform one another. A nested approach might be a model to explore, with accountability processes linked to an evaluation for learning, serving different but complementary purposes.
The Window is Ajar
Some evaluation approaches, such as “Theory of Change”, and “developmental evaluation” offer promise for evaluating programs that are expected to innovate and evolve over time. Like logic models on steroids, Theory of Change encourages program managers to articulate the immediate, intermediate and long-term outcomes, as well as the necessary preconditions for long-term outcomes. It also makes explicit the cause and effect assumptions behind the mechanisms in the program that create change, and encourages testing these assumptions. These elements become the basis for evaluation planning. This builds policy evidence that can be generalized, mitigating the need to prove attribution for the same change mechanism in each application. This is particularly useful when direct attribution is not feasible, as is often the case in place-based approaches. Developmental evaluation was created to support social innovation, such as when general principles are applied to a new context or in cases where experimentation is part of the on-going program model. It enables the theory of change behind a program to emerge over time as the program itself develops, and encourages on-going evaluation. These approaches place learning and evaluation at the local project level as a front and centre objective.
In Canada, federal government programs using theory of change evaluation are being piloted at HRSDC and CIDA, and TBS - under the 2009 Evaluation Policy - is encouraging these new approaches. A window appears to be opening for new ways to evaluate these programs nationally, which may have application in unexpected ways. Theory of Change could be used to link evaluations of current programs to those of previous program iterations – ones that are now considered separate for evaluation purposes, but are the same in essence. This would mean taking a flexible approach to evaluation research questions and strategically focusing on what an evaluation can effectively achieve at any given time, with a vision to creating a long-term analysis. Potentially, a federal framework for place-based evaluation could communicate findings across a wide range of programs with a similar change mechanism and contribute even more effectively to a growing knowledge base.
In the United States, the United Kingdom and Australia, there is increasing activity within the field of place-based evaluation, indicating growing empirical evidence across disciplines that could be effectively mined to assist a better understanding of both the strengths and weaknesses of place-based approaches. Continued research holds promise for an improved learning environment, an expanded evidence base and would move toward integration of economic, environmental and social goals within initiatives.
Bradford, Neil and Chouinard, Jill Anne, 2010. "Learning Through Evaluation? Reflections on Two Federal Community Building Initiatives." The Canadian Journal of Program Evaluation Vol. 24. No 1 p 55-77.
Cantin, Bernard. 2010. "Integrated Place-Based Approaches for Sustainable Development." Horizons: Sustainable Places, volume 10, issue 4. Policy Research Initiative: Ottawa
Koontz, Tomas M. and Craig W. Thomas,. 2006. "What Do We Know and Need to Know about the Environmental Outcomes of Collaborative Management?" Public Administration Review. December 2006, Special Issue on Collaborative Public Management.
Mayne, John. 2008. Contribution analysis: An approach to exploring cause and effect. Institutional Learning and Change Brief 16.
Potvin, Louise and David McQueen. 2008. Health Promotion Evaluation Practices in the Americas Values and Research. New York.
Shepherd, Robert P. 2010. "In Search of a Balanced Evaluation Function: the Sate of Federal Program Evaluation in Canada." Presented April 7-9, 2010 to the International Research Symposium on Public Management: Berne, Switzerland.
Shugart, Ian, and Thomas Townsend. 2010." Bringing "Place" In – Exploring the Role of the Federal Governmen." Horizons: Sustainable Places, volume 10, issue 4. Policy Research Initiative: Ottawa.
Sridharan, Sanjeev and Elizabeth Lopez. 2004. "Methodological Lessons Learned for the Process Evaluation of the Comprehensive Strategy for Serious, violent and Chronic Juvenile Offenders." Social Policy Journal of New Zealand Issue 22, July 2004.