Evaluating complex programmes: Reflections on realism and resilience

What works and why in implementing and achieving outcomes in adaptation and resilience-building projects? This paper considers the challenges faced and lessons learned in the mid-term review of BRACED projects to provide insights into using realist principles to evaluate complex, resilience-building programmes in an international development context.

• There is value in applying a ‘realist’ way of thinking throughout the programme cycle – not just at evaluation stage. This may be especially relevant for large, complex multisectoral programmes that all contribute to common higher-level outcomes and impact.  
• Even if outcomes are yet to be realised, achievement of outputs can be viewed with a realist lens as important pathways towards change, allowing us to capture where implementers have put in place or effectively carried out important processes.
• As part of a broader theory of change approach, applying a realist lens forces you to ask important questions of how and why projects arrived at their outcomes, and formalises this.  
• Realist approaches allow us to deal with multiple understandings of the term resilience. The process of moving away from rhetoric and asking basic but important questions helps us to improve our understanding of what resilience is in different contexts, and how resilience is built. Moving from rhetoric to realities means the BRACED final evaluation may not measure ‘impact’ or ‘resilience’ as defined in project documentation or baseline. Our growing understanding of what matters for building resilience means that some important elements may not be captured in project logframes. We need   to be flexible and iterative in our approach.
• Realist framing of pathways to change focuses on context-mechanism-outcome (CMO) configuration, which assumes a linear process of activity to outcome and outcome to result. By allowing our CMOs to contain multiple, detailed steps, complementing this with an analysis of barriers and enablers of change, and unexpected and unintended effects, outcomes or consequence, we found we were able to retain much of the richness and dynamics that help us to tell the story. This strengthened our ability to analyse processes and outcomes that do not fit easily into a CMO ‘box’.