Day 3 ⏐ Session 2
Shifting perspectives:
why innovators and investors need new approaches to learning
In this session our team of panelists explore the dance between bringing an evidence based perspective whilst also being able to stay nimble and responsive to intuition, evolving together in complex environments.
This is a conversation that really pulls at questions around the meaning and purpose of evaluation, what’s it really for, what’s evaluated and how it can shake of a bad reputation for being over bureaucratic, professsionalised and controlling.
We heard perspectives from Lee Alexander Risby, Laudes Foundation (CH); Ben Ramalingam, author and Director, UK Humanitarian Innovation Hub (UK); Anne Bergvith Sørensen, Homes for All (Hjem Til Alle) (DK) and Chris Clements, Social Finance (UK).
“I think we have to rethink the purpose and position of evaluation in complex settings. You have to ask the question, what are we evaluating, why, and for whose benefit? Are we doing evaluation for accountability? Or, are we doing evaluation for learning? We need to operationalise our intentions about evaluation much more clearly”
Resources
Anne Bergvith Sørensen
Inspiration from the Zoom chat:
Ben Ramalingam
This is a framing developed for MEL of systemic and adaptive programmes which might be helpful
Supporting adaptive management - monitoring and evaluation tools and approaches
Chris Clements
Lee Alexander Risby
Quotes from the session
“Systems are interconnected. They involve lots of things coming together. And so shifting systems isn't necessarily very collaborative. It’s an often a protracted, uneven and evolving process. Understanding where you are in that process when you're just one actor, when you're seeking alliances with others and you can't see the whole picture is a difficult thing.” (Charlie Leadbreater)
“I think there are ‘clock’ problems and there are ‘cloud’ problems, but many of the problems we face in the humanitarian sector are ‘crowd’ problems. In the context of aid, problems are human beings, or groups of human beings that have perceptions and ideas and dependencies and interdependencies and they have conflicts with each other and they cooperate with each other. And that's a different kind of challenge again, when especially when you're talking about crisis situations.” (Ben Ramalingam)
“How we can tap into the kind of evaluation that a context needs to match the dynamics of the system that we're talking about.” (Ben Ramalingam)
“...in really, really unpredictable setting, stays somewhere like Somalia, which has both high levels of famine and high levels of conflict, you can't necessarily use evaluation in the traditional sense at all… you kind of have to probe, sense respond, learn as you go and figure out what works. Evaluation there is about observing activities and the possible results with those activities in a very context specific way. It's about sense making and drawing people into dialogue.” (Ben Ramalingam)
“What we have tried to do is say we need to focus on our guiding star. Our guiding star is our vision. It’s to end homelessness. So when we are tracking change, we are trying to check signs of change in all different levels…. when we're talking about shifts on the macro level we're talking about whether we’re changing the discourse, the way we talk about homelessness in Denmark the way we talk about the solutions to homelessness in Denmark. Are we changing the narratives and the mental models of how we understand homelessness in Denmark? ” (Anne Bergvith Sørensen)
“We're getting away from the focus on accountability and learning… To us, that's not what evaluation is about. I want evaluation to make us accountable to learn.” (Lee Alexander Risby)
“We still do want to hold on to that evaluation for accountability as a reason for thinking about evaluation because for us, we're practitioners first and foremost, we want to understand [if we are] getting any closer to the change that we seek? It’s why we are here. Of course, the challenge is, how do you do that without overclaiming and recognising the complexity that exists? (Chris Clements)
“So I guess our starting point when it comes to evaluation is how you can hold loosely to the concept of attribution and instead approach evaluation knowing that systems shifting efforts, they're messy, they're emergent, and capturing contribution is likely to be a much more meaningful starting place for evaluation. But actually, you've still got the same question of ‘contribution to what?’ and I believe you still need to answer that question” (Chris Clements)
“I want to talk about money, the economics of evaluation. Of course, in many bureaucracies there's a kind of mechanistic estimation of evaluation budget, a cookie cutter approach that assumes that you can have a cost and duration of evaluation as programmed without any consideration of the existing data or the size or scope of the intervention in question.” (Ben Ramalingam)
“As evaluation becomes associated with bureaucratic failures that are higher up the system… evaluators end up telling leaders the bad news about their floored assumptions and then the evaluators are then taken out and shot because they're the ones that have turned up with the bad news. But the whole thing is completely farcical in some ways” (Ben Ramalingam)
“...how do we bring an evidence based perspective to our programmes so that at the heart of our programmes there is an engine for intuition and action. Now, if we start using the term ‘evaluation’ to talk about that engine for intuition and action we're going to quickly be linked to a bureaucratic function of evaluation and not to the kind of visionary ideal evaluation that many of the people in this room are talking about it.” (Ben Ramalingam)
“And I think it’s evaluative thinking, how we critically engage with the world, how we understand it, how we understand a problem as systemic, how we understand our interventions, the feedback that we get, how we interpret that with different kinds of stakeholders. In that kind of context. evaluation it’s not about judge, jury and executioner is much more guidance and Sherpa it's about helping test assumptions” (Ben Ramalingam)
“I don't know if we're measuring shifts in power, but we are acknowledging the fact that we are talking about the shift in power or decision making power within the system. The concrete shift that we're trying to push for is from a system that's handling homelessness with mainly temporary emergency solutions to a system that's handling homelessness with housing and social support because we know that works.” (Anne Bergvith Sørensen)
“...do you delve into the issues that you care about and then just add the line at the end ‘with a special focus on people of colour, women, people with disabilities, migrants, LGBTQI communities’ at the end of your strategies? Or, do you actually have that at the core of what you're doing? A strategic approach to shifting power dynamics point agency towards people in communities that are really considered to be the focus of your work.” (Ben Ramalingham)
“...how do we shift the agency towards those people who are actually at the frontline doing most of the important work so that the resources are reallocated to change the status quo. We do need to frame it in gender terms. We do need to frame it in racial terms. The challenge of that is, who you are and what you stand for when you say those things?” (Ben Ramalingham)
