This piece was originally published as part of the John D. and Catherine T. MacArthur Foundation’s Learning as We Go evaluation Perspectives series, with co-authors Anne Gienapp, Lindsay Hanson, and Hallie Preskill. The blog introduces a case story on The MacArthur Foundation’s Approach to Evaluation and Learning.
In October 2018, MacArthur convened a meeting of its six evaluation and learning partners (ELPs), one for each of its Big Bets and Enduring Commitments. While all the ELPs were experienced evaluators with foundation clients, the meeting represented a new and unique opportunity. It was the first time any of us had met with a client alongside our client’s other consultants for the explicit purpose of sharing the successes and ongoing tensions of our respective engagements—something that does not often occur in our field.
However, with the encouragement from the Office of Evaluation, we recognized that the benefits of being open about our experiences outweighed the potential downsides or uneasiness of speaking about our challenges and revealing what we sometimes safeguard as exclusive content and tactics. First, it offered us an opportunity to draw on each other’s work to gain a deeper and more comprehensive understanding of how to best support MacArthur’s approach to strategy, evaluation, and learning. Moreover, it lent legitimacy to our roles by enabling us to model the behavior and thinking that we are collectively working toward within MacArthur and across the philanthropic sector. As evaluators, we should lean into the discomfort and perceived risk of making our work visible and open to scrutiny, so we can reflect on our decisions, invite feedback, and ultimately improve upon what we are doing.
In fact, the energy we all derived from this exchange spurred our decision to establish a community of practice among the ELPs. We meet regularly, without MacArthur’s participation, to highlight challenges and successes about our work with, swap ideas, and learn from each other’s experiences.
One of the early tasks set by this community of practice was to find venues to more broadly share our experience of working as an ELP within MacArthur’s approach to evaluation and learning. The community of practice considered a range of options, including writing a peer-reviewed article for journal publication, presenting at field-wide conferences, and developing a series of blog posts. In the end, we settled on this case story. It is based on interviews with MacArthur staff (leadership, program, and evaluation) and the ELP community of practice; our review of extant literature on the philanthropic sector; and written materials about MacArthur’s approach to evaluation and strategy.
This case story is not intended to present MacArthur’s approach as the way to do evaluation; there are many ways to evaluate programming, each one being more or less appropriate for any given context. Instead, we are sharing what we learned to inform efforts within philanthropy and beyond to link evaluation with strategy, drive informed decision making, and promote transparency and accountability in the social sector.
We have seen how MacArthur’s approach to evaluation is rooted in values, especially in the value of diversity of thought that truly challenges assumptions and invites criticism of their strategies. That invitation for dynamic feedback, however, brings inherent tensions. Being a “critical friend,” the role ELPs are supposed to fill, requires a delicate balance. We must have deep knowledge of MacArthur’s complex strategies while leading evaluations that test the logic and coherence of those strategies.
To manage this balance, our role requires bringing a high level of rigor to the methodologies we employ so we can be confident in the findings we present. There is a personal factor that is essential to the work, too. To drive uptake of our evaluation findings, there must be strong working relationships that are grounded in mutual respect and trust between the ELPs, Evaluation Officers, and the program teams. The attention that goes into building those relationships is critical to the utility of evaluation, ensuring that program teams can translate learnings into evidence-based decisions about their work. And, this mutual respect and trust requires ongoing cultivation and commitment from us as ELPs, the program teams, and the Evaluation Officers serving a critical intermediary role.
Through the process of interviews and writing this case story, our ELP community of practice gained insights into the dynamics of MacArthur’s approach to evaluation. We confirmed the importance of relationship building and management as a critical pillar of ELPs’ engagements with MacArthur and that building a strong support network of evaluators has improved the value of our respective work. We hope that this take-away and the others that are highlighted in the case story can be helpful to foundations and others that, like MacArthur, continue to grapple and experiment with ways to get the most out of evaluation and learning.