How we created and tested a measurement and learning framework for Collaborative Learning Networks

Editor’s note: The following blog post is the second installment in a series exploring Results for Development’s Collaborative Learning Network Measurement and Learning Framework. You can read the first installment in the series here.

The past few years have seen a growing push for international organizations in the global development and social impact communities to support local systems strengthening through a catalytic-yet-indirect “intermediary” or “facilitator” role. This is a positive and necessary shift but it poses challenges in measuring and understanding catalytic contributions to systems change and impact. Following R4D’s first blog post on this topic, “Why Measuring and Learning about Collaborative Learning?”, the second blog by Collaborative Impact delves into the content of the framework itself and the process taken to co-create and stress-test the framework with the Linked Immunization Action Network (Linked) and the Strategic Purchasing Africa Resource Center (SPARC).

Traditional direct technical assistance and intervention models offer straightforward metrics for gauging success. However, the role of facilitator involves influencing and supporting without direct control over outcomes, making it more difficult to measure results. How can we effectively measure the indirect contributions of Collaborative Learning Networks (CLNs) to, for instance, health systems strengthening in developing countries and to building global public goods that can influence or direct development funding and practice? And how can we ensure that measurement informs stakeholders’ understanding of long-term systems change in ways that enable them to improve their collaborative learning practice?

How can it generate the evidence that funders need to know if their investments are making a difference; the insights that network managers need to grow and improve their networks; and the lessons that practitioners require to understand best networking practices for catalyzing system change in their countries?

To address these challenges, Collaborative Impact helped R4D to develop a measurement and learning (M&L) framework tailored to capture the multi-layered outcomes of its CLNs within and across different contexts. The framework was co-created and stress-tested through collaborative efforts with the lead facilitators and implementing partners of the Linked Immunisation Action Network and the Strategic Purchasing Africa Resource Center.

Phase 1: Laying the groundwork

Collaborative Impact started with painting a collective vision and framework for measurement and learning, drawing on R4D’s tapestry of CLN experiences and insights from the past 15 years, gathered through interviews and document review.

The vision for an integrated, inclusive and utilization-focused measurement and learning framework that is feasible and resourced was ambitious from the outset. To achieve this vision, the framework would need to be at the heart of CLN managers’ and partners’ daily routines, and therefore practical, adaptive and feasible accounting for their available resources and capacities. Furthermore, it needed to move beyond anecdotal evidence and be inclusive of members’ views in the CLN stories of success (and failure). The purpose thus was to develop a framework and toolkit that would empower CLN managers and partners to dynamically learn from, adapt, and enhance their strategies, while demonstrating the added value of CLNs to the world.

The proposed framework constituted an M&L user map; an actor-centered Theory of Change (ToC); CLN maturity rubrics; a set of prioritized M&L activities; methods and questions; and cyclical M&L timelines feeding into strategy, planning and implementation.

The Theory of Change offers a dynamic storyboard that visualizes the anticipated pathways to impact — from network performance to network-level outcomes across countries, to system-level outcomes in the members’ countries — all of which can be tailored to specific CLNs and their contexts in their geographic and thematic areas. CLNs seek to collaboratively generate knowledge and solutions that can be adopted and promoted by their members to influence system change agents’ policies and practices, mobilize resources, and build commitment and ownership among the broader supportive ecosystem actors in their countries, leading to desired systems reforms and ultimately to impact.

But network-level outcomes can only be achieved if CLNs effectively perform their functions. This requires successful strategy implementation for member engagement; knowledge creation, sharing and uptake of knowledge; CLN measurement and learning; strong partnerships; and resource mobilization. Network performance must be driven by active member engagement based on shared interest that evolves into shared experience, and ultimately, frequent and spontaneous member interactions based on shared intent and identity. Together with the network sponsorship, these constitute the dimensions of network maturity.

The CLN Maturity Rubric helps CLN managers and partners track the overall maturity of their network and select and customize M&L activities and methods from the M&L framework that are most feasible and relevant to their network’s maturity stage and context to maximize learning and strategic adaptation.

CLN maturity was conceptualized along four dimensions:

• Network sponsorship: the level of decentralization reflected in the relative dependence on a backbone organization, and funders’ trust in the network’s ability to deliver value for money.
• Shared interest/motivation: the balance of individual vs. shared interest, the patterning of member interactions, and whether knowledge generation is member driven.
• Shared experience: the extent to which members’ experiences draw on shared knowledge generation and mutual benefit, and are shared by members’ beyond the network.
• Shared intent and identity: the extent to which members have a shared sense of purpose and ownership, and identify with and champion the network externally.

Phase 2: Right-sizing and stress-testing with pilot CLNs

What does it look like when aspiration meets reality? With the vision and model collectively discussed and validated by R4D leadership and CLN managers, we moved to the pivotal phase of right-sizing and stress-testing with two self-selected pilot CLNs: the Linked Immunisation Action Network and the Strategic Purchasing Africa Resource Center.

These were not just test cases; they were collaborative ventures to build ownership through collaborative planning and refinement of key elements of the framework in a series of workshops and individual assignments with the pilot CLN leads, as well as to make necessary adjustments based on these pilot CLNs’ direct feedback and unique challenges in the process of right-sizing and stress-testing selected methods with CLN partners, guided by Collaborative Impact.

Right-sizing involved the adjustment of scale and ambition of the piloting process to make it feasible and fit with the pilot CLNs’ timelines and planned activities. Stress-testing concerned the deliberate testing of selected tools and processes to ensure their suitability and adaptability to different CLN contexts. Given the unique opportunities and constraints faced by Linked and SPARC, a flexible approach to right-sizing and stress-testing was paramount. For example, stress-testing could range from getting feedback from stakeholders on a given tool or method to implementing the tool or method at some scale. To enable right-sizing for stress-testing, partners were guided to first set their ambition and then provided with a set of sliders to help decide on trade-offs between rigor and feasibility considering a range of parameters.

SPARC and Linked were then taken through an interactive planning process to determine the specific approach to stress-testing their chosen methods. Light-touch support and exchanges during the process helped these CLNs get unstuck, overcome barriers and get the most out of the process.

“The stress-testing process was inherent to the development of Linked’s proof of concept,” R4D Associate Director Chloe Lanzara says. “By participating in stress-testing early on in our own process, we were able to explore different methods in a structured way and get a head start on right-sizing outcome harvesting as the key method used for generating evidence on Linked’s contributions to country-level outcomes. It was a two-way process that supported Linked in our own measurement endeavors, while enabling us to contribute to broader learning around how collaborative learning networks can measure progress and document impact.”

Phase 3: Collaborative sense-making with all CLNs

The focus of the final phase was on integrating the insights and experiences from the pilot phase — first with the two pilot CLNs, and then at an institutional level with all CLNs. Members of the Linked and SPARC teams reflected on their experiences with stress-testing their selected methods and tools, highlighting lessons learned and challenges faced, and drawing conclusions about the value of the methods and tools and the stress-testing process itself.

While each CLN team had distinct experiences, both found that they were able to get something out of the process, whether building a deeper understanding with key partners on pathways to change or generating outputs that could be used to demonstrate contribution to impact to a funder. Moreover, reflection on challenges faced and obstacles overcome presented the opportunity not only to refine the framework and its toolkit, but to understand the critical requirements for institutionalization of the framework.

A larger sense-making event was subsequently organized for participants from across R4D’s wider CLN practice/portfolio. This provided an opportunity for the Linked and SPARC teams to directly share their pilot experiences and insights with their peers; for representatives from other CLNs to consider the relevance of the M&L framework for their own CLNs; and, in considering the successes and challenges, for R4D to explore the requirements for institutionalization of the framework.

What was challenging, and what made it work?

This ambitious assignment was not without its challenges!

From the outset, the inherent complexity and diversity of CLNs created the challenge of developing a framework and language that could accommodate this. Past frustrations with attempts at M&L for CLNs had also resulted in some degree of skepticism — was it even possible to develop such a framework for CLNs and would it be worth the effort to do so? Finding time to work on M&L activities that had not been budgeted for at the level of individual CLNs was not without its challenges, and the term “magic-time” (to denote the extra effort allocated to working on M&L) became a powerful reminder of how challenging this was.

This intersected with the constraints imposed by a time-bound Organizational Effectiveness grant (that made this initiative possible but required delivering in a relatively short space of time), while having to navigate the already full calendars of participating CLNs and their respective members. This presented challenges in setting the ambition, determining who to engage and how, and creating adequate opportunities for engagement throughout the process to ensure a robust output validated and owned by R4D.

So what made it work?

The shared vision developed at the outset served as the “guiding star,” outlining the collectively agreed, non-negotiable principles that should guide the process and output. This was made possible through Collaborative Impact’s Participatory Impact Assessment and Learning Approach (PIALA) focused on collaboratively making trade-offs to enhance value addition and ensure feasibility and relevance to users. Multi-stage collaborative sense-making, moving back and forth between institutional and pilot levels made it possible to integrate experiences, build alignment and ownership and co-create the way forward.

Similarly, the combination of carefully crafted assignments with interactive sessions for all methodological work and the use of complexity-aware methods that could demonstrate value in a relatively short window were crucial. Last, but by no means least, the commitment of and championing by the sponsor team and CLN pilot leads were of absolutely fundamental importance.

Ultimately, the practicality of the framework lies in its utilization-focused nature, offering a flexible array of tools, guiding questions and methods that enable real-world application without overwhelming the users. It is designed to be feasible, focusing on must-haves that adapt to the maturity level of each CLN, ensuring no one is left grappling with tools that don’t fit their context or capabilities.