Topic 05
How to measure the impact of Collaborative Learning
As discussed in Topic 03: The Collaborative Learning Theory of Change, Collaborative Learning Networks (CLNs) seek to contribute to long-term systems strengthening and improvements in development practice.
But how do we know if this is happening?
The Collaborative Learning Network Measurement & Learning (M&L) Framework was produced by Collaborative Impact in collaboration with R4D as a tool to support practitioners to assess and improve the performance of CLNs and to measure their contributions to impact.
This framework can be used for (1) adaptively managing and improving the performance of an individual CLN, (2) measuring and learning across multiple CLNs, and/or (3) assessing a CLN’s progress towards achieving in-country systems change.
Key elements of the M&L Framework include:
- The Collaborative Learning Network Theory of Change (ToC) helps track a CLN’s performance and contribution to systems change, aggregate findings and draw lessons across a CLN portfolio.
- Prioritized M&L questions to guide the M&L activities and help draw lessons within and across CLNs.
- A guide to assessing CLN maturity stage, which ultimately helps determine the appropriate M&L priorities and activities.
- A list of must-do M&L activities that describe the main measurement and learning events that need to happen to improve CLN performance and demonstrate the added value of the Collaborative Learning approach.
- A guide to refining and testing the M&L activities to ensure they are feasible, viable and able to generate the desired information.
- A planning guide for M&L that outlines key roles and responsibilities for M&L activities and how to budget for these activities.
1. A Theory of Change for Collaborative Learning Networks
A Theory of Change articulates the various levels of change a Collaborative Learning Network (CLN) is expected to contribute to. It provides a shared language and conceptual map that can anchor measurement and learning.
Each CLN typically has its own individual Theory of Change with network-specific terminology, details and result statements. It’s essential for each CLN to develop its Theory of Change before determining methodological options for measurement and learning. The Theory of Change should include two main components:
- CLN performance consisting of network strategies, outputs and outcomes. This section defines the core CLN activities (e.g., around knowledge uptake, network maturity and sustainability, member engagement, and facilitation capacity) and the shorter-term outputs and outcomes that result directly from those activities.
- CLN contribution to impact consisting of longer-term system outcomes and contributions to impact on the lives of target populations. This section of the TOC might include outcomes related to ecosystem coordination, changes in policies and practices, etc.
To learn more about the CLN Theory of Change, refer to Topic 03 in this toolkit or dive into the full M&L Framework.
2. Prioritized M&L questions
After developing a Theory of Change, it’s important to develop M&L questions. Having clear M&L questions from the outset helps to define the objectives and goals of the monitoring and learning process and ensures that evaluation and learning efforts focus on the relevant aspects of the CLN. The M&L questions also inform data collection, enabling the measurement of progress over time and promoting learning and reflection. Ultimately, the evaluation methodologies that are chosen should answer the M&L questions.
The table below presents a set of prioritized M&L questions when thinking about CLN performance and CLN contribution to impact.
| M&L Questions Related to CLN Performance | |
|---|---|
| Questions at the network level: | Which models or strategies for member engagement are most effective in the different stages of network maturity? What types of learning topics are best addressed through Collaborative Learning? How do we best identify these learning topics? Which models or strategies for knowledge creation, sharing and uptake are most effective? Which peer learning facilitation formats are most effective and useful? What are the capacity requirements for the technical facilitation partners to make strategies for knowledge creation, sharing and uptake effective? |
| Questions at the member/country level: | What determines the uptake and adaptation of knowledge in the countries? What types of interim outcomes can be early indicators of Collaborative Learning success towards achieving desired network outcomes? What are the most important indicators of network health and sustainability? |
| M&L Questions Related to CLN Contribution to Impact | |
|---|---|
| Questions if looking across multiple CLNs: | How does Collaborative Learning contribute to system change and ecosystem strengthening? Under what conditions does Collaborative Learning effectively enable the translation of knowledge into action? |
| Questions at the member/country level: | How is the CLN contributing to system reform and ecosystem strengthening in the target countries? What are the most effective impact pathways and strategies? |
3. CLN stages of maturity
M&L activities should be adapted to the characteristics and level of maturity of a CLN. The CLN M&L Framework provides a rubric tool to assess maturity along four different dimensions (listed below). For each dimension, there are key variables (bulleted below) which can be used to determine whether the CLN is in the Immature Stage, Early Maturing Stage, Advancing Maturing Stage or Highly Mature Stage.
Network sponsorship
- The extent to which the structure of governance, coordination and facilitation is decentralized, and the network backbone organization is supporting the network from the outside (versus leading from the center)
- The extent to which the network funders trust the network’s ability to generate tangible outcomes and deliver value-for-money and consequently are more flexible and less directive (versus more restricted and directive in their funding)
Shared interest / motivation
- The extent to which member engagement is driven by ‘shared interest’ that draws on collective member interests and needs (versus individual member interests and needs)
- The extent to which members interact frequently, collaboratively and freely/ spontaneously (versus ad-hoc, consultatively and formally)
- The extent to which knowledge creation and innovation is member driven (versus sponsor-driven)
Shared experience
- The extent to which members’ ‘shared experience’ draws on collectively (versus individually) built knowledge and insights
- The extent to which members’ experience involves mutual and collective benefits as a result from participation in the network
- The extent to which members share their experiences and benefits with professionals and peers outside their network and countries
Shared intent and identity
- The extent to which members have developed a shared sense of common purpose and take ownership of the network functioning and outcomes
- The extent to which members identify with the network and communicate its common purpose and identity
See the entire rubric (with full descriptions of each stage of maturity) and explore the full CLN M&L Framework.
4. The ‘must-do’ M&L activities
Once the appropriate M&L questions have been developed and the CLN stage of maturity has been determined, measurement and learning activities can begin. M&L activities are the specific actions taken to generate evidence on CLN performance and contribution to impact. The outcomes of these activities should answer the M&L questions.
The table below outlines several ‘must-do’ or prioritized M&L activities at the portfolio level and the individual CLN level.
| M&L Activities Related to CLN Performance | |
|---|---|
| Activities at the network level: | Monitoring network maturity Assessing the effectiveness of partnerships at critical moments of network maturing Periodic CLN learning and reflection moments around network performance Monitoring routine engagement and knowledge outputs Performance monitoring of/by technical facilitation partners |
| Activities at the member/country level: | Tracking changes in leadership, relationships and capacity of ecosystem actors Monitoring knowledge uptake and implementation |
| M&L Activities Related to CLN Contribution to Impact | |
|---|---|
| Activities at the network level: | Structured country impact case studies |
| Activities if looking across multiple CLNs: | Independent impact evaluations Periodic learning and reflection events across multiple networks |
To learn more about the prioritized M&L activities, including at which stage of maturity each activity applies, the proposed timeframe and frequency of the activities, and the corresponding roles and responsibilities, explore the full CLN M&L Framework.
5. Right-sizing and stress-testing
After the selection of M&L activities, it’s essential to subject them to a process of refinement and testing. This ensures their feasibility, viability, and ability to generate the desired information.
Right-sizing is the process of changing the size/scope of something so that it works better for a given purpose and context. Thus, with respect to the CLN M&L Framework, right-sizing refers to adjustments that can be made to M&L activities and methodologies to ensure they are appropriate and feasible within the time and resources available. Paring back the methodologies may compromise rigor and the resultant quality of outcomes, yet striking a balance is imperative when time, budget and available team resources are constrained.
The graphic below illustrates example tradeoffs to consider when right-sizing a methodology to determine the ideal balance between level of effort vs. level of rigor.
After employing right-sizing to identify the most suitable version of methodologies for their context, the team can begin to test them using stress-testing. Stress-testing is the process of deliberately testing functionality given various situations or scenarios. It allows teams to begin gathering information in order to make adjustments, before rolling out the chosen M&L activities on a larger scale. Stress-testing can range from probing of initial concepts, prototyping working models, or piloting through a larger-scale test. For the CLN M&L Framework, we focused on lighter touch prototyping of our M&L activities and methods.
Stress-testing M&L activities and methods helps to determine whether they:
- Are well designed, coherent, relevant and can work across different contexts
- Are feasible to implement in terms of capacity, cost and time requirements
- Enable the team to generate the required data/evidence that is credible, accepted by and useful to relevant stakeholders
- Can yield insights responding to the Learning Questions and support learning among key stakeholders
- Can help teams to test and deepen their understanding of aspects of the ToC that they correspond to
The more situations and pressures the methods are subjected to, the more confident teams can be about using them.
Explore the full CLN M&L Framework.
6. Planning for M&L
Conducting effective M&L activities requires careful planning to determine the right timing, outline the roles and responsibilities of key actors, and appropriately budget resources for the activities.
M&L timing
The following questions can help determine timing for M&L activities.
- Is there demand for M&L from the donor?
- What resources are available for M&L and what would they permit?
- Is the network at an appropriate stage in its cycle (e.g., entering a new stage, going through a review)?
- Are there clear gaps or value propositions to be addressed?
- Does the team have capacity in terms of time, resources and skills?
- Can M&L be integrated with other, already planned activities?
M&L roles and responsibilities
M&L activities typically involve a team of people, including the CLN manager, CLN members, in-country partners and independent consultants. The following table outlines some of the roles and responsibilities of key actors.
| Roles and responsibilities when assessing a Collaborative Learning Network (CLN) | |
|---|---|
| CLN manager | Conduct M&L readiness assessment (using key questions above) Contract external consultant, if needed |
| CLN manager (with appropriate technical assistance if required) | Develop a CLN-specific M&L framework Develop a CLN Theory of Change and CLN learning questions Prioritize relevant M&L activities and methods Plan and allocate resources for M&L Implement CLN M&L plan including: -Piloting, adapting and stress testing selected M&L activities -Preparing documentation, capacity development and roll-out plan -Integration, improvement and within-CLN scaling of useful M&L activities Organize and facilitate learning sessions Produce learning briefs Lead knowledge management of evidence generated |
| Partners and consultants | Collect and analyze data Produce reports Organize and facilitate learning sessions |
| Communications team or consultants | Edit reports and learning briefs Produce evidence-based impact stories |
Budgeting for M&L
Effective budgeting for M&L activities considers time, money (including for personnel, events, knowledge products, etc.) and expertise. Usually, an M&L budget constitutes 5-10% of the overall program budget.
When budgeting for M&L, be sure to:
- Calculate personnel costs accurately
- Estimate resources for bringing in additional expertise and coaching support (e.g., evaluation consultants)
- Estimate resources needed to facilitate stakeholder participation (e.g., managers, CLN members, partners, funders and external experts) in M&L processes (e.g., consultations, focus groups, surveys, sensemaking, learning events, contributions to reports and communication products, uptake or application of M&L insights/results)
- Allow for contingencies and complexity/unpredictability, acknowledging that CLN environments are dynamic, and that flexibility and responsiveness are of equal — or even greater — importance than good planning in such environment (e.g., by estimating the high and low scenario costs and setting aside the difference in funds for unexpected changes in design or execution).
For more details on planning for M&L, including sample plans for various M&L scenarios, visit the full CLN M&L Framework.
