Moving beyond boring baseline assessments

Ah, the baseline. The forgotten child of researchers everywhere. Everyone’s least favorite, easy to forget, “let’s get this over with” part of evaluation and learning.

Researchers do it because we need something to compare the real star (endline data) to after practitioners do the things they do. And for many of us, our motto with baseline data is “no news is good news”…if the baseline tells you that things are pretty much exactly as you anticipated, then you sigh a little sigh of relief and tuck it away in a folder until you have to run your pre-post analysis.

But baselines have hidden talents, if we only stop using them simply as a point of comparison. And that was our experience in Kenya for the Strengthening Mixed Health Systems (SMHS) project, which R4D began in 2019 with support from Merck for Mothers.

Full disclosure: we actually set out to do a boring baseline. We had a research question: “what factors help and hinder the success of public-private engagements to improve maternal and newborn health.” We had a plan to provide third-party support and engagement brokering between the sectors in two sites in Kenya and India (more on India at the end). We even had a framework of factors that we wanted to test for, but that was for the endline. The baseline was just something to compare the endline to later.

But then…things happened. Well, one really big thing, specifically a global pandemic. And that was when we realized, maybe the baseline needs to be something more than we originally planned. This led us to a few different times and ways baselines can be stars — or at least really important supporting actors.

Role 1: The informant (when things go sideways) 

Our approach in Kenya was comprised of several key activities, co-led by partners Insight Health Advisors and described in more detail here. In pursuit of improved health outcomes by fostering engagement between the public and private sector, we identified champions in each sector, worked with them separately on the challenges they faced, facilitated a workshop with representatives from both sectors to identify root causes of maternal health problems, and then provide support as those partners pursued actions to address those problems that they identified.

And those partners did identify problems and design actions to prioritize in the health sector during a workshop…in February 2020. This meant that, one month later, every health priority and resource and role was turned on its head due to COVID-19.

To the extent that our baseline was supposed to provide a basis of comparison for what we did over the next 12 months later, the pandemic erased this plan — because any change we observed 12 months later could be related to our program, but more likely it would have something to do with COVID.

However, the baseline did provide us with important information about where partners were starting, in terms of environmental factors and structural factors they believed might affect how the program rolled out as well as relational or engagement factors that could be leveraged or be a challenge.

For example, the baseline in Kenya revealed some important staring points, such as:

• Many of the gaps presented in one sector were things that the other sector revealed as things they had well-covered. From specialized equipment to human resources to monitoring health data, these complimentary haves/needs presented clear opportunities for engagement between the sectors.
• At the same time, the baseline surfaced significant misconceptions and gaps in mutual understanding between sector actors, a key hurdle that was important to know early in the process. For example, both sectors raised several points about resource availability and flexibility within the opposite sector that appeared to be incorrect when we asked the other sector about these perspectives.
• Finally, while partners cited many “hard” factors for success (resources and legal arrangements and referral processes), the baseline revealed that “soft” factors, such as will to engage and shared vision were raised as frequently. And for many of these factors, such as joint planning and implementation, the baseline interviews also flagged that these were big gaps in how the sectors were currently engaging.

While these findings were critical to our experience in Kenya, they also reveal more universal lessons for those working on public-private engagements, highlighting the need for those supporting engagements to consider where there are complementary “haves” and “needs” across partners and how some of the “softer” factors may be critical and foundational for engagements between the public and private sector.

Role 2: The Hypothesis Tester (when you think you know, but maybe you don’t)

The factors that we analyzed as part of the baseline did not come out of thin air. These were all components of the environmental context or structure of public-private engagements or how partners engage that we had seen from our experience — and from our review of the literature — were likely to play an important role in the success of public-private engagements.

And before we even conducted the baseline, we compiled this experience and evidence base to develop an ecosystem of the types of factors that were emerging and how they were related.

But this ecosystem was still a theory in February 2020. It was evidence-informed, but it was not something we had begun to gather primary data regarding. And while our hope was that we could use our endline analysis in Kenya, India, and other sites to assess whether these factors changed over the course of the SMHS project support, there was also a more basic question that we needed to ask about this hypothesis — did we even have the factors right? And the baseline was the first set of primary data we were able to use to start answering this question.

Baseline interviews with public and private sector representatives in Kenya helped us identify new factors we missed in our original hypothesis and helped us refine ones that needed more nuance. Factors such as accountability and joint leadership and power across the sectors and the distinction between broad political will for public-private engagements and specific will to engage expressed by the partners — these all evolved and emerged because of what we heard from baseline respondents in terms of things they expected to help the SMHS project succeed and things they viewed as likely obstacles.

Again, while this experience is specific to the SMHS project, our learnings from utilizing the baseline to validate and revise our factor hypothesis is not an approach that would only work in this project. Using this type of baseline even as formative research can provide valuable insights for a myriad of research questions, both within and outside of the mixed health systems space.

Role 3: The Adapter (when data can and should inform action)

At the end of the day, the best reason to collect data is to inform the actions that we take. Traditional baselines do serve this purpose — but in a round about way, by serving as a comparator after we collect data at the end of the project. But this is also one of the most frustrating things about traditional baselines: that they have so much information that they can reveal, but when implemented as part of an impact evaluation, that information often just has to be ignored until the endline. And when you want to know if this program impacted that outcome, then you just can’t jump in with a new program, just because the baseline suggested that the new program might be warranted.

But when your baseline becomes obsolete as a pre-test because the world changes so dramatically after you conduct it, then that data can help you adapt to meet the moment — which is what happened in our Kenya work.

For example, many of the actions that the partners designed in February 2020 dealt directly with health problems — training on engaging with the National Hospital Insurance Fund, agreements to share resources across the sectors for better quality.  But the baseline revealed big gaps in past engagement between the sectors.  For example, several respondents cited that the private sector was fragmented and thus would have difficulty speaking with a collective voice to public sector health managers.

And both sectors acknowledged that the private sector did not have a seat at the table even for big strategy discussions, such as annual workplanning for the county.

None of these issues were as close to “health outcomes” as the actions originally planned, but they did represent really big roadblocks that could stand in the way of things like “resource sharing agreements” advancing or being utilized.

The need to pivot (COVID-19) was something that was outside of our — and everyone’s — control.

But the baseline provided an additional resource in Kenya that our partners could use to be very deliberate about that pivot — shifting their actions not just to the pressing health needs related to the emerging pandemic but also the need to strengthen mutual understanding and joint leadership and other engagement factors that might help guide the success of their work together.

Role 4: The Global Informant — for our project and for others (when you can’t even finish a baseline before thing go sideways)

The final way we used the “ruined” Kenya baseline was not in Kenya at all. We had designed the SMHS project to pursue two parallel, if slightly staggered, approaches in Kenya and India.  The programs were slightly different, as were the staged in the engagements (in India, we were working with an existing, if relatively new, program), but much of our approach and our evaluation and learning plan was the same.

But, with the work in India lagging a few months behind Kenya, we did not have the opportunity to launch a baseline in India before we had to pause everything as COVID numbers rose.

While there are myriad differences between the Kenya and India programs as well as context, we still sought to take what we did have from the Kenya baseline to inform our plans in India. This included taking our hypothesized factor ecosystem, adapted and strengthened from what we learned from the Kenya baseline, and creating it into a “progression model” that helped partners themselves identify strengths and potential areas for development in their existing engagement. This is something we were able to pilot and are now validating with partners in India, and it is something that may not have been a part of this work but for that “ruined” Kenya baseline.

Three lessons for mixed health systems researchers

Our baseline experience reveals a few lessons for researchers and practitioners working on mixed health systems strengthening that apply beyond the confines of this project:

• Researchers and practitioners alike should look beyond traditional outcomes to the factors that may influence outcomes. Every public-private engagement experiences shocks (even if it isn’t a global pandemic), and collecting data from the start on the factors that help and hinder engagement can help partners mitigate or at least lessen the impact of these shocks.
• Researchers should consider whether it makes sense to actually use the baseline as formative research. While some programs are well-placed for conducting large scale rigorous impact evaluations, projects operating in complex environments and complex systems may benefit more from using early data collection to inform the project design itself.
• Practitioners — including governments and private providers — don’t need to rely on researchers to utilize data for program design. Some of the most valuable data out there is not research-driven; it is driven by the needs and interests of those leading programs and engagements and is used to make those engagements stronger. As an added bonus, this type of data doesn’t fall off track when global shocks hit.

This is by no means a manifesto to stop doing the types of baselines that can help explain the impact of programs; there is a critical place for a Baseline Role 5 (the Means to Assess Impact). But there is a world of things baselines can help us understand and strengthen and do if we step outside the box, especially when things don’t go exactly as planned.

The Strengthening Mixed Health Systems project is supported by funding from Merck, through Merck for Mothers, the company’s global initiative to help create a world where no woman has to die while giving life. Merck for Mothers is known as MSD for Mothers outside the United States and Canada.

Photo © Irene Angwenyi/USAID Kenya