How to use lean testing to design and improve a program

In a field that is accustomed to process evaluations and randomized controlled trials, I often get asked “what is lean testing?” from international development colleagues.

In short, lean testing is a process of “running many small tests and adjusting efforts after discovering what works and what doesn’t.” It can be a valuable tool for implementers who are unsure about how to design an intervention; the rapid nature of lean testing allows for course correction during piloting and implementation instead of waiting until the end of the project.

At R4D, we have been using lean testing as part of our Evaluation and Adaptive Learning approach, including in our USAID-funded Rapid Feedback Monitoring, Evaluation, Research and Learning (RF MERL) program.

Our RF MERL partnership with Lecture Pour Tous (LPT) in Senegal, an early grade reading program supported by USAID/Senegal and implemented by Chemonics International, is a great example of how lean testing is effectively applied in practice.

Among its various activities, LPT aims to facilitate teacher coaching and peer-to-peer learning by giving teachers, directors and inspectors SIM cards for mobile phones that allow free calling and texting. LPT is also sending teachers and directors motivational and informative text messages to improve teaching in classrooms. A motivational text would say something like, “Congratulations for the great results your students achieved in the last test!” An informative text, for example, would remind teachers to point out the letters and syllables when reading aloud to students. Before scaling the activity to all regions in Senegal, LPT wanted to understand whether teachers and directors are using the LPT SIM card correctly, whether they are facing any challenges, and whether they understand the content of text messages.

To that end, R4D and members of the RF MERL Consortium — USAID/Senegal, LPT, R4D and Mathematica Policy Research (MPR) — used lean testing to help answer those questions.

What we tested

Over two weeks, a team of six researchers from R4D, MPR and Chemonics visited schools in Matam and Kaolack regions and interviewed 13 school directors and teachers. At each school, we conducted usability and message tests.

1. To test whether teachers and directors were able to properly use the SIM card, we guided respondents through a series of usability tasks. In real time, a member of the RF MERL team sent messages to the user to test (a) message receipt and (b) the respondent’s ability to open the message. We also asked respondents to open saved messages, type a new message, and send messages to the interviewer. We timed each task and observed any challenges users faced.
2. We also tested the content of future messages LPT was planning to send to determine whether the messages were being internalized in the way that LPT intended and whether they were likely to lead to the desired behavior (e.g., developing a work plan to prepare for an upcoming class) from the user (teachers and school directors). We conducted these tests by sending users six pairs of messages with similar content. Each message pair conveyed the same information but with variations in wording, length, tone, and degree of specificity. We then asked respondents a series of questions about each message, such as what the message means, whether there was anything they didn’t understand, how it relates to their role, whether they find it useful, and what kind of feelings it provokes.

5 key findings

The results of the lean testing were illuminating. Almost all users wanted the messages to be clearer and simpler than currently drafted. From these findings, we were able to identify a set of principles for what made for an effective text message. We were able to better understand how teachers and school directors use technology for work, whether they find LPT push useful messages, and how they are using the SIM cards to communicate with colleagues. We also noted and signaled to LPT technical issues related to late or undelivered messages. Some of the things we found were:

• Most respondents use some form of ICT – whether accessing the internet through smartphones, using a personal computer to prepare for lessons, or calling colleagues – for work.
• In general, users are able to perform basic tasks such as sending and receiving messages through their smartphones, and younger respondents displayed a higher level of “ICT fluency” than their older peers.
• Teachers and school directors prefer to call rather than text one another, citing that calls are more efficient, easier to perform, and provide more direct communication than text messages.
• Most users prefer to receive push messages outside of work hours to prevent distraction and allow enough time to prepare for class the next day. All users wanted to respond to the automated push messages.
• Users consider push messages to be reminders, not new information. Some users were interested in receiving messages in local languages.

At the end of the two weeks, the RF MERL team presented these findings to LPT and USAID/Senegal so they could immediately address technical issues. LPT is using the findings from the Lean Tests to course correct and prepare for the next phase of the activity.

What we learned: 3 principles for successful lean testing

Lean testing in Senegal wouldn’t have been as fruitful if we hadn’t followed three key principles:

1. Have implementing partners join and actively engage in lean testing. Although partner participation needs to be carefully managed so as to not influence or bias user response, it is an effective way to get partner buy-in to the approach and findings. In Senegal, having the support of LPT’s Senior Monitoring, Evaluation and Learning Specialist was invaluable as we were able to get LPT’s direct input in the iteration and development of new messages.
2. Iterate and don’t be afraid to explore tangential points of inquiry. For many researchers who are trained in conducting rigid evaluations, lean testing can be uncomfortably fluid. However, it is important to remember that lean testing is specifically designed to be adaptive. This approach can lead to the discovery of important but unanticipated findings. In Senegal, we tweaked messages mid-interview, at times, by switching a word or using a different pronoun to explore whether these changes were better received. Sometimes the team veered off the interview protocols for a while to explore unanticipated but emerging themes and reactions from respondents. This flexibility provided a rich and nuanced understanding of the challenges users face and allowed us to generate relevant and contextualized solutions.
3. Debrief with partners immediately after data collection. After an intensive period of lean testing, it can be difficult to synthesize the collected data and summarize findings to partners. In Senegal, we prepared for the debrief by taking notes of emerging findings at the end of each field day following discussions with the data collection team. We found that presenting even a set of “emerging” findings at the debrief was helpful for LPT and USAID/Senegal as it provided an immediate sense of key findings and allowed them to act promptly on pressing issues. A more complete findings memo and Learning Check workshop were later produced and presented to partners.

Lean testing is a flexible approach to tinkering with the design of a new intervention. To the international development practitioner anxious about designing and conducting lean tests, I hope lessons from Senegal can provide some clarity and guidance. In a practical sense, lean testing can be applied effectively if it is done in collaboration with partners, if the principle of flexibility in data collection is embraced, and if emerging findings are presented immediately following field work.