From October 2020 to March 2021, the Johnson & Johnson Center for Health Worker Innovation convened a pilot set of global partners for a 6-month community of practice to deeply engage with the topic of health worker resilience. The partners received funding from the Johnson & Johnson Foundation to dedicate time and resources to this learning community, carry out user testing with their health worker communities across India, Nepal, Kenya, and Rwanda, and to assist in the development of an evidence-based, free and adaptable mobile messaging framework designed to boost health worker resilience. The final result of this effort is a Resilience Message Program designed to be delivered to a health worker’s mobile phone over the course of six months. It is available to download for free, alongside guidance on translation and localization.
This article details the experience and reflections of Dimagi, one of the pilot community of practice partners, and serves as a useful resource for organizations looking to conduct user testing for messaging content as part of their larger process to design, implement, and evaluate a resilience-building program.
When you first start with a new intervention, there’s often a lot of uncertainty surrounding its potential for success.
You might ask yourself all kinds of questions:
- Is the scope of your idea too small to make a difference or too large to manage?
- Are the services you’re offering actually relevant to the program’s goals?
- Are the program goals relevant to your audience?
- Are you focusing on the right audience?
- How are you planning to evaluate your impact?
In Dimagi’s nearly 20 years of work launching digitally-enabled health programs around the world, we’ve found the best way to answer these questions is to conduct iterative user testing early and often. Essentially serving as ‘rehearsals’ before the main performance, iterative user testing can uncover critical issues and challenges, allowing for earlier course corrections. Organizations can test ideas, concepts, implementation strategies and evaluation protocols, providing insights about user experience, feasibility and acceptability before large scale rollouts.
Here’s how Dimagi conducted iterative user testing to inform content, design and delivery mode decisions for a behavior change resilience message program for community health workers (CHWs) in India.
Our Theory of Change
In early 2021, Dimagi participated in the Johnson & Johnson Resilience Collaborative pilot project, an initiative to develop and test messaging meant to increase the resilience of CHWs in low-resource settings.
We believed that by sharing regular, personalized and relatable communications, we could improve CHWs’ resilience in the face of the daily stresses of their jobs to protect against burnout. We called the project “Mala Didi,” named after the CHW persona we designed to deliver our messages.
At the outset of the user testing design phase, we knew the core learning objectives were to:
- Gather feedback on the content shared with CHWs, as well as their preferred frequency of interactions and method of receiving the messages (text/images)
- Collect early anecdotal evidence of these methods in helping CHWs manage stress and avoid burnout
- Identify any changes needed for the content, message methods, and data collection tools
Actively articulating our objectives before designing any aspect of the user testing helped our team keep the focus on participants, refine our data collection activities, and avoid scope creep (especially important when resources are limited).
Our Methodology
Over the years, we’ve found human-centered design to be a key principle, so we began our work with a series of formative field visits and interviews with CHWs in Madhya Pradesh. We gathered initial insights on localizing CHW resilience messaging and interventions in India for our Mala Didi program.
We put these insights to the test, one-by-one, across three phases:
Phase 1: The Content
In the first phase of user testing, using telephone interviews with 9 CHWs, we explored which key behaviors and concepts resonated (and more importantly which did not). We also gathered feedback on the likelihood of CHWs engaging in various stress-reduction activities.
EARLY INSIGHT TO ACTION: Messages were further localized to ensure their relevance with our CHWs, including specific examples of stressful situations CHWs had faced, ways to seek support, and clarity for more abstract concepts like “mindfulness."
In the second phase of user testing, we tested CHWs’ voice, tone and delivery preferences via in-person interviews with 14 CHWs. The results showed us that 11 out of 14 CHWs preferred messages written in the tone and voice of a health expert, because the tone was friendly and users felt that they could relate to the messages.
EARLY INSIGHT TO ACTION: Because of this, we concepted a digital persona named “Mala Didi” who embodied this formal, but personable tone.
We polled 14 CHWs on the optimal delivery method for these messages. All users confirmed a preference for WhatsApp.
EARLY INSIGHT TO ACTION: WhatsApp would be the most ideal platform for further usability testing.
Using WhatsApp as the messaging platform, we entered a one-month period of user testing with 10 CHWs. To ensure the contextual relevance our CHWs needed, these messages included personalized multimedia, such as an image overlayed with a reminder message related to resilience. The adaptability of the platform also meant that we would be able to analyze the acceptability of our messages, the likely impact of these messages on CHWs, and considerations for scale.
For our user testing, we also tested the following evaluation methods:
- Baseline assessment survey: In-person interviews assessed the technical literacy of the participants, their preferred time of the day for receiving messages, their current stress levels, and any existing coping mechanisms for the same.
- Daily message tracker: Once we started sharing the messages, our team maintained a tracker to record daily read receipts and interactions over WhatsApp. The percentage of messages read by the CHWs ranged from 74% to 100%, with most CHWs reading around 87% of the messages sent by Mala Didi.
- Weekly check-ins: At the end of each week, a pulse check was conducted with all the CHWs to gauge their initial responses to the messages. The participants were sent a series of five questions over WhatsApp and were asked to reply over text. In general, as CHWs got further along in the check-in process, their response rate decreased.
- End-line assessment survey: A final survey was conducted via telephone interviews to assess the perceived impact of the messages, openness to receiving multimedia with the messages, and connection with Mala Didi.
EARLY INSIGHT TO ACTION: Message-based check-ins will see better response rates with a limited number of questions (likely one or two).
Takeaways
The three-phase iterative user testing provided us with early insights resulting in early actions to improve the design and evaluation of a scaled-up behavior change resilience message program for CHWs. Specifically, sharing multimedia content added a personalized touch to the content that was shared and helped CHWs retain key takeaways from the resilience messages shared.
It encouraged us to think extensively about the localization process and how the messages can be made more relevant to the workers by focusing on translation and updating the content with recognizable examples from their day-to-day work. Additionally, topics which are sensitive for the audience were explained using examples.
It also allowed us to assess usability, the channel of communication, and the tone in which the messages were delivered.
By testing multiple methods of data collection (in-person, messaging, and phone calls), we gained a better idea of how a full-scale deployment of the initiative might employ these measurement tools to continually analyze their impact.
In conclusion, we believe that user testing provides a cost-effective approach to identify key design and evaluation adaptations that may be needed before the implementation of a larger scale program.
For more information and learning opportunities, consider joining The Resilience Collaborative, the newly launched global learning community that aims to advance learning and drive adoption of evidence-based strategies for health worker resilience, particularly in low-resource settings.