- RCT:
- randomized controlled trial
- ToC:
- theory of change
- WHO:
- World Health Organization
Background
Complex interventions—those that contain several interacting components—to improve clinical outcomes and population health are growing because of the increasing recognition that multilevel approaches are needed to solve complex problems in health systems and care delivery. (1) In maternal and neonatal health, complex interventions are needed to improve quality of care. Yet few studies of these complex interventions are powered to detect true differences in mortality because of the numbers needed for a relatively rare outcome, such as maternal mortality. Although perinatal mortality is more common, large sample sizes are still required. Studies are often powered to examine outcomes or process measures proximal to mortality, such as morbidity or complications that could lead to mortality. The BetterBirth Trial was one of the largest cluster randomized controlled trials (RCTs) to target maternal and perinatal mortality as primary outcomes. Given its scale, this trial has led to important lessons learned that could be applied to other large-scale studies with complex interventions in maternal and newborn health.
Many epidemiology and biostatistics textbooks have described how to theoretically design RCTs to maximize scientific rigor. (2) However, there are few, if any, resources on how to actually implement complex interventions on a large scale, evaluate their impact, and disseminate the results. (3) Balancing an unbiased and robust study design with the reality of implementing a complex intervention in the real world can pose ongoing challenges for research teams. Some generally accepted best practices to minimize common challenges of trial implementation do exist, such as for institutional review board navigation, site selection, and stakeholder engagement. This perspective reflects on the experience of the BetterBirth Trial and provides insights around best practices of important components of trial design and implementation that may be initially overlooked.
The BetterBirth Trial
The BetterBirth Trial was a matched-pair cluster RCT designed to establish whether the BetterBirth intervention was effective in reducing early (7-day) maternal mortality and morbidity and perinatal mortality in Uttar Pradesh, India. (4) The BetterBirth intervention was developed to improve the quality of intrapartum care through the World Health Organization (WHO) Safe Childbirth Checklist (WHO Checklist) and accompanying implementation strategy focused on facility leadership buy-in, staff engagement, peer-to-peer coaching, and data feedback. The trial included a total of 120 primary-level facilities and enrolled approximately 160,000 woman-newborn pairs. While adherence to essential birth practices increased among birth attendants in intervention facilities, there was no impact on maternal mortality and morbidity or perinatal mortality. (5) Given the scale of this trial, we propose a roadmap of practical lessons learned to mitigate or plan for similar challenges in designing and implementing other studies around large-scale complex interventions: partnerships and collaboration, theory of change (ToC), formative work and pilot testing, intervention implementation strategy, data quality assurance systems, and dissemination strategy.
Partnerships and Collaboration
A foundational principle of global health research is partnership with local experts through designing, testing, and communicating results. (6) To implement a trial of this size, scope, and scale, we established a collaboration among research, implementation, and government partners, which was essential to the success of the study. Ariadne Labs, a joint innovation center at Brigham and Women’s Hospital and Harvard T.H. Chan School of Public Health, led the research in collaboration with Community Empowerment Lab and Jawaharlal Nehru Medical College in India. The Governments of India and Uttar Pradesh collaborated on study design and implementation, leading the study’s tripartite agreement for implementation. Key priorities in our collaboration were rigorous methodology, high-quality data collection and interpretation, and dissemination of results locally and globally through context-specific communications. WHO provided a global policy perspective and aided in development and early testing of the WHO Safe Childbirth Checklist, and Population Services International implemented the study in Uttar Pradesh. The trial funder, the Bill and Melinda Gates Foundation, participated in decisions about study location and design, but did not provide input on data collection, management, analysis, or interpretation of the data. Their partnership was important for understanding the funding landscape and learning about other projects in Uttar Pradesh that may have influenced the BetterBirth Trial. In addition, we established a scientific advisory committee of researchers, clinicians, and public health professionals that provided oversight for the study and supported interpretation of the results.
Recommendations.
Clearly outline roles and responsibilities with established deliverables for optimal collaboration.
Utilize multidisciplinary expertise because it is deeply valuable for complex interventions and results in stronger interventions and implementation approaches.
Clarify roles around communication of the study, including peer-reviewed authorship, early in the collaboration and revisit as needed.
Develop an advocacy plan with policy leaders and key stakeholders who are not part of the study team but who will be affected by the study results.
Maintain open communication with funders about deliverables and budget limitations, including possible modifications to either based on learnings along the way.
Theory of Change
A ToC is a “theory of how and why an initiative works, which can be empirically tested by measuring indicators for every expected step on the hypothesized causal pathway to impact.” (7) This is often represented as a visual map of the underlying hypothesis behind the research question for complex interventions and program evaluations. The BetterBirth Trial had a basic ToC: the BetterBirth intervention (WHO Checklist implemented through facility engagement, peer coaching, and data feedback) would lead to increased adherence with the 28 essential birth practices listed in the WHO Checklist, and this behavior change would lead to improved maternal and newborn outcomes (Fig). This ToC, however, was overly simplified and did not account for every expected step between the increased adherence to practices and decreased mortality and morbidity.
BetterBirth theory of change. Graphic image courtesy of Ariadne Labs.
Recommendations.
It is a valuable exercise to develop a comprehensive ToC for any complex intervention study or program evaluation.
The ToC should be reviewed and refined with all relevant stakeholders, including the research/evaluation team and the implementation team.
Map every primary and secondary outcome and their measurement to the ToC.
Ensure that research and implementation teams understand the ToC.
Use consistent language and develop a communication strategy around the ToC to drive the research and implementation forward, especially when multiple teams are involved and staff turnover occurs.
Anticipate potential lessons learned if the hypothesis is proven true and if it is proven false. Develop the ToC so that it can help with interpretation of the study findings, regardless of whether they are positive or null.
Formative Work and Pilot Testing
Nearly all complex intervention studies require a phase of formative work or background research to understand the relevant contextual factors for the study site(s). This phase may occur while developing the grant proposal and may continue through the beginning of the grant period. Subsequent pilot testing may take place depending on the stipulations of the grant. In preparation for the BetterBirth Trial, formative work was done with a diverse group of experts and stakeholders around developing the items on the WHO Checklist. (8)
Once the content of the WHO Checklist was established, a pilot pre-post intervention study was conducted in Karnataka, India, in 2010, which demonstrated improvement from an average of 10 of 29 essential birth practices to 25 of 29 after implementation of the WHO Checklist. (9) This pilot study provided promising results that supported the first part of the underlying ToC at a subdistrict-level hospital in Karnataka, India. However, the BetterBirth intervention needed to be adapted for primary-level facilities in Uttar Pradesh. The BetterBirth team revised the intervention through an adaptive design process with implementation, evaluation, and feedback in Uttar Pradesh. (10) An important lesson learned through this process was the importance of peer-to-peer coaching instead of physician-led coaching to support behavior change in birth attendants.
Recommendations.
Formative work around the context at study facilities is essential for understanding and defining the problem. Results from this work should then inform the design and implementation of any complex intervention, especially those that depend on behavior change. Examples of topics that should be explored for intrapartum quality improvement interventions include birth attendant competency (knowledge, skills, and attitudes), women’s preferences around childbirth and relevant societal norms that may influence health-seeking behavior, motivation and incentive structures in health facilities, leadership and management structures, referral systems, and workflow analysis regarding how the intervention would integrate into existing workflows.
If possible, a pilot study should be conducted in a similar context to the one that will be used for the large-scale trial.
Gathering qualitative and quantitative information during the pilot phase is critical because it is an opportunity to fail quickly and learn deeply. Observations, workflow mapping, interviews, and focus group discussions can provide insights into how the intervention functions in real-world circumstances beyond the information collected in quantitative surveys.
Some complex interventions may require multiple adaptation cycles to ensure they are optimally integrated into the workflow and organizational culture of the facilities.
While some adaptations may be required, it is important to consider that making too many changes at once may result in an inability to identify which change was most important.
Intervention Implementation Strategy
Complex interventions developed for large-scale trials or evaluations require clear implementation strategies so that they are implemented in a standardized way. The BetterBirth team developed an implementation guide that outlined the distinct phases of intervention implementation: Engage, Launch, and Support. (11) First, the Engage phase included gaining commitment from facility, district, and state leadership for implementing the BetterBirth intervention. This phase also included adaptation of the WHO Checklist to Uttar Pradesh Ministry of Health guidelines. Second, the Launch phase included a motivational event to introduce the checklist to the birth attendants and assess existing quality gaps at the facility. We used locally produced videos and flipbooks to support the Launch event. (12)(13) Last, the Support strategy included a tapered 8-month peer-to-peer coaching program to encourage adoption of practices in the checklist and resolve barriers, and a data feedback structure through which coaches shared visual observation charts with facility and district staff to foster change. To facilitate sustainability, local facility champions were identified to support continued use of the WHO Checklist beyond the trial period.
Recommendations.
The local implementation team and research team need to be aligned through a standardized strategy with clear roles and scope of work to guide intervention implementation. This is especially important in anticipating and planning for staff turnover depending on the length of the trial.
Consider including front-line users of the intervention in addition to local leadership on the implementation team to maximize buy-in and integration of the intervention into existing workflows.
Because implementation challenges requiring adaptation may arise, it is important to budget and plan sufficient time for the early formative work in developing the implementation strategy for a complex intervention.
Recognize and work to achieve a balance between the science and art of implementation within the confines of the trial design.
The implementation team should be embedded in the local context as much as possible to facilitate troubleshooting and ensure the intervention is implemented with fidelity.
Measure the fidelity to the intervention implementation strategy during the trial or evaluation.
Consider whether refresher training is needed for the implementation team, depending on staff turnover, fidelity to implementation strategy, and duration of the study.
Data Quality Assurance Systems
The BetterBirth Trial achieved high ascertainment of health outcomes through a call center that successfully followed up with the majority of women (89,858/91,770, 97.9%) in addition to field workers who followed up with the remaining 1.6% (n=1,447). Only 0.5% (n=465) of enrolled women were lost to follow-up. (14) The Data Quality Monitoring and Improvement System was a robust, multicomponent system that ensured high-quality data throughout the BetterBirth Trial. The 5 components were: 1) monitoring and evaluation expert team who supported data quality assurance, 2) standard operating procedures for data collection, 3) training for data quality, 4) electronic data collection and reporting system, and 5) data quality assurance protocol, including data collection audits, rapid data feedback, and supportive supervision. (15)
Recommendations.
Build a data quality assurance team with data science expertise and a data quality assurance strategy, including continuous data cleaning to identify potential problems that can be addressed.
Invest in a system to minimize loss to follow-up and maximize data quality and budget for this appropriately.
Adapt data collection and follow-up methods to be contextually appropriate and culturally sensitive.
Develop a rigorous process for designing and piloting data collection instruments, keeping in mind the purpose of collecting each variable and potential linkages required among multiple data sets.
Ensure that there are fail-safe procedures for data backup and protection.
Dissemination Strategy
Once preliminary data analyses were complete, the BetterBirth team worked through a systematic process to refine analyses and develop key messages from the findings. First, the BetterBirth team presented the results among the internal partners, including the governments of India and Uttar Pradesh, Ariadne Labs, the Scientific Advisory Committee, and the Bill and Melinda Gates Foundation. We also partnered with the communications specialists at Ariadne Labs to develop a dissemination strategy for the results of the trial. Study participants, including facility and district staff and leadership, learned about the results at district-level meetings across Uttar Pradesh. Then, the BetterBirth team disseminated the results globally and targeted diverse audiences, such as academics, program implementers, and clinicians. Some examples include a keynote address and panel discussion at the Royal College of Obstetricians and Gynecologists World Congress; peer-reviewed publications; grand rounds and lectures at universities; podcasts and webinars; media interviews and blogs; and donor reports. In addition, the BetterBirth team worked with WHO to build a checklist collaborative between 2012 and 2015. This collaborative included 34 groups in 29 countries that implemented the WHO Checklist in 234 sites. Some of these groups conducted feasibility and evaluation studies of their adaptation and implementation of the WHO Checklist (16) and trial results were shared with this group through a technical consultation in Sri Lanka. (17)
Recommendations.
Partner with a communications specialist and graphic designer to develop messaging from the trial results and roll out a dissemination strategy to reach different audiences, including the study participants. The dissemination method will depend on the audience and need to be tailored to the appropriate context. It may be valuable to prepare and pilot specific presentations depending on the audience.
Consider whether some data analyses could be completed for earlier dissemination while waiting for the main trial results.
Building early global collaborations before and during the trial may be beneficial in maximizing impact.
Conclusions
We believe this roadmap of lessons learned in the BetterBirth Trial can aid the design, implementation, and dissemination of future large-scale studies of complex interventions. Recommendations include partnering with multidisciplinary collaborators, developing a theory of change, investing in formative work and pilot testing, building an intervention implementation strategy, creating data quality assurance systems, and partnering with communications specialists in a dissemination strategy. It is important for the research community to consolidate lessons learned from other similar trials and use (or create if needed) the standard frameworks for these large-scale trials with complex interventions so that best practices are identified and implemented. One example of such a framework is the Standards for Reporting Implementation Studies, which consist of a checklist for reporting key elements from the intervention and implementation strategy in implementation studies. (18) With the growth of translational research between clinical effectiveness and implementation in diverse global populations, advancing the field of large-scale studies with complex interventions will require guidance around intervention design and implementation strategy.
American Board of Pediatrics Neonatal-Perinatal Content Specifications
Know the issues in the organization of perinatal care (eg, regionalization, transport, practice guidelines, benchmarking data, quality improvement).
Understand the strengths and limitations of randomized controlled studies.
Footnotes
AUTHOR DISCLOSURE
Drs Molina and Semrau and Ms Bobay have disclosed funding from the Bill & Melinda Gates Foundation Grant OPP1017378: Better Birth: The WHO Safe Childbirth Checklist Trial. This commentary does not contain a discussion of an unapproved/investigative use of a commercial product/device.
- Copyright © 2019 by the American Academy of Pediatrics
In this issue
Jump to section
Related Articles
Cited By...
- No citing articles found.