Secular trends and evaluation of complex interventions: the rising tide phenomenon

Published on May 1, 2016in BMJ Quality & Safety7.043
· DOI :10.1136/bmjqs-2015-004372
Yen-Fu Chen22
Estimated H-index: 22
(Warw.: University of Warwick),
Karla Hemming28
Estimated H-index: 28
(University of Birmingham)
+ 1 AuthorsRichard J. Lilford70
Estimated H-index: 70
(Warw.: University of Warwick)
Evaluations of service delivery interventions with contemporaneous controls often yield null results, even when the intervention appeared promising in advance. There can be many reasons for null results. In this paper we introduce the concept of a ‘rising tide’ phenomenon being a possible explanation of null results. We note that evaluations of service delivery interventions often occur when awareness of the problems they intend to address is already heightened, and pressure to tackle them is mounting throughout a health system. An evaluation may therefore take place in a setting where the system as a whole is improving – where there is a pronounced temporal trend or a ‘rising tide causing all vessels to rise’. As a consequence, control sites in an intervention study will improve. This reduces the difference between intervention and control sites and predisposes the study to a null result, leading to the conclusion that the intervention has no effect. We discuss how a rising tide may be distinguished from other causes of improvement in both control and intervention groups, and give examples where the rising tide provides a convincing explanation of such a finding. We offer recommendations for interpretation of research findings where improvements in the intervention group are matched by improvements in the control group. Understanding the rising tide phenomenon is important for a more nuanced interpretation of null results arising in the context of system-wide improvement. Recognition that a rising tide may have predisposed to a null result in one health system cautions against generalising the result to another health system where strong secular trends are absent.
  • References (40)
  • Citations (30)
📖 Papers frequently viewed together
6 Authors (Peter Craig, ..., Mark Petticrew)
4,778 Citations
23 Citations
1 Author (Richard Shepherd)
58 Citations
78% of Scinapse members use related papers. After signing in, all features are FREE.
#1Karla Hemming (University of Birmingham)H-Index: 28
#2Terry P. Haines (Monash University)H-Index: 32
Last. Richard J. Lilford (Warw.: University of Warwick)H-Index: 70
view all 5 authors...
The stepped wedge cluster randomised trial is a novel research study design that is increasingly being used in the evaluation of service delivery type interventions. The design involves random and sequential crossover of clusters from control to intervention until all clusters are exposed. It is a pragmatic study design which can reconcile the need for robust evaluations with political or logistical constraints. While not exclusively for the evaluation of service delivery interventions, it is pa...
289 CitationsSource
#1Nicholas H. Osborne (UM: University of Michigan)H-Index: 19
#2Lauren Hersch Nicholas (Johns Hopkins University)H-Index: 17
Last. Justin B. Dimick (UM: University of Michigan)H-Index: 67
view all 5 authors...
Importance The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) provides feedback to hospitals on risk-adjusted outcomes. It is not known if participation in the program improves outcomes and reduces costs relative to nonparticipating hospitals. Objective To evaluate the association of enrollment and participation in the ACS NSQIP with outcomes and Medicare payments compared with control hospitals that did not participate in the program. Design, Setting, and...
136 CitationsSource
#1David A. Etzioni (Mayo Clinic)H-Index: 8
#2Nabil Wasif (Mayo Clinic)H-Index: 26
Last. Elizabeth B. Habermann (Mayo Clinic)H-Index: 32
view all 8 authors...
(adjusted odds ratio, 1.00; 95% CI, 0.97-1.03), serious complications (adjusted odds ratio, 0.98; 95% CI, 0.94-1.03), or mortality (adjusted odds ratio, 1.04; 95% CI, 0.94-1.14). CONCLUSIONS AND RELEVANCE No association was found between hospital-based participation in the NSQIP and improvements in postoperative outcomes over time within a large cohort of patients undergoing elective general/vascular operations at academic hospitals in the United States. These findings suggest that a surgical ou...
99 CitationsSource
#1Julian Bion (University of Birmingham)H-Index: 36
Background: Bloodstream infections from central venous catheters (CVC-BSIs) increase morbidity and costs in intensive care units (ICUs). Substantial reductions in CVC-BSI rates have been reported using a combination of technical and non-technical interventions. Methods: We conducted a 2-year, four-cluster, stepped non-randomised study of technical and non-technical (behavioural) interventions to prevent CVC-BSIs in adult and paediatric ICUs in England. Random-effects Poisson regression modelling...
157 CitationsSource
#1Noreen Dadirai Mdege (Ebor: University of York)H-Index: 10
#2Mei-See Man (Ebor: University of York)H-Index: 15
Last. David Tappin (Ebor: University of York)H-Index: 77
view all 4 authors...
Abstract Objective To describe the application of the stepped wedge cluster randomized controlled trial (CRCT) design. Study Design and Setting Systematic review. We searched Medline, Embase, PsycINFO, HMIC, CINAHL, Cochrane Library, Web of Knowledge, and Current Controlled Trials Register for articles published up to January 2010. Stepped wedge CRCTs from all fields of research were included. Two authors independently reviewed and extracted data from the studies. Results Twenty-five studies wer...
228 CitationsSource
#1Karla Hemming (University of Birmingham)H-Index: 28
#2Alan Girling (University of Birmingham)H-Index: 28
Last. Richard J. Lilford (University of Birmingham)H-Index: 70
view all 5 authors...
Background Cluster randomised controlled trials (CRCTs) are frequently used in health service evaluation. Assuming an average cluster size, required sample sizes are readily computed for both binary and continuous outcomes, by estimating a design effect or inflation factor. However, where the number of clusters are fixed in advance, but where it is possible to increase the number of individuals within each cluster, as is frequently the case in health service evaluation, sample size formulae have...
104 CitationsSource
#1Amirta BenningH-Index: 2
#2Mary Dixon-WoodsH-Index: 51
Last. Richard J. LilfordH-Index: 70
view all 15 authors...
Objective To independently evaluate the impact of the second phase of the Health Foundation’s Safer Patients Initiative (SPI2) on a range of patient safety measures. Design A controlled before and after design. Five substudies: survey of staff attitudes; review of case notes from high risk (respiratory) patients in medical wards; review of case notes from surgical patients; indirect evaluation of hand hygiene by measuring hospital use of handwashing materials; measurement of outcomes (adverse ev...
93 CitationsSource
#1Amirta Benning (University of Birmingham)H-Index: 2
#2Maisoon Ghaleb (University of Hertfordshire)H-Index: 8
Last. Richard J. Lilford (University of Birmingham)H-Index: 70
view all 15 authors...
Objectives To conduct an independent evaluation of the first phase of the Health Foundation’s Safer Patients Initiative (SPI), and to identify the net additional effect of SPI and any differences in changes in participating and non-participating NHS hospitals. Design Mixed method evaluation involving five substudies, before and after design. Setting NHS hospitals in the United Kingdom. Participants Four hospitals (one in each country in the UK) participating in the first phase of the SPI (SPI1);...
117 CitationsSource
#1Peter CraigH-Index: 16
#2Curtis CooperH-Index: 174
Last. Simon G. ThompsonH-Index: 102
view all 11 authors...
The aim of this guidance is to help producers, users, funders and publishers of evidence understand how and when 'natural experiments' can be used to good effect. While there is growing interest in using natural experiments to evaluate large-scale interventions to improve population health, the relevant methodological literature is dispersed across disciplines and there is a lack of general guidance either on the range of approaches available, or on the circumstances in which they are likely to ...
48 Citations
#1Julia NeilyH-Index: 23
#2Peter D. Mills (Dartmouth College)H-Index: 26
Last. James P. BagianH-Index: 29
view all 9 authors...
Context There is insufficient information about the effectiveness of medical team training on surgical outcomes. The Veterans Health Administration (VHA) implemented a formalized medical team training program for operating room personnel on a national level. Objective To determine whether an association existed between the VHA Medical Team Training program and surgical outcomes. Design, Setting, and Participants A retrospective health services study with a contemporaneous control group was condu...
636 CitationsSource
Cited By30
#1Jason H. Wasfy (Harvard University)H-Index: 14
#2Michael K. HidrueH-Index: 6
Last. M.G. del Carmen (Harvard University)H-Index: 33
view all 11 authors...
Background: Reducing hospital readmission after acute myocardial infarction (AMI) has the potential to both improve quality and reduce costs. As such, readmission after AMI has been a target of fin...
#1Katharine Reeves (University Hospitals Birmingham NHS Foundation Trust)H-Index: 1
#2Samuel Chan (University Hospitals Birmingham NHS Foundation Trust)H-Index: 1
Last. Richard J. Lilford (Warw.: University of Warwick)H-Index: 70
view all 7 authors...
Objective To examine implementation of evidence in orthopaedic practice following publication of the results of three pivotal clinical trials. Design Case studies based on three orthopaedic trials funded in sequence by the National Institute for Health Research Health Technology Assessment (HTA) programme. These trials dealt with treatment of fractures of the humerus, radius and ankle, respectively. For each case study, we conducted time-series analyses to examine the relationship between public...
2 CitationsSource
#1Christine Soong (Mount Sinai Hospital)
#2Hyung J. ChoH-Index: 3
Last. Kaveh G. Shojania (U of T: University of Toronto)H-Index: 46
view all 3 authors...
The years since launch of the Choosing Wisely Campaign1 2 have seen an increase in studies reporting interventions aimed at reducing low-value care, from unindicated imaging3 4 and laboratory tests3 4 to prescriptions for medicines5–7 that deliver no net benefit. Many describe use of some combination of the usual suspects of intervention types: education,5 8 performance feedback data (sometimes described as audit-feedback, social comparison or peer comparison), policy changes (eg, restricting re...
#1Michel WensingH-Index: 62
#2Jeremy GrimshawH-Index: 6
#1Ana C. De Roo (UM: University of Michigan)
#2Joceline V. Vu (UM: University of Michigan)H-Index: 3
Last. Scott E. Regenbogen (UM: University of Michigan)H-Index: 27
view all 3 authors...
Abstract Background Both enhanced recovery and anesthesia literature recommend multimodal perioperative analgesia to hasten recovery, prevent adverse events, and reduce opioid use after surgery. However, adherence to, and outcomes of, these recommendations are unknown. We sought to characterize use of multimodal analgesia and its association with length of stay after colectomy. Materials and methods Within a statewide, 72-hospital collaborative quality initiative, we evaluated postoperative anal...
#1Alice GradyH-Index: 6
#2Luke WolfendenH-Index: 26
Last. Sze Lin YoongH-Index: 17
view all 15 authors...
BACKGROUND: Foods provided in childcare services are not consistent with dietary guideline recommendations. Web-based systems offer unique opportunities to support the implementation of such guidelines. OBJECTIVE: This study aimed to assess the effectiveness of a Web-based menu planning intervention in increasing the mean number of food groups on childcare service menus that comply with dietary guidelines. Secondary aims were to assess the impact of the intervention on the proportion of service ...
1 CitationsSource
#1Elia DíezH-Index: 15
#2Maria JoseLopezH-Index: 28
Last. Villalbi
view all 7 authors...
Background From 2000 to 2008, in urban areas in Spain, adolescent fertility and abortion rates underwent unprecedented increases, consecutive to intensive immigration from developing countries. To address unmet needs for contraception information and services, a community-based, gender-sensitive and culturally adapted brief counselling intervention (SIRIAN program) was launched in some deprived neighbourhoods with a high proportion of immigrants in Barcelona. Once a randomized controlled trial d...
#1Oddveig Reiersdal Aaberg (University of Stavanger)H-Index: 1
#2Randi Ballangrud (NTNU: Norwegian University of Science and Technology)H-Index: 5
Last. Marie-Louise Hall-Lord (NTNU: Norwegian University of Science and Technology)H-Index: 19
view all 4 authors...
ABSTRACTDespite a growing awareness of the importance of interprofessional teamwork in relation to patient safety, many hospital units lack effective teamwork. The aim of this study was to explore ...
#2Maria Leuschner (Otto-von-Guericke University Magdeburg)H-Index: 1
Last. Andreas D. NiederbichlerH-Index: 1
view all 4 authors...
Abstract Background Non-technical skills (NTS) failures have been implicated in a large proportion of surgical errors. The objective of this meta-analysis was to investigate whether NTS training of theatre staff improves patient outcomes. Methods In a systematic literature search all interventional studies evaluating the effects of NTS training of theatre staff were identified. Primary outcomes included mortality, morbidity, readmission rate and length of hospital stay. Secondary outcomes includ...
2 CitationsSource
#1Eilís Keeble (Nuffield Trust)H-Index: 1
#2Martin Bardsley (Nuffield Trust)H-Index: 23
Last. Nicholas Mays (Lond: University of London)H-Index: 39
view all 5 authors...
Objective To examine whether any differential change in emergency admissions could be attributed to integrated care by comparing pioneer and non-pioneer populations from a pre-pioneer baseline period (April 2010 to March 2013) over two follow-up periods: to 2014/2015 and to 2015/2016. Design Difference-in-differences analysis of emergency hospital admissions from English Hospital Episode Statistics. Setting Local authorities in England classified as either pioneer or non-pioneer. Participants Em...