Checklists for external validity: a systematic review

Published on Dec 1, 2014in Journal of Evaluation in Clinical Practice1.536
· DOI :10.1111/jep.12166
Anne-Kirstine Dyrvig4
Estimated H-index: 4
(University of Southern Denmark),
Kristian Kidholm11
Estimated H-index: 11
(OUH: Odense University Hospital)
+ 1 AuthorsHindrik Vondeling14
Estimated H-index: 14
(University of Southern Denmark)
Rationale, aims and objectives The quality of the current literature on external validity varies considerably. An improved checklist with validated items on external validity would aid decision-makers in judging similarities among circumstances when transferring evidence from a study setting to an implementation setting. In this paper, currently available checklists on external validity are identified, assessed and used as a basis for proposing a new improved instrument. Method A systematic literature review was carried out in Pubmed, Embase and Cinahl on English-language papers without time restrictions. The retrieved checklist items were assessed for (i) the methodology used in primary literature, justifying inclusion of each item; and (ii) the number of times each item appeared in checklists. Results Fifteen papers were identified, presenting a total of 21 checklists for external validity, yielding a total of 38 checklist items. Empirical support was considered the most valid methodology for item inclusion. Assessment of methodological justification showed that none of the items were supported empirically. Other kinds of literature justified the inclusion of 22 of the items, and 17 items were included on the basis of consensus. On 36 occasions, the items were presented without methodological justification for inclusion. Assessment of frequency/occurrence showed that items were mentioned in one to at most 17 checklists. Conclusion This paper provides building blocks for the development of a new checklist for external validity. The next step is provision of empirical evidence for the checklist items to be selected, and finally, development and validation of a checklist on external validity.
  • References (26)
  • Citations (11)
📖 Papers frequently viewed together
54 Citations
18 Citations
41 Citations
78% of Scinapse members use related papers. After signing in, all features are FREE.
#1Piet N. PostH-Index: 16
#2Hans de BeerH-Index: 6
Last. Gordon H. Guyatt (McMaster University)H-Index: 201
view all 3 authors...
Rationale, aims and objectives Randomized controlled trials (RCTs) are the preferred source for evidence for the effect of treatment. However, patients participating in RCTs often manifest important differences from patients seen in practice. Therefore, guideline developers have to decide whether the results are generalizable to the target population not represented in RCTs. Method A systematic review of the literature was undertaken to identify methods to decide whether to generalize the result...
16 CitationsSource
#1Mona Nasser (Plymouth University)H-Index: 22
#2Johannes C. van der Wouden (Radboud University Nijmegen)H-Index: 74
Last. Floris A. van de Laar (Radboud University Nijmegen)H-Index: 15
view all 4 authors...
Results. We identified a wide variety of approaches to address the issue of generalizability in systematic reviews. We extracted four major primary care themes from the two definitions of primary care and person-centred medicine that we mapped across the methods of generalizability. We suggested a two-step approach: Step 1 planning the scope of a review using four-layer schematic model and Step 2 to follow four steps to incorporate the concept of generalizability to primary care in the review. C...
7 CitationsSource
#1Russell E. GlasgowH-Index: 110
#2David A. Chambers (NIH: National Institutes of Health)H-Index: 22
Background: Current approaches to medical science generally have not resulted in rapid, robust integration into feasible, sustainable real world healthcare programs and policies. Implementation science risks falling short of expectations if it aligns with historical norms. Fundamentally different scientific approaches are needed to accelerate such integration. Methods: We propose that the key goal of implementation science should be to study the development, spread and sustainability of broadly ...
104 CitationsSource
#1Kristian Kidholm (OUH: Odense University Hospital)H-Index: 11
Last. Mickael Bech (University of Southern Denmark)H-Index: 20
view all 8 authors...
OBJECTIVES: Telemedicine applications could potentially solve many of the challenges faced by the healthcare sectors in Europe. However, a framework for assessment of these technologies is need by decision makers to assist them in choosing the most efficient and cost-effective technologies. Therefore in 2009 the European Commission initiated the development of a framework for assessing telemedicine applications, based on the users' need for information for decision making. This article presents ...
138 CitationsSource
#1David AtkinsH-Index: 23
#2Stephanie Chang (AHRQ: Agency for Healthcare Research and Quality)H-Index: 21
Last. David B. Matchar (NUS: National University of Singapore)H-Index: 72
view all 7 authors...
Abstract Objective To describe a systematic approach for identifying, reporting, and synthesizing information to allow consistent and transparent consideration of the applicability of the evidence in a systematic review according to the Population, Intervention, Comparator, Outcome, Setting domains. Study Design and Setting Comparative effectiveness reviews need to consider whether available evidence is applicable to specific clinical or policy questions to be useful to decision makers. Authors ...
46 CitationsSource
#1Ron Goeree (McMaster University)H-Index: 46
#2Jing HeH-Index: 3
Last. Natasha BurkeH-Index: 14
view all 7 authors...
Background: Health technology assessments (HTA) generally, and economic evaluations (EE) more specifically, have become an integral part of health care decision making around the world. However, these assessments are time consuming and expensive to conduct. Evaluation resources are scarce and therefore priorities need to be set for these assessments and the ability to use information from one country or region in another (geographic transferability) is an increasingly important consideration.
53 CitationsSource
THERAPEUTIC AND PREVENTION CLINICAL RESEARCH IS typically performed to address questions of efficacy (“Can intervention work in the ideal study setting?”), effectiveness (“Does it work, generalized to real-world settings and applied to individual patients?”), and cost-effectiveness (“Is it worth it and should it be paid for?”). To date, both public and private research enterprise has predominantly funded efficacy research. Comparative effectiveness research holds promise to generate much-needed ...
68 CitationsSource
#1David Moher (Ottawa Hospital Research Institute)H-Index: 100
#2Sally Hopewell (University of Oxford)H-Index: 43
Last. Douglas G. Altman (University of Oxford)H-Index: 209
view all 9 authors...
Overwhelming evidence shows the quality of reporting of randomised controlled trials (RCTs) is not optimal. Without transparent reporting, readers cannot judge the reliability and validity of trial findings nor extract information for systematic reviews. Recent methodological analyses indicate that inadequate reporting and design are associated with biased estimates of treatment effects. Such systematic error is seriously damaging to RCTs, which are considered the gold standard for evaluating in...
3,512 CitationsSource
#1Laura C. Leviton (Robert Wood Johnson Foundation)H-Index: 26
#2Laura Kettel Khan (CDC: Centers for Disease Control and Prevention)H-Index: 37
Last. David CottonH-Index: 5
view all 5 authors...
Evaluability assessment, also commonly known as exploratory evaluation, has assisted the field of public health to improve programs and to develop a pragmatic, practice-based research agenda. Evaluability assessment was originally developed as a low-cost pre-evaluation activity to prepare better for conventional evaluations of programs, practices, and some policies. For public health programs, however, it serves several other important purposes: (a) giving program staff rapid, constructive feedb...
74 CitationsSource
#1Olaf M. Dekkers (LUMC: Leiden University Medical Center)H-Index: 55
#2E von Elm (University of Bern)H-Index: 40
Last. Jan P. Vandenbroucke (LUMC: Leiden University Medical Center)H-Index: 78
view all 5 authors...
Background External validity of study results is an important issue from a clinical point of view. From a methodological point of view, however, the concept of external validity is more complex than it seems to be at first glance. Methods Methodological review to address the concept of external validity. Results External validity refers to the question whether results are generalizable to persons other than the population in the original study. The only formal way to establish the external valid...
119 CitationsSource
Cited By11
#1Heather Menzies Munthe-Kaas (FHI: Norwegian Institute of Public Health)H-Index: 10
#2Heid Nøkleby (FHI: Norwegian Institute of Public Health)H-Index: 1
Last. Lien Nguyen (FHI: Norwegian Institute of Public Health)H-Index: 1
view all 3 authors...
Background Systematic reviews of research evidence have become an expected basis for decisions about practice guidelines and policy decisions in the health and welfare sectors. Review authors define inclusion criteria to help them determine which studies to search for and include in their reviews. However, these studies may still vary in the extent to which they reflect the context of interest in the review question. While most review authors would agree that systematic reviews should be relevan...
1 CitationsSource
#1Georgina Gethin (Monash University)H-Index: 18
#2John D. Ivory (National University of Ireland, Galway)H-Index: 1
Last. Carolina Dragica Weller (Monash University)H-Index: 10
view all 5 authors...
: We set out to evaluate quality of reporting of data related to external validity from randomized controlled trials (RCTs) assessing treatment interventions for active venous leg ulcers. Using a systematic review study design, we identified 144 full-text RCTs of treatment interventions, where the wound was assessed and published in English from 1998 to 2018. We found that the median study sample size was 75.5. Weighted mean wound size was 13.22 cm2 and weighted mean wound duration was 22.20 mon...
2 CitationsSource
#1Inger von Bültzingslöwen (University of Gothenburg)H-Index: 8
Last. Jörgen PaulanderH-Index: 10
view all 6 authors...
OBJECTIVES: There is a need for monitoring dental health and healthcare, as support for quality development, allocation of resources and long-term planning of dental care. The aim of this paper is to describe the concept and implementation of the Swedish Quality Registry for Caries and Periodontal Diseases (SKaPa). MATERIALS AND METHODS: The SKaPa receives information by automatic transfer of data daily from electronic patient dental records via secure connections from affiliated dental care org...
#1Quan Nha Hong (McGill University)H-Index: 12
#2Pierre Pluye (McGill University)H-Index: 29
The past decade has been rich with methodological advancements in systematic reviews, several of which were inspired by the literature on mixed methods research. Systematic mixed studies reviews—that is, reviews combining qualitative and quantitative evidence—are increasingly popular as they can provide a better understanding of complex phenomena and interventions. However, they raise new challenges, especially regarding how to perform critical appraisal of the included studies that vary regardi...
9 CitationsSource
#1Susan Baxter (University of Sheffield)H-Index: 19
#2Maxine Johnson (University of Sheffield)H-Index: 9
Last. Andrew Booth Ba (University of Sheffield)H-Index: 43
view all 6 authors...
There have been calls for greater consideration of applicability and transferability in systematic reviews, to improve their usefulness in informing policy and practice. Understanding how evidence is, or is not applicable and transferable to varying local situations and contexts, is a key challenge for systematic review synthesis in healthcare. Assessing applicability and transferability in systematic reviews is reported to be difficult, particularly in reviews of complex interventions. There is...
#1Dylan Kneale (IOE: Institute of Education)H-Index: 8
#2James Thomas (IOE: Institute of Education)H-Index: 36
Last. Richard D. Wiggins (IOE: Institute of Education)H-Index: 23
view all 4 authors...
This paper critically explores how survey and routinely collected data could aid in assessing the generalisability of public health evidence. We propose developing approaches that could be employed in understanding the relevance of public health evidence, and investigate ways of producing meta‐analytic estimates tailored to reflect local circumstances, based on analyses of secondary data. Currently, public health decision‐makers face challenges in interpreting ‘global’ review evidence to assess ...
3 CitationsSource
#1Mahrukh Saleem (BU: Bournemouth University)H-Index: 1
#2Teresa Burdett (BU: Bournemouth University)H-Index: 1
Last. Vanessa Heaslip (BU: Bournemouth University)H-Index: 9
view all 3 authors...
The significance of sanitation to safeguard human health is irrefutable and has important public health dimensions. Access to sanitation has been essential for human dignity, health and well-being. Despite 15 years of conjunctive efforts under the global action plans like Millennium Development Goals (MDGs), 2.3 billion people have no access to improved sanitation facilities (flush latrine or pit latrine) and nearly 892 million of the total world’s population is still practicing open defecation....
1 CitationsSource
#1Helen Burchett (Lond: University of London)H-Index: 15
#2Laurence Blanchard (Lond: University of London)H-Index: 2
Last. James Thomas (IOE: Institute of Education)H-Index: 7
view all 4 authors...
Public health interventions can be complicated, complex and context dependent, making the assessment of applicability challenging. Nevertheless, for them to be of use beyond the original study setting, they need to be generalisable to other settings and, crucially, research users need to be able to identify to which contexts it may be applicable. There are many tools with set criteria for assessing generalisability/applicability, yet few seem to be widely used and there is no consensus on which ...
4 CitationsSource
#1Tamara Schloemer (PHRI: Public Health Research Institute)H-Index: 1
#2Peter Schröder-Bäck (PHRI: Public Health Research Institute)H-Index: 8
Improving the public’s health in different countries requires the consideration of diverse health care systems and settings. For evidence-based public health, decision-makers need to consider the transferability of effective health interventions from a primary context to their specific target context. The aim of this systematic review was to develop a model for the assessment of transferability of health interventions through identification and systematization of influencing criteria, including ...
8 CitationsSource
#1Anja Maria Brænd (University of Oslo)H-Index: 5
#2Jørund Straand (University of Oslo)H-Index: 23
Last. Atle Klovning (University of Oslo)H-Index: 10
view all 3 authors...
When reading a report of a clinical trial, it should be possible to judge whether the results are relevant for your patients. Issues affecting the external validity or generalizability of a trial should therefore be reported. Our aim was to determine whether articles with published results from a complete cohort of drug trials conducted entirely or partly in general practice reported sufficient information about the trials to consider the external validity. A cohort of 196 drug trials in Norwegi...
2 CitationsSource