Match!

Assessing the accuracy of forecasting: Applying standard diagnostic assessment tools to a health technology early warning system

Published on Aug 1, 2004in International Journal of Technology Assessment in Health Care1.42
· DOI :10.1017/S0266462304001229
Sue Simpson13
Estimated H-index: 13
(University of Birmingham),
Chris Hyde49
Estimated H-index: 49
(University of Birmingham)
+ 2 AuthorsAndrew Stevens30
Estimated H-index: 30
(University of Birmingham)
Cite
Abstract
Objectives: Early warning systems are an integral part of many health technology assessment programs. Despite this finding, to date, there have been no quantitative evaluations of the accuracy of predictions made by thesesystems. We report a study evaluating the accuracy of predictions made by the main United Kingdom early warning system. Methods: As prediction of impact is analogous to diagnosis, a method normally applied to determine the accuracy of diagnostic tests was used. The sensitivity, specificity, and predictive values of the National Horizon Scanning Centre's prediction methods were estimated with reference to an (imperfect) gold standard, that is, expert opinion of impact 3 to 5 years after prediction. Results: The sensitivity of predictions was 71 percent (95 percent confidence interval [Cl], 0.36-0.92), and the specificity was 73 percent (95 percent Cl, 0.64-0.8). The negative predictive value was 98 percent (95 percent Cl, 0.92-0.99), and the positive predictive value was 14 percent (95 percent Cl, 0.06-0.3). Conclusions: Forecasting is difficult, but the results suggest that this early warning system's predictions have an acceptable level of accuracy. However, there are caveats. The first is that early warning systems may themselves reduce the impact of a technology, as helping to control adoption and diffusion is their main purpose. The second is that the use of an imperfect gold standard may bias the results. As early warning systems are viewed as an increasingly important component of health technology assessment and decision making, their outcomes must be evaluated. The method used here should be investigated further and the accuracy of other early warning systems explored.
  • References (8)
  • Citations (14)
Cite
References8
Newest
Published on Oct 10, 2008
J. Andr Knottnerus1
Estimated H-index: 1
,
Frank Buntinx43
Estimated H-index: 43
Published on Jun 1, 2003in Journal of Public Health1.65
Ruairidh Milne27
Estimated H-index: 27
,
Andrew Clegg34
Estimated H-index: 34
,
Andrew Stevens30
Estimated H-index: 30
Reports produced in response to the need for health technology assessment (HTA) vary greatly in the methods they use, depending on the decision-maker’s needs, the technology’s characteristics and the resources available. HTA reports vary from the brief, such as ‘vignettes’ produced when a new technology emerges, to the exhaustive, such as ‘Cochrane reviews’ synthesising a mature evidence base. They may address a wide range of different questions. ‘Classic HTAs’, typically those reports prepared ...
Published on Feb 23, 2002in BMJ27.60
J. André Knottnerus5
Estimated H-index: 5
,
Johannes C. van der Wouden73
Estimated H-index: 73
,
Jean Muris34
Estimated H-index: 34
This is the first of a series of five articles Development and introduction of new diagnostic techniques have greatly accelerated over the past decades. The evaluation of diagnostic techniques, however, is less advanced than that of treatments. Unlike with drugs, there are generally no formal requirements for adoption of diagnostic tests in routine care. In spite of important contributions, 1 2 the methodology of diagnostic research is poorly defined compared with study designs on treatment effe...
Published on Oct 1, 2001in Health Policy2.08
David Hailey12
Estimated H-index: 12
(The Heritage Foundation),
Leigh Ann Topfer1
Estimated H-index: 1
,
Fiona Wills1
Estimated H-index: 1
(The Heritage Foundation)
Abstract A pilot project to provide advice on new and emerging medical technologies to decision makers in a provincial health care system was undertaken by a health technology assessment (HTA) program. Briefs were prepared on technologies which were not yet available in the province and which might have a significant impact on health care. These were sent to the ministry of health and regional health authorities and made available through the agency's website. Reaction to the briefs was sought f...
Published on Sep 1, 2001in Health Affairs5.71
Victor R. Fuchs44
Estimated H-index: 44
,
Harold C. Sox20
Estimated H-index: 20
In response to a mail survey, 225 leading general internists provided their opinions of the relative importance to patients of thirty medical innovations. They also provided information about themselves and their practices. Their responses yielded a mean score and a variability score for each innovation. Mean scores were significantly higher for innovations in procedures than in medications and for innovations to treat cardiovascular disease than for those to treat other diseases. The rankings w...
Published on Jan 1, 1999in Health Technology Assessment3.82
Glenn Robert29
Estimated H-index: 29
,
Andrew Stevens30
Estimated H-index: 30
,
John Gabbay24
Estimated H-index: 24
Published on Apr 1, 1998in Statistical Methods in Medical Research2.39
Siu L Hui1
Estimated H-index: 1
(IU: Indiana University),
Xiao H Zhou1
Estimated H-index: 1
(IU: Indiana University)
This paper reviews statistical methods developed to estimate the sensitivity and specificity of screening or diagnostic tests when the fallible tests are not evaluated against a gold standard. It gives a brief summary of the earlier historical developments and focuses on the more recent methods. It covers Bayesian approaches and longitudinal studies with repeated testing. In particular, it reviews the procedures that do not require the assumption of independence between tests conditional on the ...
Cited By14
Newest
Irene Eriksson9
Estimated H-index: 9
(KI: Karolinska Institutet),
Mia von Euler20
Estimated H-index: 20
(KI: Karolinska Institutet)
+ 2 AuthorsBjörn Wettermark32
Estimated H-index: 32
(KI: Karolinska Institutet)
Background Early awareness and alert systems have been established in many countries but evidence on their ability to accurately prioritize new medicines (for early assessment) is limited.
Published on Jul 1, 2017in BMJ Open2.38
Lucy Doos6
Estimated H-index: 6
(University of Birmingham),
Claire Packer11
Estimated H-index: 11
(University of Birmingham)
+ 2 AuthorsAndrew Stevens30
Estimated H-index: 30
(University of Birmingham)
Objective To describe and classify health technologies predicted in forecasting studies. Design and methods A portrait describing health technologies predicted in 15 forecasting studies published between 1986 and 2010 that were identified in a previous systematic review. Health technologies are classified according to their type, purpose and clinical use; relating these to the original purpose and timing of the forecasting studies. Data sources All health-related technologies predicted in 15 for...
Published on Feb 1, 2015in Health Affairs5.71
Jeffrey C. Lerner3
Estimated H-index: 3
(ECRI Institute),
Diane C. Robertson2
Estimated H-index: 2
(ECRI Institute),
Sara M. Goldstein1
Estimated H-index: 1
(ECRI Institute)
Health technology forecasting is designed to provide reliable predictions about costs, utilization, diffusion, and other market realities before the technologies enter routine clinical use. In this article we address three questions central to forecasting’s usefulness: Are early forecasts sufficiently accurate to help providers acquire the most promising technology and payers to set effective coverage policies? What variables contribute to inaccurate forecasts? How can forecasters manage the var...
Claire Packer11
Estimated H-index: 11
(University of Birmingham),
Matthew Fung3
Estimated H-index: 3
(University of Birmingham),
Andrew Stevens30
Estimated H-index: 30
(University of Birmingham)
Objectives: The aim of this study was to assess the accuracy of the English National Horizon Scanning Centre (NHSC) in identifying and filtering pharmaceutical developments using end user and international collaborator databases of emerging technologies as proxies for new drugs of likely significance to health services and/or patients. Methods: We used the NHSC information system and the list of National Institute for Health and Clinical Excellence (NICE) technology appraisals to estimate the fa...
Published on Dec 1, 2010in BMC Health Services Research1.93
Björn Wettermark32
Estimated H-index: 32
(Karolinska University Hospital),
Marie Persson5
Estimated H-index: 5
(Stockholm County Council)
+ 6 AuthorsLars L. Gustafsson52
Estimated H-index: 52
(Karolinska University Hospital)
Background New pharmacological therapies are challenging the healthcare systems, and there is an increasing need to assess their therapeutic value in relation to existing alternatives as well as their potential budget impact. Consequently, new models to introduce drugs in healthcare are urgently needed. In the metropolitan health region of Stockholm, Sweden, a model has been developed including early warning (horizon scanning), forecasting of drug utilization and expenditure, critical drug evalu...
Published on Jan 1, 2009
Activities related to horizon scanning of technologies for health care, a form of health technology assessment carried out at the beginning of the life cycle of those technologies, have been adopted in several developed countries since the 90s, with the purpose of improving the efficiency of decision making related to the incorporation of new technologies. Horizon scanning consists of a systematic approach to identify and assess new and emerging technologies in order to warn health care decision...
Published on Aug 1, 2008in Health Policy2.08
C. Wild16
Estimated H-index: 16
,
Thomas Langer2
Estimated H-index: 2
Objectives All western healthcare systems are confronted with a rising number of new health technologies. To support decision-making processes with sound information about new health technologies, some countries have established "Horizon Scanning Systems (HSS)". This paper gives an overview of processes and practices of HSS.Method The paper is based on a literature review (Medline and Embase) and on unpublished information gathered from HSS-agencies.Results The 13 current HSS have been collabora...
Philipp Storz4
Estimated H-index: 4
,
Kai Kolpatzik1
Estimated H-index: 1
+ 2 AuthorsBertram Häussler8
Estimated H-index: 8
(Technical University of Berlin)
Objectives: The aim of this study was to identify research and development on genetic testing to find out if research addresses important disease areas, how far it is from potential clinical use, and what consequences might arise for the prioritization of health technology assessment (HTA) activities. Also a horizon scanning methodology developed in Germany is demonstrated. Methods: A systematic search on genetic testing was conducted in an innovation database (ZIM database). Based on a daily mo...
Kieran Murphy1
Estimated H-index: 1
(University of Birmingham),
Claire Packer11
Estimated H-index: 11
(University of Birmingham)
+ 1 AuthorsSue Simpson13
Estimated H-index: 13
(University of Birmingham)
Objectives: The aim of this study was to define an effective early warning system, to identify and rank the characteristics of an effective early warning system for emerging health technologies, and to evaluate current early warning systems against these characteristics. Methods: An iterative Delphi-type process with the thirteen members of the International Information Network on New and Changing Health Technologies (EuroScan). We synthesized key characteristics that network members had graded....
Published on Jan 1, 2007in Social Science & Medicine3.09
Karla Douw6
Estimated H-index: 6
(University of Southern Denmark),
Hindrik Vondeling14
Estimated H-index: 14
(University of Southern Denmark)
 Several countries have systems in place to support the managed entry of new health technologies. The big challenge for these so-called horizon-scanning systems is to select those technologies that require decision support by means of an early evaluation. Clinical experts are considered a valuable source of information on new health technologies, but research on the relevance of their input is scarce. In 2000, we asked six Danish expert oncologists to predict whether a sample of 19 new anticanc...