With the current economic climate, the budget crisis within the health-care system, and the rapid development of health-care technologies in the field of neonatology, it is essential for decision makers at every level to understand how to interpret published economic evaluations to help guide both practice and budget decisions. The objective of this article is to provide a stepwise approach to interpreting economic evaluations conducted alongside randomized, controlled trials (RCTs). The emphasis in assessing the validity of such an analysis is to ensure that the authors are transparent in their approach with respect to factors such as the intervention and comparator, time horizon, study perspective, sources for costs and effects, appropriate combining of costs and effects into a single metric, and assessment of uncertainty and time preference.
After completing this article, readers should be able to:
Define the common economic evaluations used in health technology assessments.
Review factors that should be considered when reading an economic evaluation.
The rapid growth of technology and drug development over the past 3 decades has promoted substantial expansion of the field of neonatal intensive care and contributed to enormous prematurity-associated costs, estimated by the Institute of Medicine at $26.2 billion annually in the United States. Even this estimate is likely to be conservative because it excludes costs beyond early childhood, with the exception of four major morbidities associated with prematurity: intellectual disability, cerebral palsy, blindness, and deafness. Overall, the United States has the highest per capita health-care expenditure in the world, an average of more than $7,000, which is projected to exceed $13,000 per capita by 2019, based on the current rate of increase. Proportionately, the total health-care expenditure amounts to approximately 16% of the gross domestic product, which is higher than any other nation. A report published by the Commonwealth Fund in 2010 demonstrates that the United States has almost double the per capita health-care spending as six other developed and economically similar nations (Australia, Canada, Germany, Netherlands, New Zealand, and United Kingdom), while ranking last among them for most measures of quality of the care delivered.
The current economic climate has forced a re-examination of strategies of resource allocation to ensure more effective care, while concomitantly trying to limit health-care expenditures. Such a balance of costs and effectiveness is achieved through the methodology of economic evaluation. Economic evaluation can be undertaken using computer modeling (“decision analysis”) or alongside observational studies or RCTs. Assessing cost outcomes alongside RCTs has several advantages: 1) it ensures that all relevant data are collected, thus reducing the need for assumptions; 2) it allows analysis at the patient level, thus ensuring that statistical uncertainty is appropriately handled; and 3) it takes advantage of the RCT’s powerful mechanisms for reducing bias, thus optimizing the validity of the conclusions. Despite these advantages, only 2% of RCTs have included economic evaluations.
The purpose of this article is to provide readers with the necessary tools to understand and critically appraise reports of economic evaluations alongside clinical trials. In addition, the article and suggested references should serve as a starting point for those interested in applying this methodology to assess the cost-effectiveness of current programs or interventions.
Terminology and Definitions
“Economic evaluation” refers to health technology assessments that incorporate measurement of both the costs and consequences of the interventions being compared. Several varieties of economic evaluation are described in Table 1with hypothetical examples from neonatology. These studies differ from each other not in how they handle costs, but rather in how they express the effectiveness of the intervention. The most common type of economic evaluation performed alongside RCTs is a cost-effectiveness analysis (CEA), in which effectiveness is expressed as a change in a clinical outcome such as the change in bronchopulmonary dysplasia, mortality, or neurodevelopmental outcome. Cost-utility analysis (CUA) is a specific type of CEA in which clinical outcomes are weighted according to individuals' preferences, or “utilities,” and are then converted to quality-adjusted life years (QALYs) or disability-adjusted life years. Cost of illness, cost analyses, and cost minimization studies measure only costs without consideration of effectiveness and, therefore, are considered only “partial” economic evaluations.
The decision as to which type of evaluation to perform depends on the goal of the research question and the target audience. For example, a CUA is most appropriate for decision makers who consider interventions across programs or disciplines, in which case comparison is facilitated by a common measure of effectiveness such as the QALY. On the other hand, a budget impact analysis or a cost minimization study may be more useful for an individual hospital or agency seeking to determine the feasibility of adopting a new technology in the context of a given budget. This article emphasizes CEA as a primary methodology for economic evaluation alongside RCTs, but the general approach to costing is similar across the studies listed in Table 1.
Reading an Economic Evaluation
A number of factors should be considered in the design and interpretation of economic evaluations alongside RCTs (Table 2).
Step 1: Intervention and Comparators
The results of economic evaluations are presented as costs and effects of one program in comparison to another. The particular choice of comparator (or control group) is very important. Choosing a comparator that is seldom used may lead to results that are irrelevant to clinical practice. Conclusions may even be misleading if lower-cost and reasonably effective options are ignored. Thus, the comparator should represent a viable standard of care: either a commonly used alternative therapy or, in the case of new technologies, a placebo.
Step 2: Time Horizon
The time horizon for economic evaluation is defined as the time period over which the cost and effectiveness data are examined. It is imperative to specify the time horizon precisely because the choice may have an important impact on the results. An intervention that carries a large upfront cost may appear very expensive at the time of first discharge home, but if the intervention prevents major morbidities and their associated costs, an analysis later in life may reveal it to be economically appealing. The United States Panel on Cost-Effectiveness in Health and Medicine, therefore, recommends that analyses use a lifetime time horizon. Of course, data collection alongside RCTs is dictated by the therapy being tested and is often finite due to funding and follow-up limitations. The longest time horizon possible should be selected in this case. If cost differences are expected to persist beyond this point, computer modeling (decision analysis) may be undertaken to extend the applicability of the results.
Step 3: Perspective
The perspective refers to the choice of stakeholders whose interests are included in the economic evaluation. The United States Panel on Cost-Effectiveness in Health and Medicine recommends that the analysis be undertaken from a societal perspective, which is the most comprehensive approach, including all costs, regardless of the parties to whom they accrue. In addition to the medical expenses and overhead of running a program or hospital, this approach includes costs to other parties such as the educational system as well as family out-of-pocket expenses and work absenteeism. There are two primary advantages of such an approach: preventing misleading results due to cost-shifting between payers (eg, from an insurance company to a family through earlier discharge) and addressing the essential policy issue of the economic impact to society at a national level.
On the other hand, when a decision maker at the family, hospital, or state level is trying to balance a budget, a CEA from that individual perspective may be more useful. Ideally, authors should report a broader (societal or third-party payer) perspective for their base-case analysis, followed by an illustration of how the cost-effectiveness would differ if the focus were narrowed to a smaller group of stakeholders such as the family.
Step 4: Resource Utilization and Costs
Resource utilization and costs may be divided into several categories. Direct medical costs are those that can be attributed directly to medical care for an individual patient, such as days in hospital, visits to physicians, medications, or home medical equipment. In some cases, these are associated with overhead (or “indirect medical”) costs such as lighting or human resources necessary to run a hospital or practice. Direct nonmedical costs are those that are associated with care of an illness but do not involve medical services, such as family out-of-pocket expenditures for meals, parking, or child care. Finally, illness and its treatment have an impact on the ability of the patient or family to work. Costs related to work absenteeism or limitations to employment are known as “productivity losses.” As noted previously, the types of costs included in an analysis are determined by the perspective of the study. A societal perspective, for example, demands consideration of all costs related to the condition and its treatment; a hospital perspective omits family expenditures and productivity losses.
In practice, resources used, such as days of hospitalization or doses of medication, are measured in the trial, and costs are assigned to each. The sources of these costs are very important. If there is a functioning market for the resource, as for some outpatient drugs, the wholesale price in that market is the best choice. For hospitalizations, the price charged is often an arbitrary output of the hospital's accounting system and does not reflect the true input of resources used. Thus, hospital charges should be converted to costs. In the United States, this is most often done by using the “cost-to-charge ratios” that the Centers for Medicare and Medicaid Services requires from each hospital.
In the final step, the resource counts are multiplied by the assigned costs, an average cost per patient determined for each of the groups in the trial, and the difference in average costs calculated:
Step 5: Effectiveness
Effectiveness in economic evaluation may be expressed most simply as the change in a clinical parameter, such as survival without bronchopulmonary dysplasia or neurodevelopmental impairment. Studies that use such measures are known as CEAs.
Because it is difficult to compare the results of analyses that use different effectiveness measures, CUAs instead express effectiveness as a “utility.” A utility is an individual's preference for living in a particular health state or clinical outcome, expressed on a scale, for which 0 represents death and 1 represents perfect health. For example, in a widely used version, the average self-reported utility of blindness is 0.35 to 0.48. Utilities may be measured directly from participants in a clinical trial using a variety of methods or published values may be applied to a description of a health state called a “health status measure.” When the utility or preference for a health state is multiplied by the time that an individual spends in that state, the result is expressed as QALYs. QALYs have the significant advantage that they may be compared across studies that would otherwise have different effectiveness measures. There are certain methodologic limitations in using QALYs in neonatology trials because very young children cannot express their formal preferences for an outcome, and the use of surrogates such as parents is of unclear validity.
Finally, cost-benefit analyses apply a dollar value to the health outcome, often by asking individuals what they would be “willing to pay” to avoid the outcome. Such analyses rarely have been undertaken in pediatrics, in part because of the conceptual difficulties of this valuation process.
Once an effectiveness measure has been collected, the difference in average effectiveness between the two study groups is calculated:
Step 6: Cost-effectiveness
Once both costs and effectiveness have been measured, the incremental cost-effectiveness ratio (iCER) is calculated as the difference in costs between groups divided by the difference in effectiveness between groups:
In nonmathematical terms, the iCER represents the additional cost for each additional beneficial outcome achieved. Because it expresses the two outcomes of cost and effectiveness simultaneously, its interpretation is sometimes confusing. Results may be clarified by showing them on a “cost-effectiveness plane,” in which the x-axis represents the difference in effects and the y-axis represents the difference in costs (Fig. 1). Thus, results in the right lower quadrant are both more effective and cost less. Such results are referred to as “Dominant” strategies and always are appealing to a policy maker. Conversely, the left upper quadrant represents results that are less effective and have higher costs; such “Dominated” strategies are never economically (or clinically) appealing. Many economic evaluations yield results in the right upper quadrant, which demonstrates a gain in effectiveness that is associated with higher costs. Finally, the left lower quadrant represents interventions that have lower costs but worse outcomes. Although this option is both politically and socially less desirable because the discussion involves a less optimal approach, it is an alternative that arguably might be considered in the current economic crisis of health-care affordability.
In addition to the cost-effectiveness plane, results can be represented using the cost-effectiveness acceptability curve (CEAC) (Fig. 2). The y-axis represents the probability (0 to 1) that a treatment would be cost-effective given a willingness-to-pay threshold per outcome on the x-axis (expressed in $). The CEAC allows the decision maker to view the uncertainty in the iCER point estimate. A probability of 0.95 is equivalent to a 95% confidence that the iCER would be equal to or less than the said amount.
A commonly quoted acceptable threshold in the United States for an acceptable iCER is $50,000 to $100,000/QALY, but this number has been criticized as being an arbitrary cut-off. It is preferable to interpret the iCER in the context of the available budget and relative to iCERs of other accepted interventions in the population.
Step 7: Uncertainty
The iCER described so far is only a point estimate and does not consider uncertainty arising from sampling (statistical uncertainty) or from varying estimates of some parameters such as costs (parameter uncertainty). Two approaches are employed to address uncertainty. In the first, “deterministic sensitivity analysis,” the value of an input parameter is changed and the analysis is repeated to determine whether the iCER has changed substantially. Statistical uncertainty is addressed using either standard statistical tests or “probabilistic sensitivity analysis.” Although the latter techniques are beyond the scope of this review, the end result is conceptually similar to a confidence interval.
Step 8: Discounting
For most individuals, the opportunity to have money or benefits today is preferable to having money or health benefits in the future. To address this “time preference,” economic evaluations “discount” future costs and effects when studies have a long time horizon, typically beyond 1 year. The discount rate is set to be equal to the cost of capital in large markets, such as those for federal treasury bills. The discount rate in the base case is usually set at 3% but varies in sensitivity analysis between 0 and 5%.
As science and health-care innovations continue to grow at a rapid pace, so do the costs associated with financing care. A thorough understanding of the methodology of economic evaluation is important in ensuring that scarce societal resources are used to achieve the greatest health for vulnerable patients.
American Board of Pediatrics Neonatal-Perinatal Medicine Content Specifications
Differentiate cost-benefit from cost-effectiveness analysis.
Understand how quality-adjusted life years are used in cost analyses.
Understand the multiple perspectives (eg, of an individual, payor, society) that influence interpretation of cost-benefit and cost-effectiveness analyses.
Drs Dukhovny and Zupancic have disclosed no financial relationships relevant to this article. This commentary does not contain a discussion of an unapproved/investigative use of a commercial product/device.
- cost-effectiveness analysis •
- cost-effectiveness acceptability curve •
- cost-utility analysis •
- incremental cost-effectiveness ratio •
- quality-adjusted life year •
- randomized, controlled trial
- Copyright © 2011 by the American Academy of Pediatrics
- Drummond MF,
- Jefferson TO
- Drummond MF,
- Sculpher MJ,
- Torrance GW,
- O'Brien BJ,
- Stoddard GL
- Glick HA,
- Doshi JA,
- Sonnad SS,
- Polsky D
- Gold MR,
- Siegel JE,
- Russell LB,
- Weinstein MC
- Hunink MGM,
- Glasziou PP,
- Siegel JE,
- et al
Medical Care. 2009;47(supplement 7_1). [The entire supplement is dedicated to methodology in cost-effectiveness analyses]
- Zupancic JAF
International Society for Pharmacoeconomics and Outcomes Research (ISPOR): http://www.ispor.org
Society for Medical Decision Making (SMDM): http://www.smdm.org
Paediatric Economic Database Evaluation (PEDE) Project, University of Toronto: http://pede.ccb.sickkids.ca/pede/about.jsp
CEA Registry: Center for the Evaluation of Value and Risk and Health: https://research.tufts-nemc.org/cear/default.aspx
National Institute for Health and Clinical Excellence (NICE): http://www.nice.org.uk/
Canadian Agency for Drugs and Technologies in Health (CADTH): http://www.cadth.ca/
Health Utilities Group/Health Utilities Index (HUG/HUI): http://www.fhs.mcmaster.ca/hug
EQ-5D-Standarized Instrument for Use as a Measure of Health Outcomes: http://www.euroqol.org/