Understanding economic evaluation: A policy perspective for clinicians

See related papers on quality of life measurement in respiratory disease in Can Respir J 1997;4(3). Correspondence: Dr M Giacomini, Centre for Health Economics and Policy Analysis, McMaster University, HSC-3H25, 1200 Main Street West, Hamilton, Ontario L8N 3Z5. Telephone 905-525-9140 ext 22143, fax 905-546-5211, e-mail giacomin@fhs.mcmaster.ca M Giacomini, J Hurley. Understanding economic evaluation: A policy perspective for clinicians. Can Respir J 1997:4(4):190-199. The rhetoric of ‘efficiency’ frames much current debate about how limited health care resources should be used. Clinicians increasingly turn to economic evaluation literature to discern evidence-based claims of ‘efficiency’ or ‘cost effectiveness‘ from empty ones. Economic evaluation research is designed to compare health services on the basis of their efficiency (eg, how well they produce health benefits relative to resource costs). Although economic studies appear throughout the respirology literature, relatively few are complete economic evaluations. Economic evaluation studies serve various purposes, including critical evaluation and persuasive marketing, which produce studies that vary in research agendas and scientific rigour. This paper is intended to serve clinicians and consumers of economic evaluation studies by: introducing economic evaluation research information as a policy making tool; describing the three basic elements and three basic types of economic evaluation (cost-benefit, cost-effectiveness, and cost-utility analyses); and reviewing some limitations of economic evaluation information for policy decision making. The usefulness of economic evaluation research for policy making depends not only on the scientific merit of the analysis but also crucially on whose specific concerns the research questions address.

E conomic evaluation in health care has been defined for- mally as "the comparative analysis of alternative courses of action in terms of both their costs and their consequences" (2).Benjamin Franklin's historic correspondence on "moral algebra" suggests that the reasoning underlying economic approaches to making tough decisions is neither new nor necessarily esoteric.What is relatively new, however, is the popularity of applying such reasoning to health care decision-making explicitly and formally.Many forces drive the use of economic evaluation in health care, but probably foremost is the widely perceived crisis in health care spending.Health care providers, managers and payers who cope with fixed budgets increasingly turn to economic evaluation when pursuing value for the health care dollar.
Economic findings appear sporadically throughout medical and respirology literature.Diverse clinical and administrative topics illustrate the many ways that economic information may be constructed to aid decision-making in the field.Economic criteria have been used to evaluate administrative protocols such as standard treatment regimens or pharmacy formularies.Economic criteria are used to argue the value of preventive interventions such as air filters for health care workers, diagnostic interventions such as the diagnosis of Pneumocystis carinii pneumonia, and treatment interventions such as antibiotics for respiratory infections.Economic criteria frequently guide provincial and private insurance coverage policies for new devices and drugs.
Increasingly, practicing clinicians need to understand not only economic evaluation research studies, but also how information in economic evaluations can inform policy making.The objectives of this paper are the following: to introduce economic evaluation research information as a policy making tool; to describe the three basic elements and three basic types of economic evaluation; and to review certain limitations of economic evaluation information for policy decision-making.The field of theoretical and applied health economic evaluation is vast.It rumbles with internal debate and is often pelted by external criticism.Nevertheless, it grows monthly in methodological sophistication and popularity.This paper does not attempt to cover all of the issues concerning the responsible conduct and use of economic evaluation studies in health care.Rather, it is intended to sketch the basic construction of economic evaluation information and to describe how this construct can guide decision makers in different policy contexts.

ECONOMIC EVALUATION INFORMATION
AS A POLICY TOOL The fundamental principle that guides economic evaluation is, of course, an economic perspective on health policy problems.Economics is primarily concerned with how material resources are created, distributed and used among individuals and society.By and large, the economic world understands and solves policy problems based on the assumptions that material resources are limited, that wants and needs for these resources are unlimited, and that, in the economic realm, people are motivated by self-interest.Consequently, there is a constant struggle to both produce more resources (such as goods and services) and allocate them appropriately among members of the community.Societies have developed formal and informal ways of moving resources from one interested party to another (eg, markets, government funding, taxation, theft, charity, etc).No universal mechanism or principle applies to all goods (3).Instead, policy decision makers are constantly faced with the task of improving systems for determining who gets which resources on the basis of what claims to use toward what social or individual ends.A fundamental policy problem in health economics concerns how to allocate health services, their attendant benefits (in health, well being, etc) and costs (in expertise, effort, dollars, etc) to both individuals and populations in need.A related problem is how to allocate resources on a broader social scale among health care and other programs such as education and housing.
The problem with health resource allocation arises at micro, meso and macro levels of health policy making (4,5).At the micro level, clinicians are familiar with the resource allocation problem in terms of medical triage (eg, how should too few intensive care beds be distributed among too many needy candidates?)One solution may be to fund more intensive care units, but this only bumps the dilemma up to the meso level of the health system administrator (eg, how should too few dollars be allocated among more intensive care beds and other services such as primary care, laboratory services,

Economic evaluation
Dear Sir, In the affair of so much importance to you, wherein you ask my advice, I cannot, for want of sufficient premises, counsel you what to determine; but, if you please, I will tell you how... my way is, to divide half a sheet of paper by a line into two columns; writing over the one pro, and over the other con; then during three or four days' consideration, I put down under the different heads short hints of the different motives, that at different times occur to me, for or against the measure.When I have thus got them all together in one view, I endeavor to estimate their respective weights; and, where I find two (one on each side) that seem equal, I strike them both out¼ and thus proceeding I find at length where the balance lies; and if, after a day or two of farther consideration, nothing new that is of importance occurs on either side, I come to a determination accordingly¼Wishing sincerely that you may determine for the best, I am ever, my dear friend, yours most affectionately, B Franklin ("Moral Algebra, or Method of deciding doubtful Matters with One's Self".Letter from Benjamin Franklin to Joseph Priestley, 1772 [1]) etc?).If the administrator wishes to avoid such dilemmas by simply demanding more resources for the entire health care system, the public policy maker faces a difficult trade-off at a higher macro level of policy making (eg, how many dollars should be taken from education, roads or private taxpayers' pockets to build up the health care system?).Economic trade-offs are inevitable at all three levels of health policy decision-making, from the design of clinical protocols to the administration of health services, to the public administration of social programs.In this sense, the clinician, as well as the bureaucrat, acts as a 'policy maker'.In this paper, 'policy makers' are those who decide how to use resources to produce health, whether they are providers, administrators or payers.Policy makers at all three levels increasingly turn to economic evaluation, among other types of information, to identify problems, devise solutions and make prudent spending decisions.
Economic evaluation is concerned with how resources (money, expertise, technology and capital) can be used to produce a desired outcome.In health care, this outcome is usually health but may also be more broadly conceived as well-being.Each, in turn, can be measured at the individual or population level.A patient who occupies a hospital bed leaves one less bed available for others.A physician who bills the provincial government for a procedure leaves fewer dollars available for other physicians to draw on for the care of other patients.Every investment in one health service has an opportunity cost (ie, it leaves less to invest in other means for improving health).Fiscal pressures have challenged decision makers to make more conscious and careful investments for producing health, particularly not to waste resources.From an economic perspective, prudent decision-making involves considering opportunity costs to maximize the health benefit from a given amount of resources (beds, procedures, drugs, etc) to avoid needless waste.The purpose of economic evaluation is to compare various configurations of these resources based on their efficiency (ie, how well they produce health or another desired outcome, relative to their resource cost).The decision maker's task, then, is to choose the most efficient configuration of resources.
In recent decades, most concern regarding health policy efficiency has focused on health care technologies, which include not only drugs, devices and procedures, but also programs and delivery systems (6).Policy makers -those buying or using health technologies -turn to economic evaluation to determine the most efficient technology for a given purpose.On the other hand, entrepreneurs -those that develop and sell health technologies -turn to economic evaluation to convince policy makers that technology is the entrepreneur's most efficient choice.A recent brochure advertising a conference on pharmacoeconomics and quality of life studies, for example, describes such research as "realworld applications to effectively communicate product value ¼ gain ¼ formulary acceptance ¼ increase credibility ¼ influence decision makers" (7).Economic evaluation research serves the conflicting purposes of critical evaluation and persuasive marketing.These produce a field of research that varies tremendously in scientific agenda and rigour and often confuses policy makers wishing to make decisions based on economic evidence.Now, problems of identifying good economic evaluation studies and considering their potential biases and limitations for informing policy decisions are addressed.

ELEMENTS OF AN ECONOMIC EVALUATION
The concept of efficiency is always based on the relationship between two elements: production outputs and resource inputs -more commonly termed costs and benefits.Further, because policy making involves choices and trade-offs among options, the relative efficiency of alternative choices (rather than any one technology in a vacuum) is key.Economic evaluation is designed to indicate which of two or more interventions is the most efficient course of action.It is important to recognize that doing nothing or maintaining the status quo is always one of the possible alternative.
The three basic ingredients of an economic evaluation, then, are exemplified in Ben Franklin's letter.The first necessary elements are choices (alternative policies from which to choose).Choices include whether to embark on one or more possible courses of action, which may include maintaining the status quo.The second element is a thoughtful and thorough compilation of each alternative's likely benefits (Ben Franklin's "pros").The third necessary element is an equally thoughtful and thorough account of each alternative's likely costs (Ben Franklin's "cons").Analysts must gather and analyze these three types of information to perform an economic evaluation of health care services.It can be analytically onerous to identify realistically the relevant trade-offs and to measure diligently all the relevant and possible costs and benefits.Excellent primers for clinicians describe techniques for conducting a rigorous economic evaluation of health care (7).The types of costs and benefits that are commonly considered in economic analyses of health care technologies are reviewed briefly below.
The costs of a health technology include direct and indirect costs as well as nonmonetary burdens imposed by the service itself.Common resource costs include labour, lost leisure time, capital investments, such as equipment and training, and overhead.Costs may accrue to any number of parties.For example, the physician (labour, effort and time), the patient (lost wages, lost leisure time, disability from side effects), the hospital (overhead, nursing, etc), the family (wages or leisure foregone to care for a convalescing family member at home) or the payer (reimbursment for physician and hospital bills).Further, costs can shift from one party to another (eg, a patient sent home early from the hospital may lower the insurer's hospital bill but may increase the burdens borne by the patient and family).Consequently, it is important for economic evaluations to specify a viewpoint such as whose costs and benefits are to be considered.
In economic evaluations, costs are usually expressed in financial terms.This requires each resource and burden to be assigned a dollar value, which is straightforward for some costs but difficult for others.Analysts must make assumptions to assign dollar values to costs for which no market 'prices' or literal budget allocations are available (eg, lost leisure) and for resources that are widely shared (eg, institutional overhead).
Identification and quantitative measurement of health care benefits are fast growing fields of health services research.Various approaches have been developed by economists, decision analysts and clinicians.Different approaches tend to serve different types of policy decision making.The approaches fall roughly into two classes: those that express benefits in terms of improved health outcomes ('effectiveness') and those that express benefits in more generic terms.
Methods for measuring health outcomes range from particular to general.At one end of the spectrum, a specific clinical end-point (eg, respiratory volume or prevention of a particular disease) may be used, and two technologies may be compared to determine which achieves the outcome at the least cost.However, this only provides comparative information about technologies designed to accomplish the same thing and may not take into account issues such as side effects.A more general measure is disease-related quality of life, which more broadly assesses the improvements of symptoms and disabilities of patients with a particular disease (9).This allows comparisons among technologies that are designed to combat the same disease but perhaps target somewhat different manifestations of the disease.
Others have developed even more general measures of health quality of life that apply across a spectrum of diseases, interventions and health states (10)(11)(12).Such measures allow for economic evaluations that compare the cost effectiveness of technologies that treat different diseases.They are designed to assess survival gains adjusted for the quality of added life with the premise that people may prefer to live a shorter, healthier life than a longer, severely impaired one.If this is the case, an intervention that achieves the former might be judged more 'effective' (despite lower life expectancy) than the latter.
A final outcome class expresses the benefits of health care in yet more general terms.These measures estimate the overall value a person places on an intervention, including potentially both health and nonhealth related benefits.An example is 'willingness to pay' (ie, a person's hypothetical willingness to pay to gain the outcomes that the intervention offers).The more general the measure, the more feasible it becomes for policy makers to compare interventions that are very different in their effects.Such generality informs decision makers especially at the administration and public policy levels because they often have to allocate resources among programs that target different populations, deal with different diseases and have different effects.
General benefit measures, such as willingness to pay, originate from familiar economic market reasoning.Underlying values are reflected in the trade-offs one makes every day.If one prefers to see a movie rather than buy a book for the same price, the movie is said to generate more 'utility' or satisfaction.Health economists have imported this consumer reasoning to measure how people value health or well-being.Researchers ask patients (or citizens) about their willingness to pay to obtain various experiences of good health or forego experiences of ill health.The responses can then be used to estimate the value that individuals place on those experiences.This approach has practical, commonsense appeal to many, especially economists.
However, ethicists note that not all human experiences can, or should, be valued in monetary terms, particularly willingness to pay (3).For example, because 'money can't buy you love' it would be inappropriate to expect people to estimate the monetary value of relationships (even if, in practice, relationships can be financially costly or beneficial and are sometimes foregone for material gain).Similarly, some question whether it is valid, in either moral or measurable terms, to ask people to place monetary values on health if citizens are socialized to value health quite separately from money (the issue of escalating health care costs notwithstanding).
Another criticism is that a person's ability to pay will partly determine willingness-to-pay, which leads the values of the wealthy to dominate any population estimate of the utility of a health state.If the wealthy and the poor have different opinions, the poor will be disadvantaged by such a measure.Some people have proposed adjusting responses for ability-to-pay, but the validity of such weights remains controversial (13).One policy analyst argues that such measures should represent the values of the stakeholders, or those who would be directly affected by the policy decision (14).Other advocates of economic evaluation in health care oppose emphasizing any particular group's values and instead promote measurement of how the "general public", on average, values health interventions (15).
Unfortunately, no measure of benefit is ideal.Not all measures are relevant to all policy decisions.For instance, diseaserelated quality of life measures can only be used for decisions about different technologies for dealing with a particular disease, and cannot be used to compare the cost effectiveness of technologies for different health problems.Also, a growing ethics literature seriously questions both the construction and application of general health outcome measures to health care policy making via economic evaluation (14,(16)(17)(18).
When choosing an 'appropriate' measure of benefit for an economic evaluation, the analyst must side with one set of assumptions or another and must tolerate one type of measurement imprecision or another.Also, unfortunately, the different outcome measures do not necessarily yield the same answer regarding cost-effectiveness (19); that is, a given service, at a given cost, may have varying cost-effectiveness depending upon one's definition and measurement of effectiveness.Because different measures can alter conclusions regarding effectiveness or cost effectiveness, the analyst is further challenged to choose his or her necessarily imperfect outcome measure wisely and to describe critically and clearly its strengths and limitations.

TYPES OF ECONOMIC EVALUATION
Three types of economic evaluation are delineated roughly by the benefit measure used and the types of policy decision making they can support.The three types are Economic evaluation cost-benefit, cost-effectiveness and cost-utility analyses (20).cost-benefit, cost-effectiveness and cost-utility analyses (20).
Cost-benefit analysis supports the policy question: will this choice result in a net gain or loss of benefit?In costbenefit analysis, both costs and benefits are measured in the same currency, usually dollars.Analysts subtract total costs from total benefits to determine, simply, in Ben Franklin's terms, "where the balance lies" (ie, whether a choice yields a net gain in benefit, after costs).Both costs and benefits must be expressed in the same terms so that net benefit can be determined by subtracting losses from gains.This requirement creates controversy in health care.Some find it morally noxious to assign dollar values directly to life or health.Some also feel that cost-benefit analysis invites impractical and nonsensical comparisons between social programs when they are conceptualized as financial investments.For example, in theory, one could apply cost-benefit analysis to a comparison of the municipal bus system and the community hospital to discover that, indeed, one pays off far better than the other.This does not mean that it is necessarily wise public policy to divest from one or the other.To be useful, an economic evaluation has to address trade-offs that are both relevant and realistic to the decision maker.
Cost-effectiveness analysis is used to inform decision makers about alternative means to a prespecified end.It answers the policy question: given a desired health outcome, which technology achieves it least expensively?Benefits need not be expressed in monetary terms but can be expressed, and measured, in 'natural units' (eg, the number of accurate diagnoses yielded by competing diagnostic technologies).Cost-utility analysis is perhaps viewed as an extension of cost effectiveness analysis, but one in which the desired health outcome is specifically quality-adjusted life-years.It is more general than cost effectiveness analysis because it allows comparisons among any interventions that affect quantity and quality of life and can more easily consider diverse outcomes such as side effects and multiple effects.
A vast academic industry has grown around the task of converting the quality of human experiences into measurable quantities to support cost-effectiveness and cost-utility analyses (as well as more generic 'outcomes research').Although practical and promising, this field abounds with methodological and philosophical debate.Most quality of life measures are based on the assumption that it is meaningful to have individuals rank possible health states along some single scale from 'bad' to 'good' and then aggregate these judgments to get a collective measure of the value of a health state.Many have questioned whether individuals' preferences can be meaningfully summed in this way and used as a proxy for more societal valuations (21).Also, researchers necessarily impose some of their own value judgments (wisely or not) in the way they develop measures such as quality-of-life indices, quality-adjusted life-years or patient preferences.Measurement scales depend upon quite specific (that is, thought-circumscribing) questions so "human values must also shape the very questions asked" (16).

WHAT IS NOT AN ECONOMIC EVALUATION
In a policy environment of growing cost consciousness, terms such as 'cost-effectiveness', 'efficiency' and 'cost saving' carry powerful persuasive currency.Medical journals are peppered with erroneous claims of treatments' cost-effectiveness or efficiency.These term have become passwords to legitimize an intervention as worthwhile.Entrepreneurs advocating particular interventions sometimes apply these labels too liberally.Three main sources of misleading information about the economic efficiency of an intervention are incidental and unsubstantiated efficiency claims reported from research that did not focus on economic evaluation; economic studies of only an intervention's cost (or worse, cost savings) that inadequately address the other important elements of a full economic evaluation; and economic evaluations that are not rigorously designed, conducted or reported according to scientific standards.
One of the most basic features of a good economic evaluation report is that it reports an economic evaluation per se.Often, papers address 'economic issues' as an afterthought to a clinical topic (usually in the concluding discussion section).Such ad hoc discussions or claims can be an unreliable source of economic evaluation information.True economic evaluation studies take great care in framing the research question, build on a large, existing literature concerning both theory and empirical issues, involve numerous methodological judgments and produce rich findings.Consequently, the report of an economic evaluation study should possess the customary heft of a research paper, with introduction, methodology, findings and discussion sections each devoted explicitly to the problem of economic evaluation.Unless authors claiming cost-effectiveness refer to substantive economic evaluation evidence, readers should dismiss such claims as rhetoric.
Commonly, economic analyses in the medical literature do not report complete economic evaluations but instead address economic costs (or benefits) in isolation.For example, cost studies might account for the social financial burden of particular respiratory diseases such as tuberculosis or respiratory distress in newborns.Other cost studies address delivery issues such as typical hospital costs for a particular course of care.While these studies sometimes offer useful information (especially for designing economic evaluation studies), they should not be confused with genuine economic evaluations.
When evaluating an intervention, the authors should first identify and discuss the alternatives.These alternatives should be evaluated in commensurate detail.They should be realistic and make sense within the context of the policy decisions to be made.Second, costs of all alternatives should be identified and measured in common terms.The basic 'laundry list' of costs should make sense and seem complete.Clear discussions of issues regarding cost measurement are available elsewhere (22).Finally, the benefits (usually effectiveness in the form of some health outcome measure) of all alternatives should be clearly identified and measured.The list of benefits, or "pros", should be thoughtful and thorough.
Further discussion of the problems of identifying and measuring benefits for various types of economic analyses are available (23)(24)(25).
Finally, true economic evaluations, even those that possess all the above elements and are presented in an adequately detailed and focused report, vary in thoroughness and scientific rigour.Features of a good economic evaluation study have been enumerated by Drummond et al (26) and in the field of pharmaceutical evaluation by Torrance et al (27).Such features usually include a clear analytical question specifying what is being compared and on what basis; a comprehensive description of the alternatives being compared; good evidence of effectiveness; thorough identification and accurate measurement of all important costs and consequences; and sensitivity analyses that address how controversial or discretionary assumptions, as well as different decision makers' perspectives, might affect the conclusions of the analysis.Researchers attempting to satisfy these criteria are faced with formidable methodological challenges, and many economic evaluations fail to specify adequately their analytical perspectives, limitations or biases (28).Serious readers should consult methodological references for a more comprehensive introduction to empirical problems.This paper does not address the technical dimensions, except to tell readers that like all research, economic evaluation studies can be designed, conducted and reported either well or poorly.Further, half the challenge in economic evaluation is to be painstakingly explicit about what the study did and did not consider, and how costs and benefits might have been addressed differently under different policy assumptions.

ISSUES CONCERNING THE USE OF ECONOMIC EVALUATIONS -MAKING POLICY DECISIONS
Once individuals obtain true economic evaluation information they must then figure out how to apply the information wisely to decision making.The usefulness of information provided by even a high quality economic evaluation study depends crucially on the policy context (28).No one economic evaluation simply indicates the best course to take.It is important to understand and interpret the findings of economic evaluations critically.As with all critical appraisal of research literature, this involves considering possible biases.
Of equal importance, critical reading requires careful consideration of how well the study's research question and findings relate to the actual decision to be made in a given policy context.The critical reader must cultivate a sense of not only what is a good economic evaluation, but what is a policyrelevant economic evaluation.Even meticulously executed economic evaluations can suffer from the following limitations: low generalizability (the technologies, applications, and populations in the research setting may not correspond to the setting where the technology would actually be applied); irrelevant viewpoint (crucial stakeholders' costs or benefits may be disregarded due to the particular viewpoint taken by the analysis); missing information (some costs and benefits will be disregarded due to the technical difficulty of measuring and incorporating them into the analysis, and if they are important to the decision, they are left to be considered in some other ad hoc manner); and distributive concerns (the technology that most efficiently produces benefits may not be the one that most fairly allocates benefits and costs).These issues will be illustrated using the example of a cost effectiveness analysis of high efficiency particulate air-filter (HEPA) respirators, published in The New England Journal of Medicine (29).A public policy-level decision maker, the United States Centers for Disease Control and Prevention (CDC), had recommended that HEPA respirators to prevent outbreaks of tuberculosis in hospital workers.However, unfortunately for administrative-level decision makers (hospital administrators), HEPA respirators cost about 10 times as much as conventional dust-mist respirators.The study analyzed the costs of switching to the HEPA respirators versus the benefits gained in preventing cases of occupationallyacquired tuberculosis in hospital workers at the University of Virginia Hospital.The study concluded that, "...preventing a single case of occupational tuberculosis during the next 41 years by implementing the proposed requirements for HEPA respirators and a respiratory-protection program would cost this hospital between $1,333,090 and $18,508,947" (29).The study generated several critical letters to the editor (30,(31)(32)(33) that illustrate some of the issues arising in trying to apply the results of a published study to local policy decision-making elsewhere.Generalizability: If an intervention is cost-effective in a research setting, will it be cost-effective in the clinical setting?A similar problem is found in clinical trials, where researchers distinguish between 'efficacy' (how well a technology works under ideal circumstances) and 'effectiveness' (how well a technology works in typical practice).This problem of generalizability in the practice context is compounded when using economic evaluation research.For example, a treatment that is cost-effective in Canada may not be cost effective in the United States, simply because of differences in funding and organization of health care (34).Cost-effectiveness information is sensitive not only to how actual clinical practices and target populations alter the average effectiveness of an intervention, but also to how clinical and administrative structures affect costs.
A program to provide an ostensibly cost-effective intervention can be designed in an efficient or inefficient way (35).In the words of one health administration expert: One can ensure that any service will be costeffective, compared to others.One need only to write the standards, requirements, and reimbursement methods in the right way... Of course, if one overdoes it ... one may make those services available only to a small fraction of those who could potentially benefit from them, but that is the appropriate logical extension of a preoccupation with a single criterion [cost-effectiveness] to the exclusion of all others (35).
Program administrators obsessed with the 'bottom line' might drastically limit the use of a program to maintain its cost effectiveness.However, clinicians left to their own devices have a tendency to do just the opposite; they expand the uses of an intervention to populations and purposes that have never been studied.Neumann and Weinstein (36) described a 'Peter Principle' for new technologies, which dictates that, "...a technology will expand its use until it has found its way into medical applications that are cost ineffective" (emphasis added).Expansive technology diffusion can make cost effectiveness information obsolete once the economic evaluation has done its job of justifying payment coverage.
A second issue is that cost-effectiveness information, like clinical trial evidence, is only literally applicable to the populations studied.If, for example, only males, the very sick or the elderly were studied, cost-effectiveness would simply remain unknown for females, the less sick or the young.Decision makers may extend study findings to these populations only hypothetically, not literally.If decision makers are concerned not only with overall population health but also with the effectiveness and cost differences among certain subpopulations (eg, between the sexes, the sick and well, young and old, etc) they can only get this information from economic evaluations that address the relative costs and effectiveness of treatments among these particular groups.Consequently, some economic evaluations explicitly study effectiveness or cost differences in subpopulations of the central study population.
However, studies that focus on a more general population may offer only limited subpopulation analyses.One problem is that different demographic groups can often be expected to have different evaluations of their health states and even to experience different salient costs and benefits, which necessitates collecting specialized information on their behalf.Another is that sample sizes, or the number of patients participating in the research, are usually selected to detect statistically significant effects in the population of primary interest.Because the subpopulation samples are necessarily smaller, the study may not have adequate statistical power to determine significance of findings in the subpopulations.Correcting both of these problems adds expense and effort to the research, which, when designed correctly, actually treats all subpopulations as primary research populations in their own right.
How generalizable is the study of HEPA respirators?The report addressed costs and benefits to a single hospital in Virginia.The generalizability of the results to other hospitals is limited by, among other things, the cost structure of the hospital, the prevalence of tuberculosis-infected patients at the hospital and even by the behaviour of staff using the respirators.A physician at a South Bronx hospital in New York, for example, points out that the findings do not apply to that hospital (30).This is because they experience over 10 times the prevalence of tuberculosis in patients, and only a few dedicated staff visit tuberculosis patients' rooms about a third as often (necessitating the distribution of fewer expensive HEPA respirators).These differences substantially lower the cost of using the HEPA respirators to prevent new cases of tuberculosis at the New York hospital.Another physician from a North Carolina hospital estimates that the cost effectiveness for their own hospital would be even higher than for the Virginia hospital because highly sophisticated isolation room technology prevents infection more effectively, even with the old respirators (31).Staff from yet another North Carolina hospital report that the study results are not necessarily valid because, in actual use, the HEPA face masks often do not seal properly due to employee error or problems with construction (32).Leaky seals cancel the preventive effect of the mask, which may make cost effectiveness estimates much higher than for properly functioning equipment.Missing information: Economic evaluation is an entirely quantitative undertaking.Costs and benefits of all alternatives are measured and weighed as quantities.Consequently, anything of value that is unquantifiable is generally left out of the analysis.Any cost must be expressed as a number (usually in terms of dollars), whether it be suffering, inconvenience or out-of-pocket spending.Any 'benefit' must also be expressed as a number, whether it be longer life, fewer trips to the bathroom, the ability to walk or happiness.The field of health economics has spawned many creative, compelling, alarming and often controversial ways of capturing health experiences on quantitative scales.Regular users of economic evaluations should become especially familiar with the literature on outcome measurements.
Numbers used to weigh costs and benefits are not so much hard facts as ways of expressing what people value.Inevitably, much relevant information about individual and social values is lost when experiences of health or illness are converted to numerical values.This lost information is unavailable to influence the findings of economic evaluations.While information informing an economic analysis is quantitative, some information crucial to the policy decision will inevitably be qualitative.Decision makers may ignore such information; a compelling economic evaluation may inadvertently blind them to it (37), or they might try to consider it in addition to results of an economic evaluation.Most treatises promoting the use of economic evaluation include a disclaimer advising decision makers to consider economic evaluation as only one source of valuable information (38).However, there are no clear rules for how decision makers should use unquantified costs and benefits to modify conclusions of a quantitative economic evaluation.
What information was left out of the HEPA respirator analysis?The authors themselves suggest other variables that, if measured, may render HEPA respirators even less cost effective (29).The respirators are bulky, uncomfortable and make it difficult for some staff to talk and even breathe properly.This could compromise patient care at an undetermined cost.Some opportunity costs were not well defined.For instance, for a hospital to comply with the CDC policy and purchase HEPA respirators, it has to reduce spending on other services.If these services are effective there may be health costs.However, it is important to remember that if these other services are even less cost-effective than HEPA respira-tors, the hospital might increase overall efficiency by investing in HEPA respirators.Viewpoint: An economic evaluation that is on the right topic and addresses the right population may still present findings that do not apply to the policy decision at hand.Two aspects may particularly affect the relevance of the evaluation: the viewpoint taken by the analysis and stakeholder interest in the study design.
The analytical viewpoint determines whose costs and benefits are considered important.An economic evaluation of a medical intervention may take any number of viewpoints (2).In the case of the HEPA respirators, for example, the researchers took the viewpoint of a single university hospital (29), but only the hospital's costs (eg, cost of purchasing respirators and implementing their use) and benefits (eg, tuberculosis cases prevented in hospital workers) were included.Economists have argued that a public-interest economic evaluation should take what has been called a 'societal' viewpoint, "¼that is, all costs and benefits should be identified regardless of who incurs the costs or who receives the benefits" (27).Societal costs and benefits may be considered for narrow individual health interventions, as well as for broad investments in social programs.Had they taken a societal point of view, the CDC would have considered the broader costs and benefits, such as the possibility that one infected health care worker might infect others in the community, creating additional burdens on them, their employers and the health care system.
Closely related to the question of whose costs and benefits to consider is the issue of which potential trade-offs to consider.Different policy makers have different choices to make, and economic evaluation is most relevant when the alternatives in the study are the same alternatives that the policy maker is considering.Recall that a technology cannot be efficient in itself but only in comparison to something else.
The way that the contest is set up can crucially determine who wins, and is one of several design features that can be manipulated by parties interested in a particular outcome.Economic evaluation is not only a scientific enterprise but also a powerful marketing tool for commercial interests and for convincing providers and payers of the value of their technology.Proprietary interests, in particular pharmaceutical companies, have become quite heavily involved in the development and practice of economic evaluation of medical care.Policy analysts are divided on whether this is a good thing because it generates more economic evaluation studies and, thus, more information for decision-making, or a bad thing because the information may be tainted by a commercially self-interested perspective.Privately interested studies may fail to consider how costs and benefits will be borne throughout society (39,40) or may compare technologies with irrelevant alternatives to make them look good (41).
Relevant trade-offs are different at different levels of policy making.For example, a physician may be interested in comparing the relative efficiency of two medications for treating a particular disease to minimize the patient's out-ofpocket spending.The hospital administrator may be inter-ested in the relative efficiency of in-patient versus out-patient management of the disease.The public policy analyst might consider a much wider range of alternatives for dealing with the same disease (eg, whether money might be better invested in prevention than in treatment services).The breadth of a public policy maker's view is still usually limited to trade-off possibilities within a given budget or agency.Academic analysts, on the other hand, often take the broadest societal trade-offs.To theorists, relevant alternative investments may comprise entire economic sectors (health care, education, private industry, etc).Such determinants of health analyses, for example, consider national prosperity or nutrition as reasonable investment alternatives to medical care that may more efficiently combat disease in populations (42).Unfortunately, few decision makers are poised to make the radical resource reallocations suggested by these studies, even when their findings are enlightening and their recommendations seem prudent in principle.
Proprietary interests tend to bring their own concerns to economic evaluation.They naturally see alternative technologies as competitors in the market.For example, a commercial manufacturer may want to see its own product compared with a clearly weaker alternative, rather than with the most promising alternative (41).The pharmaceutical industry has become one of the principal commissioners and funders of not only in-house economic evaluations but, importantly, of drug evaluations published in peer-reviewed health journals (40,41).Hillman et al (41) note that, for a number of reasons, economic evaluation of drugs may be more easily biased by proprietary interests.
Economic analysis is difference from clinical trials conducted at academic centers subject to FDA [Food and Drug Administration] regulations.In contrast to clinical trials, economic studies generally use data and analytical methods of varying degrees of precision and power that are usually unstandardized, rather than standardized designs and analytic techniques used under the scrutiny of an external regulatory agency; involve subjective opinion and interpretation about what the results demonstrate, rather than limiting themselves to narrow conclusions about safety and efficacy; look at products selected by the pharmaceutical firms from among agents already (or nearly) in clinical use, rather than at all drugs for which FDA approval is sought; may make selective use of propietary data gathered after marketing has begun, rather than disclosing all data by FDA requirement; and are funded and overseen most commonly by marketing departments, rather than medical or scientific divisions (40).
Within the field of health economics there is a highly polarized debate over whether the involvement of private commercial interests necessarily biases economic evaluations to make them misleading to socially interested decision makers, or whether such biases could simply be eliminated through rigorous scientific scrutiny of the resulting studies (27,28,40,41,43,44).The New England Journal of Medicine will not even review papers coauthored by pharmaceutical company employees (44).Similarly, the editors of Health Economics urge authors to declare their commercial affiliations and urge readers to judge even peer-reviewed cost effectiveness studies very critically, especially where results favor new commercial products (39).In contrast, some who develop scientific standards for economic evaluation believe that rigorous methodology and peer review will overcome any potential biases in design (27,43).Underlying these debates are fundamental ideological disagreements regarding the ability (or responsibility) of commercial industries to serve the public interest and the role of market competition versus regulation in getting them to do so.Distributive concerns: Economic evaluation, as it is currently practised, is firmly rooted in utilitarian principles.Utilitarianism suggests that the best policy choice is the one that yields the greatest net benefit for a population (regardless of how benefits are distributed among individuals in the population).This assumption guides a number of technical aspects of economic evaluation including how costs and benefits are defined, how they are aggregated across individuals and how the results of the study may be used to indicate the appropriate policy choice.It is beyond the scope of this paper to discuss all the ways that utilitarian assumptions permeate economic evaluation (45).It is simply noted that utilitarian-based reasoning (and economic evaluations based on it) promote a particular view toward equity and that alternative principles exist for the fair distribution of resources.Drummond (19) has described the ethic driving economic evaluation as follows: Economic efficiency embodies an ethic of its own, ie, resources should not be used in a given activity when they would generate more benefits if used elsewhere.Inefficient use of technology means that the community as a whole is worse off.Therefore, it could be argued that it is unethical not to consider costs.
While this ethic has common sense appeal (especially to budget managers), it may conflict with other ethics that traditionally concern public and clinical policy makers.Distribution of benefits and costs across individuals and groups must usually be considered.The most efficient choice may actually load all benefits onto one segment of a population and all costs onto another, even though the 'net' benefit to the population as a whole is great.Consequently, the most efficient distribution may not be the most equitable, morally acceptable or politically viable.Some studies suggest that health policy decision makers tend to prefer less efficient technologies that yield broadly distributed benefits over more efficient technologies that benefit relatively few individuals (45).Others have noted a popular preference for the dramatic 'rescue' of a few, over more modest benefit to an entire population (47).An alternative ethic in health care is the 'contractarian' idea that the best health care investment is the one that maximizes a popu-lation's health, subject to limits on the suffering or burden that any person or group bears (48,49).
The HEPA study and its critics illustrate some of this tension.The CDC's public policy directive for hospitals to switch to HEPA respirators does not benefit (or burden) all hospitals or their workers equally.If this national-level decision is reversed based on economic evaluation data from a Virginia hospital, showing that the program can be exorbitantly expensive, this decision puts a higher number of health care workers in a New York hospital at risk of contracting multi-drug-resistant tuberculosis (30).Whether it is fair to burden some hospitals (and their workers) with policies that yield more benefit to others is a question of social justice.The answer is often not clear because different principles of social justice are often applied to different situations (3,50).

CONCLUSION
Clinicians interested in basing decision-making on evidence should scrutinize economic evaluation research to ensure that it is rigorous, complete and relevant.Compared with clinical evaluation (eg, randomized, controlled clinical trial evidence), standards and techniques for economic evaluation are in an early stage of scientific development.Economic evaluation is also currently subject to less public interest regulation, is often subject to less professional peer oversight (41) and is often funded or conducted by drug and device manufacturers who bring vested interests to the studies (35).These factors make the critical appraisal of economic evaluation studies both more important and more difficult.
On the positive side, economists and other researchers make constant and substantial improvements to the theory and tools of economic evaluation.Perhaps as important, publicly interested technology assessment bodies have established preliminary guidelines for setting scientific and ethical standards in the field (27,51).Nevertheless, much of the economic evaluation information available in today's clinical literature falls below basic standards for conduct (28,52).
The concept of 'cost-effectiveness' now frames much of the debate over how limited health care resources should be used.Economic evaluation studies have growing appeal to health policy decision makers at all levels who wish to streamline clinical practice protocols, design efficient delivery organizations or ration health and other public services.Clinicians will find themselves presented with 'evidence', as well as 'rhetoric', of cost-effectiveness in support of policy changes.They may increasingly turn to economic evaluation literature to discern evidence-based claims from empty ones.It is hoped that this paper will serve the novice consumer with some guidelines regarding the basic elements of an economic evaluation, as well as cautions regarding how to identify biases and limitations common to this research.Readers interested in learning more will find excellent introductory literature describing basic techniques of economic evaluation (8,(53)(54)(55), as well as much policy literature on the controversies surrounding its applications to clinical practice, technology assessment and ethical health care rationing.