Past Conference

Expert Judgment Workshop Documents

Mar 13, 2004 - Mar 14, 2004

About the Event

Expert Judgment Policy Symposium and Technical Workshop   

Policy Guidance Documents

Guidelines for Carcinogen Risk Assessment EPA/630/P-03/001F March 2005 | Cancer_Guidelines_3-25-05.pdf, 468KB

Risk Assessment Forum - U.S. Environmental Protection Agency

Abstract: These guidelines revise and replace the U.S. Environmental Protection Agency's (EPA's, or the Agency's) Guidelines for Carcinogen Risk Assessment, published in 51 FR 33992, September 24, 1986 (U.S. EPA, 1986a) and the 1999 interim final guidelines (U.S. EPA, 1999a; see U.S. EPA 2001b). They provide EPA staff with guidance for developing and using risk assessments. They also provide basic information to the public about the Agency's risk assessment methods.

Circular A-4 to the Heads of Executive Agencies and Establishments, September 17, 2003 | OMBa-4.pdf, 550KB

Office of Management and Budget (OMB)

Abstract: This Circular is designed to assist analysts in the regulatory agencies by defining good regulatory analysis - called either "regulatory analysis" or "analysis" for brevity - and standardizing the way benefits and costs of Federal regulatory actions are measured and reported. Executive Order 12866 requires agencies to conduct a regulatory analysis for economically significant regulatory actions as defined by Section 3(f)(1). This requirement applies to rulemakings that rescind or modify existing rules as well as to rulemakings that establish new requirements.

Estimating The Public Health Benefits Of Proposed Air Pollution Regulations | NRC2002.pdf, 1.92MB

National Research Council of the National Academies

Abstract: The U.S. Environmental Protection Agency (EPA)has estimated that thousands of premature deaths and numerous cases of illness, such as chronic bronchitis and asthma attacks, could be prevented by reducing exposure to air pollution. These estimates come from regulatory health benefits analyses, which attempt to quantify changes in the expected cases of mortality and illness that are likely to result from proposed air pollution regulations. The estimates are often controversial, and the methods used to prepare them have been questioned.

In 2000, Congress recognized concerns about the methods used by EPA and emphasized the need for "the most scientifically defensible methodology in estimating health benefits." It directed EPA to ask the National Academy of Sciences "to conduct a study of this issue and recommend to the agency a common methodology to be followed in all future analyses."

Proposed Risk Assessment Bulletin | OMB_bulletin_010906.pdf, 157KB

Office of Management and Budget (OMB)

Abstract: As part of an ongoing effort to improve the quality, objectivity, utility, and integrity of information disseminated by the federal government to the public, the Office of Management and Budget (OMB), in consultation with the Office of Science and Technology Policy (OSTP), proposes to issue new technical guidance on risk assessments produced by the federal government.

Procedures Guide for Structured Expert Judgment | ROCG99.pdf, 414KB

Roger M. Cooke and L.J.H Goossens
Abstract: This document is a guide for using structured expert judgment to quantify uncertainty in quantitative models. The methods applied here have been developed by a host or researchers over the last 30 years. During the years 1990 - 1999, the European Commission and the United States Nuclear Regulatory Commission undertook a joint uncertainty study of accident consequence codes for nuclear power plants using structured expert judgment. The purpose was not only to perform an uncertainty analysis of the US accident consequence code MACCS and the European accident consequence code COSYMA. The wider purpose was to form a baseline for the state of the art in using structured expert judgment for quantifying uncertainty. The reports emerging from this work are intended to be useful outside the community of nuclear accident consequence modeling. Indeed the quantification of uncertainty in the modeling of dispersion, deposition, foodchain transport, and cancer induction, may be used in many fields of environmental modeling and health protection. In the same spirit the methods for using structured expert judgment to quantify uncertainty are applicable far beyond the accident consequence modeling community.

Expert Judgment and Air Quality
Uncertainty in Mortality Response to Airborne Fine Particulate Matter: Elicitation of European Air Pollution Experts
Tuomisto_Wilson_evans_Tainio.pdf, 283KB

Jouni T. Tuomisto, Andrew Wilson, John S. Evans, and
Marko Tainio

Abstract: The authors have performed a structured expert judgment study of the population mortality effects of fine particulate matter (PM2.5) air pollution. The opinions of six European air pollution experts were elicited. The ability of each expert to probabilistically characterize uncertainty was evaluated using 12 calibration questions -- relevant variables whose true values were unknown at the time of elicitation, but available at the time of analysis. The elicited opinions exhibited both uncertainty and disagreement. It emerged that there were significant differences in expert performance. Two combinations of the experts' judgments were computed and evaluated -- one in which each expert's views received equal weight; the other in which the expert's judgments were weighted by their performance on the calibration variables. When the performance of these combinations was evaluated the equal-weight decision-maker exhibited acceptable performance, but was nonetheless inferior to the performance-based decision-maker.

In general, the experts agreed with published studies for the best estimate of all-cause mortality from PM2.5; however, as would be expected, they gave confidence intervals that were several times broader than the statistical confidence intervals taken directly from the most frequently cited published studies. The experts were rather comfortable with applying epidemiological results from one geographic region to another. However, there was more uncertainty and disagreement about issues of timing of the effect and about the relative toxicity of different constituents of PM 2.5. Even so, the experts were in fairly good agreement that an appreciable fraction of the long-term health effects occurs within a few months from the exposure and that combustion-derived particles are more toxic than PM 2.5 on average, while secondary sulphates, nitrates and/or crustal materials may be less toxic. These assessments bring very valuable and relevant information to air pollution risk assessment.

Expanded Expert Judgment Assessment of the Concentration-Response Relationship Between PM2.5 Exposure And Mortality
Walker_pm_ee_report[1].pdf, 1,342KB

Industrial Economics, Incorporated
Draft Version: April, 2004 | PM_expert_elicitation_USEPA.pdf, 902KB

Abstract: In response to the NRC recommendations, EPA is exploring how it might incorporate expert judgment in policy analysis. As a first step in this direction, IEC worked with EPA and OMB scientists to design a pilot expert elicitation to characterize the uncertainty in the ambient PM2.5/mortality relationship. This pilot was designed to provide EPA with an opportunity to improve its understanding of the design and application of expert elicitation methods to economic benefits analysis. For instance, the pilot was designed to provide feedback on the efficacy of the protocol developed and the analytic challenges, as well as to provide insight regarding potential implications of the results on the degree of uncertainty surrounding the C-R function for PM2.5 mortality. The scope of the quantitative questions was limited in that we focused the elicitation on the C-R function of PM mass; this initial elicitation was not intended to thoroughly characterize the uncertainty surrounding individual elements of the PM2.5/mortality relationship, such as the relative toxicity of specific PM components (e.g., diesel particulates).

Peer Review of Expert Elicitation | Peer_Review_USEPA.pdf, 209KB
Carol Mansfield, RTI International

Abstract: The purpose of this memo is to synthesize responses given by reviewers selected to review and comment on the following document: An Expert Judgement Assessment of the Concentration-Response Relationship Between PM2.5 Exposure And Mortality, Prepared by Industrial Economics, Incorporated under subcontract to ABT Associates, Inc., for U.S. Environmental Protection Agency, Research Triangle Park, NC, April 23, 2004.

Expert Judgment Research Manuscripts

Are Decisionmakers at Home on the Range? Communicating Uncertainties in Cost-Benefit Analyses| (Krupnick_CommunicatingUncertainties.pdf, 276KB

Richard D. Morgenstern, Peter Nelson and Alan Krupnick

Abstract: Better and more complete information does not necessarily lead to better policies. Complex information can confound rather than enlighten, or can lead to decision paralysis. Thus, any improvements in capturing uncertainty analytically needs to be matched by improvements in its communication - not only to those who make regulatory decisions on the basis of such information, but also to stakeholders, judges, the press and the general public. Accordingly, this paper presents the results of seven in-depth interviews conducted with former Presidential appointees at the EPA concerning the use of uncertainty analysis. Specifically, results of a case study involving future controls of NOx emissions from power plants was presented to the interviewees, using different visual approaches for displaying the uncertainties. They were asked to express preferences for different regulatory outcomes based on the alternative approaches used to present the information. Tables and probability density functions (PDFs) may be best suited for communicating to high-level decisionmakers, although PDFs tend to move respondents to tight spread options. Respondents also emphasized the importance of presenting technical analysis in context.