Home | Support RFF | Join E-mail List | Contact

 
 
RFF Logo Skip navigation links
FOCUS AREAS
RESEARCH TOPICS
PUBLICATIONS
NEWS
EVENTS
RESEARCHERS
ABOUT RFF
Skip navigation links
Events
Future Events
Past Events
Event Registration

 
Join E-mail List
Please provide your e-mail address to receive periodic newsletters and invitations to public events

 
A | A | A

Expert Judgment Policy Symposium and Technical Workshop   

Policy Guidance Documents

Guidelines for Carcinogen Risk Assessment EPA/630/P-03/001F March 2005 | Cancer_Guidelines_3-25-05.pdf, 468KB

Risk Assessment Forum - U.S. Environmental Protection Agency

Abstract: These guidelines revise and replace the U.S. Environmental Protection Agency's (EPA's, or the Agency's) Guidelines for Carcinogen Risk Assessment, published in 51 FR 33992, September 24, 1986 (U.S. EPA, 1986a) and the 1999 interim final guidelines (U.S. EPA, 1999a; see U.S. EPA 2001b). They provide EPA staff with guidance for developing and using risk assessments. They also provide basic information to the public about the Agency's risk assessment methods.


Circular A-4 to the Heads of Executive Agencies and Establishments, September 17, 2003 | OMBa-4.pdf, 550KB

Office of Management and Budget (OMB)
 

Abstract: This Circular is designed to assist analysts in the regulatory agencies by defining good regulatory analysis - called either "regulatory analysis" or "analysis" for brevity - and standardizing the way benefits and costs of Federal regulatory actions are measured and reported. Executive Order 12866 requires agencies to conduct a regulatory analysis for economically significant regulatory actions as defined by Section 3(f)(1). This requirement applies to rulemakings that rescind or modify existing rules as well as to rulemakings that establish new requirements.

Estimating The Public Health Benefits Of Proposed Air Pollution Regulations | NRC2002.pdf, 1.92MB

National Research Council of the National Academies

Abstract: The U.S. Environmental Protection Agency (EPA)has estimated that thousands of premature deaths and numerous cases of illness, such as chronic bronchitis and asthma attacks, could be prevented by reducing exposure to air pollution. These estimates come from regulatory health benefits analyses, which attempt to quantify changes in the expected cases of mortality and illness that are likely to result from proposed air pollution regulations. The estimates are often controversial, and the methods used to prepare them have been questioned.

In 2000, Congress recognized concerns about the methods used by EPA and emphasized the need for "the most scientifically defensible methodology in estimating health benefits." It directed EPA to ask the National Academy of Sciences "to conduct a study of this issue and recommend to the agency a common methodology to be followed in all future analyses."


Proposed Risk Assessment Bulletin | OMB_bulletin_010906.pdf, 157KB

Office of Management and Budget (OMB)

Abstract: As part of an ongoing effort to improve the quality, objectivity, utility, and integrity of information disseminated by the federal government to the public, the Office of Management and Budget (OMB), in consultation with the Office of Science and Technology Policy (OSTP), proposes to issue new technical guidance on risk assessments produced by the federal government.


Procedures Guide for Structured Expert Judgment | ROCG99.pdf, 414KB

Roger M. Cooke and L.J.H Goossens
 
Abstract: This document is a guide for using structured expert judgment to quantify uncertainty in quantitative models. The methods applied here have been developed by a host or researchers over the last 30 years. During the years 1990 - 1999, the European Commission and the United States Nuclear Regulatory Commission undertook a joint uncertainty study of accident consequence codes for nuclear power plants using structured expert judgment. The purpose was not only to perform an uncertainty analysis of the US accident consequence code MACCS and the European accident consequence code COSYMA. The wider purpose was to form a baseline for the state of the art in using structured expert judgment for quantifying uncertainty. The reports emerging from this work are intended to be useful outside the community of nuclear accident consequence modeling. Indeed the quantification of uncertainty in the modeling of dispersion, deposition, foodchain transport, and cancer induction, may be used in many fields of environmental modeling and health protection. In the same spirit the methods for using structured expert judgment to quantify uncertainty are applicable far beyond the accident consequence modeling community.

Expert Judgment and Air Quality
 
Uncertainty in Mortality Response to Airborne Fine Particulate Matter: Elicitation of European Air Pollution Experts
Tuomisto_Wilson_evans_Tainio.pdf, 283KB

Jouni T. Tuomisto, Andrew Wilson, John S. Evans, and
Marko Tainio, 

Abstract: The authors have performed a structured expert judgment study of the population mortality effects of fine particulate matter (PM2.5) air pollution. The opinions of six European air pollution experts were elicited. The ability of each expert to probabilistically characterize uncertainty was evaluated using 12 calibration questions -- relevant variables whose true values were unknown at the time of elicitation, but available at the time of analysis. The elicited opinions exhibited both uncertainty and disagreement. It emerged that there were significant differences in expert performance. Two combinations of the experts' judgments were computed and evaluated -- one in which each expert's views received equal weight; the other in which the expert's judgments were weighted by their performance on the calibration variables. When the performance of these combinations was evaluated the equal-weight decision-maker exhibited acceptable performance, but was nonetheless inferior to the performance-based decision-maker.

In general, the experts agreed with published studies for the best estimate of all-cause mortality from PM2.5; however, as would be expected, they gave confidence intervals that were several times broader than the statistical confidence intervals taken directly from the most frequently cited published studies. The experts were rather comfortable with applying epidemiological results from one geographic region to another. However, there was more uncertainty and disagreement about issues of timing of the effect and about the relative toxicity of different constituents of PM 2.5. Even so, the experts were in fairly good agreement that an appreciable fraction of the long-term health effects occurs within a few months from the exposure and that combustion-derived particles are more toxic than PM 2.5 on average, while secondary sulphates, nitrates and/or crustal materials may be less toxic. These assessments bring very valuable and relevant information to air pollution risk assessment.


Expanded Expert Judgment Assessment of the Concentration-Response Relationship Between PM2.5 Exposure And Mortality
Walker_pm_ee_report[1].pdf, 1,342KB

Industrial Economics, Incorporated
Draft Version: April, 2004 | PM_expert_elicitation_USEPA.pdf, 902KB

Abstract: In response to the NRC recommendations, EPA is exploring how it might incorporate expert judgment in policy analysis. As a first step in this direction, IEC worked with EPA and OMB scientists to design a pilot expert elicitation to characterize the uncertainty in the ambient PM2.5/mortality relationship. This pilot was designed to provide EPA with an opportunity to improve its understanding of the design and application of expert elicitation methods to economic benefits analysis. For instance, the pilot was designed to provide feedback on the efficacy of the protocol developed and the analytic challenges, as well as to provide insight regarding potential implications of the results on the degree of uncertainty surrounding the C-R function for PM2.5 mortality. The scope of the quantitative questions was limited in that we focused the elicitation on the C-R function of PM mass; this initial elicitation was not intended to thoroughly characterize the uncertainty surrounding individual elements of the PM2.5/mortality relationship, such as the relative toxicity of specific PM components (e.g., diesel particulates).

Peer Review of Expert Elicitation | Peer_Review_USEPA.pdf, 209KB
Carol Mansfield, RTI International

Abstract: The purpose of this memo is to synthesize responses given by reviewers selected to review and comment on the following document: An Expert Judgement Assessment of the Concentration-Response Relationship Between PM2.5 Exposure And Mortality, Prepared by Industrial Economics, Incorporated under subcontract to ABT Associates, Inc., for U.S. Environmental Protection Agency, Research Triangle Park, NC, April 23, 2004.

Expert Judgment Research Manuscripts

 
Are Decisionmakers at Home on the Range? Communicating Uncertainties in Cost-Benefit Analyses| (Krupnick_CommunicatingUncertainties.pdf, 276KB

Richard D. Morgenstern, Peter Nelson and Alan Krupnick

Abstract: Better and more complete information does not necessarily lead to better policies. Complex information can confound rather than enlighten, or can lead to decision paralysis. Thus, any improvements in capturing uncertainty analytically needs to be matched by improvements in its communication - not only to those who make regulatory decisions on the basis of such information, but also to stakeholders, judges, the press and the general public. Accordingly, this paper presents the results of seven in-depth interviews conducted with former Presidential appointees at the EPA concerning the use of uncertainty analysis. Specifically, results of a case study involving future controls of NOx emissions from power plants was presented to the interviewees, using different visual approaches for displaying the uncertainties. They were asked to express preferences for different regulatory outcomes based on the alternative approaches used to present the information. Tables and probability density functions (PDFs) may be best suited for communicating to high-level decisionmakers, although PDFs tend to move respondents to tight spread options. Respondents also emphasized the importance of presenting technical analysis in context.

 

Use of Expert Opinion Elicitation to Quantify the Internal Erosion Process in Dams | BrownAspinall_Useofexpertelicitation.pdf, 337KB

A. J. Brown and W. P. Aspinall

Abstract:Expert Opinion Elicitation is a generic term for a number of similar techniques that have been developed to provide quantitative estimates of parameters which cannot readily be quantified through direct measurement or other sampling techniques. The initial motivation for their development was the 1986 Challenger Shuttle disaster in the space industry, and subsequent applications have spread into many other areas: the techniques have been widely used in the nuclear industry, for instance. One particular procedure consists of obtaining responses to a set of quantitative questions from a number of experts, including the range of uncertainty in each response, and then combining these through a weighting procedure to obtain a pooled best estimate of the parameters of interest.
 
 
Volcanic Activity: Frontiers and Challenges in Forecasting, Prediction and Risk Assessment | SparksAspinall_VolcanicActivity.pdf, 5.6MB)

R. S. J. Sparksand W. P. Aspinall
Abstract: When a volcano shows signs of unrest scientists are asked to forecast whether an eruption will happen, when it will happen and what kind of eruption it will be. They are also expected to provide information on hazardous volcanic phenomena and their effects, and how long the eruption will last. Eruptions are complex phenomena, however, involving magma ascent to the Earth's surface and interactions with surrounding crust and surface environments during eruption. Magma may change its properties profoundly during ascent and eruption, and many of the governing processes of heat and mass transfer can be highly non-linear. There are both epistemic and aleatory uncertainties involved, which can be large, making precise prediction of a certain event in time and space a formidable or impossible objective; that is, volcanoes can be intrinsically unpredictable. As with other natural phenomena, forecasting is a more achievable goal and needs to be expressed in probabilistic terms that take account of the uncertainties. Ensemble modeling in which uncertainties are sampled with Monte Carlo techniques is likely to become the basis for such forecasting. Despite the limitations, there is significant progress in anticipating volcanic activity and, in favorable circumstances, in making predictions. Data from enhanced monitoring techniques are being combined with advanced numerical models of volcanic flows and their interactions with the environment. Statistical analysis of volcanological data and improvements in methods to treat subjective information are also beginning to provide viable, complementary approaches to basic numerical modeling.
 
 
 
Alan Krupnick, Richard D. Morgenstern, Michael Batz, Peter Nelson, Dallas Burtraw, Jhih-Shyang Shih, and Michael McWilliams

Special Issue on Expert Judgment

Guest Editorial: Special Issue on Expert Judgment
Editors_Intro4EJ_special_issue, 31KB

Roger M. Cooke

Introduction: We are indeed gratified to be able to present eight impressive papers on the subject of expert judgment. It is well-known that the calls for taking uncertainty more seriously in quantitative decision support, at all levels, become ever more persistent. Quantifying uncertainty means, proximally and for the most part, using structured expert judgment. The qualifier "structured" means that expert judgment is treated as scientific data, albeit scientific data of a new type. Elicitation and representation of uncertainties, processing the expert judgment data, and utilization of results must be subjected to transparent methodological rules grounded in the scientific method itself. The first article announces the availability of the TU Delft expert judgment database to all researchers in this field. Three other articles in this volume illustrate the use of this data. The articles will be mentioned and briefly summarized in the order of their appearance.

TU Delft Expert Judgment Data Base | Cooke_Goossens.pdf, 487KB

Roger M. Cooke

Abstract: We review the applications of structured expert judgment uncertainty quantification using the "classical model" developed at the Delft University of Technology over the last 17 years (Cooke, 1991). These involve 45 expert panels, performed under contract with problem owners who reviewed and approved the results. With a few exceptions, all these applications involved the use of seed variables; that is, variables from the experts' area of expertise for which the true values are available post hoc. Seed variables are used to (1) measure expert performance, (2) enable performance based weighted combination of experts' distributions, and (3) evaluate and hopefully validate the resulting combination or "decision maker". This article reviews the classical model for structured expert judgment and the performance measures, reviews applications, comparing performance based decision makers with "equal weight" decision makers, and collects some lessons learned.

 
Expert Judgement Combination using Moment Methods Wisse_Bedford_Quigley.pdf, 213KB

Bram Wisse, Tim Bedford and John Quigley

Abstract:Moment methods have been employed in decision analysis, partly to avoid the computational burden that decision models involving continuous probability distributions can suffer from. In the Bayes linear (BL) methodology prior judgements about uncertain quantities are specified using expectation (rather than probability) as the fundamental notion. BL provides a strong foundation for moment methods, rooted in work of De Finetti and Goldstein. The main objective of this paper is to discuss in what way expert assessments of moments can be combined, in a non-Bayesian way, to construct a prior assessment. We show that the linear pool can be justified in an analogous but technically different way to linear pools for probability assessments, and that this linear pool has a very convenient property: a linear pool of experts' assessments of moments is coherent if each of the experts has given coherent assessments. To determine the weights of the linear pool we give a method of performance based weighting analogous to Cooke's classical model and explore its properties. Finally we compare its performance with the classical model on data gathered in applications of the classical model.
 
 
Sandra Hoffmann, Paul Fischbeck, Alan Krupnick, and Michael McWilliams
 
Abstract: This article looks at a new approach to expert elicitation that combines basic elements of conventional expert elicitation protocols with formal survey methods and larger, heterogeneous expert panels. This approach is appropriate where the hazard-estimation task requires a wide range of expertise and professional experience. The ability to judge when to rely on alternative data sources often is critical for successful risk management. We show how a large, heterogeneous sample can support internal validation of not only the experts' assessments but also prior information that is based on limited historical data. We illustrate the use of this new approach to expert elicitation by addressing a fundamental problem in U.S. food safety management, obtaining comparable food system-wide estimates of the foodborne illness by food-pathogen pair and by food. The only comprehensive basis for food-level hazard analysis throughout the U.S. food supply currently available is outbreak data (i.e., when two or more people become ill from the same food source), but there is good reason to question the portrayal that outbreak data alone gives of food risk. In this paper, we compare results of food and food-pathogen incidence estimates based on expert judgment and based on outbreak data, and we demonstrate a suite of uncertainty measures that allow for a fuller understanding of the results.
 
 
Oswaldo Morales, Dorota Kurowicka and A. Roelen
 
Abstract: Causes of uncertainties may be interrelated and may introduce dependencies. Ignoring these dependencies may lead to large errors. A number of graphical models in probability theory such as dependence trees, vines and (continuous) Bayesian Belief Nets ([1], [2], [3], [4], [5], [6]) have been developed to capture dependencies between random variables. The input for these models are various marginal distributions and dependence information, usually in the form of conditional rank correlations. Often expert elicitation is required. This paper focuses on dependence representation, and dependence elicitation. The techniques presented are illustrated with an application from aviation safety.
 
A Study of Expert Overconfidence | Lin_Bier.pdf, 246KB

Shi-Woei Lin and Vicki M. Bier

Abstract: Overconfidence is one of the most common (and potentially severe) problems in expert judgment. To assess the extent of expert overconfidence, we analyzed a large data set on expert opinion compiled by Cooke and colleagues at the Technical University of Delft and elsewhere. This data set contains roughly five thousand 90% confidence intervals of uncertain quantities for which the true values are now known. Our analysis assesses the overall extent of overconfidence in the data set. Significant differences in the extent of overconfidence were found among studies, among experts, and among questions within a study. Moreover, replications (multiple realizations for the same question) allowed a preliminary assessment of whether the question effect is due largely to question difficulty, or merely to random noise in the realizations of the uncertain quantities. The results of this analysis suggest that much of the apparent question effect may be due to noise rather than systematic differences in the difficulty of achieving good calibration for different questions. The results support the differential weighting of experts, since there are significant differences in expert calibration within studies.

A Paired Comparison Experiment for Gathering Expert Judgment for an Aircraft Wiring Risk Assessment | Mazzuchi_Linzey_Brunin.pdf, 129KB

Thomas A. Mazzuchi, William G. Linzey, and Armin Brunin

Abstract: Wire failure in aircraft can be attributed to several factors and the assessment of the risk of wire failure is becoming an increasingly important task. This paper will discuss the results of an actual experiment to use the paired-comparison technique for expert judgment to develop a relationship for the probability of wire failure as a function of influencing factors in an aircraft environment. The reasons for using this technique are two-fold. First, the failure probability depends on many variables including wire gauge, vibration, environmental condition etc. In addition, the wire failure data is sparse and fitting this data to a complex failure function is a nontrivial task that may involve a host of assumptions that may not be provable. We describe a method for using actual failure data and the results from a paired comparison to populate the model parameters. In the approach, paired comparison data from select environments is used to obtain failure rate estimates for the candidate environments. Next, a functional relationship for wire failure as a function of the environments is constructed using a proportional hazards model. A regression model is fit from the failure rate estimates to the environmental variables and is used as an estimate of the failure response surface. This technique is being investigated as a means to generate failure rates for an Electrical Wiring Interconnection System Risk Assessment software tool currently being developed for the FAA Tech Center.

 
Uncertainty in Mortality Response to Airborne Fine Particulate Matter: Elicitation of European Air Pollution Experts Tuomisto_Wilson_evans_Tainio.pdf, 283KB

Jouni T. Tuomisto, Andrew Wilson, John S. Evans, and 
Marko Tainio
Abstract: The authors have performed a structured expert judgment study of the population mortality effects of fine particulate matter (PM2.5) air pollution. The opinions of six European air pollution experts were elicited. The ability of each expert to probabilistically characterize uncertainty was evaluated using 12 calibration questions -- relevant variables whose true values were unknown at the time of elicitation, but available at the time of analysis. The elicited opinions exhibited both uncertainty and disagreement. It emerged that there were significant differences in expert performance. Two combinations of the experts' judgments were computed and evaluated -- one in which each expert's views received equal weight; the other in which the expert's judgments were weighted by their performance on the calibration variables. When the performance of these combinations was evaluated the equal-weight decision-maker exhibited acceptable performance, but was nonetheless inferior to the performance-based decision-maker.

In general, the experts agreed with published studies for the best estimate of all-cause mortality from PM2.5; however, as would be expected, they gave confidence intervals that were several times broader than the statistical confidence intervals taken directly from the most frequently cited published studies. The experts were rather comfortable with applying epidemiological results from one geographic region to another. However, there was more uncertainty and disagreement about issues of timing of the effect and about the relative toxicity of different constituents of PM2.5. Even so, the experts were in fairly good agreement that an appreciable fraction of the long-term health effects occurs within a few months from the exposure and that combustion-derived particles are more toxic than PM2.5 on average, while secondary sulphates, nitrates and/or crustal materials may be less toxic. These assessments bring very valuable and relevant information to air pollution risk assessment.

 
On the Performance of Social Network and Likelihood Based Expert Weighting Schemes | Cooke_ElSaadany_Huang.pdf, 286KB

Roger M. Cooke, Susie El Saadany and Xinzheng Huang
Abstract: Using expert judgment data from the TU Delft's expert judgment data base, we compare the performance of different weighting schemes, namely equal weighting, performance based weighting from the classical model (Cooke, 1991), social network (SN) weighting and likelihood weighting. The picture that emerges with regard to social network weights is rather mixed. SN theory does not provide an alternative to performance based combination of expert judgments, since the statistical accuracy of the SN decision maker is sometimes unacceptably low. On the other hand, it does outperform equal weighting in the majority of cases. The results here, though not overwhelmingly positive, do nonetheless motivate further research into social interaction methods for nominating and weighting experts. Indeed, a full expert judgment study with performance measurement requires an investment in time and effort, with a view to securing external validation. If high confidence in a comparable level of validation can be obtained by less intensive methods, this would be very welcome, and would facilitate the application of structured expert judgment in situations where the resources for a full study are not available. Likelihood weights are just as resource intensive as performance based weights, and the evidence presented here suggests that they are inferior to performance based weights with regard to those scoring variables which are optimized in performance weights (calibration and information). Perhaps surprisingly, they are also inferior with regard to likelihood. Their use is further discouraged by the fact that they constitute a strongly improper scoring rule.

Comments from Professor Tony O'Hagan OHagen_cmnts_EJspecial_issue, 14KB

  • "A Study of Expert Overconfidence" by Shi Woei-Lin and Vicki M. Bier
  • "A Paired Comparison Experiment for Gathering Expert Judgment for an Aircraft Wiring Risk Assessment" by Mazzuchi, Linzey and Brunin
  • "Eliciting Conditional and Unconditional..." by Morales, Kurowicka and Roelen
  • "Expert Judgement Combination using Moment Methods" by Bram Wisse, Tim Bedford and John Quigley

Comments from Professor Robert T. Clemen
Clemen_Cmnts_EJspecial_issue, 98KB

Several of the papers in this special issue are in one way or another linked to Cooke's "classical" method for combining expert probability distributions. This comment focuses on characteristics of that method. In particular, I consider two questions: Does the weighting scheme give the experts a positive incentive to report their beliefs honestly for each variable? How does Cooke?s method perform when evaluated out-of-sample?

Comments from Professor Simon French French_Cmmnts_EJspecial_issue.pdf, 38KB

  • "Expert Judgement Combination using Moment Methods" by Bram Wisse, Tim Bedford and John Quigley
  • "TU Delft Expert Judgment Data Base" by Roger M. Cooke and
    Louis L.H.J. Goosen
  • "A Study of Expert Overconfidence" by Shi Woei-Lin and Vicki M. Bier
  • "On the Performance of Social Network and Likelihood Based Expert Weighting Scheme" by Roger M. Cooke, Sussie ElSaadany and Xinzheng Huang

Response by Bedford et. al. to Tony O'Hagan's and Simon French' comments | Response2discussants_TB.pdf, 16KB

Response by Morales et. al. to Tony O'Hagen comments
Morales_Responce2cmnts.pdf, 14KB

Response by Lin and Bier to Tony O'Hagan's and Simon French' comments
Bier_Response2cmnts.pdf, 16KB

Response by Mazzuchi et. al. to Tony O'Hagan's comments
Mazzuchi_Response2cmnts.pdf, 22KB

Response by Roger Cooke to Simon French' and Bob Clemen's comments
Cooke_resp2cmnts.pdf, 66KB

 
 
RFF Home | Weathervane | Extending the Cure | RFF Press Terms of Use | Privacy Policy | Copyright Notice
1616 P Street NW, Washington, DC 20036 · (202) 328-5000 Site Map | Feedback | Contact