FEEM working papers "Note di lavoro"
Date: 14/9/2016

Quantifying the Effects of Expert Selection and Elicitation Design on Experts’ Confidence in their Judgments about Future Energy Technologies

Gregory F. Nemet (University of Wisconsin-Madison); Laura Diaz Anadon (John F. Kennedy School of Government - Harvard University); Elena Verdolini (Fondazione Eni Enrico Mattei, Centro Euro-Mediterraneo per i Cambiamenti Climatici)
Type: Journal
JEL n.: O13, O14, Q4
Published in: Risk Analysis, doi:10.1111/risa.12604
Keywords: Expert Elicitations, Uncertainty, Energy Technologies, Heuristic Biases, Survey Design


Expert elicitations are frequently used to characterize future technology outcomes. However their usefulness is limited, in part because: estimates across studies are not easily comparable; choices in survey design and expert selection may bias results; and over-confidence is a persistent problem. We provide quantitative evidence of how these choices affect experts’ estimates of the costs of future energy technologies. We harmonize data from 19 elicitations, involving 215 experts, on the 2030 costs of 5 energy technologies: nuclear, biofuels, bioelectricity, solar, and carbon capture. We control for expert characteristics, survey design, and public R&D investment levels on which the elicited values are conditional.  We find that, on average, when experts respond to elicitations in person, they ascribe lower confidence (larger uncertainty) to their estimates than when responding via mail or online.  In-person interviews also produce more optimistic assessments of best-case (10th percentile) outcomes.  The impacts of expert affiliation—government, private sector, or academic—and geography—US or EU—are also significant; academics and US experts have lower confidence than other types of experts.  Higher R&D investment levels have no effect on the confidence of experts’ judgments.  R&D reduces both the median and breakthrough (10th percentile) cost estimates, although the size of the effect varies across technologies. These results indicate the source, direction, and size of bias in energy technology elicitations.  They also point to the technology specificity of some of the effects.  These biases should be seriously considered, both in interpreting the results of existing elicitations and in designing new ones.


Suggested citation: Nemet, G., Anadon L. D. and Verdolini E. (2016), “Quantifying the effects of expert selection and elicitation design on experts’ confidence in their judgments about future energy technologies”, Risk Analysis, doi:10.1111/risa.12604.

Download file
Download file
Quantifying the Effects of Expert Selection and Elicitation Design on Experts’ Confidence in their Judgments about Future Energy Technologies

FEEM Newsletter & Update

Subscribe to stay connected.

Your data will be processed by FEEM - Eni Enrico Mattei Foundation - Data Controller - to receive the Foundation's Newsletter & Update via e-mail. To send the requested information, the Data Controller uses e-mail sending service providers established in the U.S.A .: for this, your consent to the transfer of your email address to the U.S.A. is required, without which it will not be possible to comply with your request. We invite you to consult the complete information to obtain detailed information to protect your rights.