FEEM working papers "Note di lavoro"
Date: 14/9/2016

Quantifying the Effects of Expert Selection and Elicitation Design on Experts’ Confidence in their Judgments about Future Energy Technologies

Gregory F. Nemet (University of Wisconsin-Madison); Laura Diaz Anadon (John F. Kennedy School of Government - Harvard University); Elena Verdolini (Fondazione Eni Enrico Mattei, Centro Euro-Mediterraneo per i Cambiamenti Climatici)
Type: Journal
JEL n.: O13, O14, Q4
Published in: Risk Analysis, doi:10.1111/risa.12604
Keywords: Expert Elicitations, Uncertainty, Energy Technologies, Heuristic Biases, Survey Design


Expert elicitations are frequently used to characterize future technology outcomes. However their usefulness is limited, in part because: estimates across studies are not easily comparable; choices in survey design and expert selection may bias results; and over-confidence is a persistent problem. We provide quantitative evidence of how these choices affect experts’ estimates of the costs of future energy technologies. We harmonize data from 19 elicitations, involving 215 experts, on the 2030 costs of 5 energy technologies: nuclear, biofuels, bioelectricity, solar, and carbon capture. We control for expert characteristics, survey design, and public R&D investment levels on which the elicited values are conditional.  We find that, on average, when experts respond to elicitations in person, they ascribe lower confidence (larger uncertainty) to their estimates than when responding via mail or online.  In-person interviews also produce more optimistic assessments of best-case (10th percentile) outcomes.  The impacts of expert affiliation—government, private sector, or academic—and geography—US or EU—are also significant; academics and US experts have lower confidence than other types of experts.  Higher R&D investment levels have no effect on the confidence of experts’ judgments.  R&D reduces both the median and breakthrough (10th percentile) cost estimates, although the size of the effect varies across technologies. These results indicate the source, direction, and size of bias in energy technology elicitations.  They also point to the technology specificity of some of the effects.  These biases should be seriously considered, both in interpreting the results of existing elicitations and in designing new ones.


Suggested citation: Nemet, G., Anadon L. D. and Verdolini E. (2016), “Quantifying the effects of expert selection and elicitation design on experts’ confidence in their judgments about future energy technologies”, Risk Analysis, doi:10.1111/risa.12604.

Download file
Download PDF file
Quantifying the Effects of Expert Selection and Elicitation Design on Experts’ Confidence in their Judgments about Future Energy Technologies

FEEM Newsletter & Update

Subscribe to stay connected.

Your personal data will be processed by Fondazione Eni Enrico Mattei. – data Controller – with the aim of emailing the FEEM newsletter & Update. The use of Your email address is necessary for the implementation of the newsletter service. You are invited to read the Privacy Policy in order to obtain additional information about the protection of Your rights.

This Website uses technical cookies and cookie analytics, as well as “third party” profiling cookies.
If you close this banner or you decide to continue navigating on this Website, you express consent to the use of cookies. If you need additional information or you wish to express selective choices on the use of cookies, please refer to the   Cookie PolicyI agree