Expert Elicitation for Uncertainty Quantification

Expert elicitation is a structured process to elicit subjective judgements from experts. It is widely used in quantitative risk analysis to quantify uncertainties in cases where there are no or too few direct empirical data available to infer on uncertainty. Usually the subjective judgement is represented as a ‘subjective’ probability density function (PDF) reflecting the expert’s degree of belief.
Expert elicitation in the context of uncertainty quantification aims at a credible and traceable account of specifying probabilistic information regarding uncertainty, in a structured and documented way. Typically it is applied in situations where there is scarce or insufficient empirical material for a direct quantification of uncertainty, and where it is relevant to obtain inscrutable and defensible results (Hora, 1992).
Several elicitation protocols have been developed amongst which the much-used Stanford/SRI Protocol is the first (Spetzler and von Holstein, 1975; see also Morgan and Henrion, 1990; chapter 6 and 7). Expert elicitation typically involves the following steps:

(1) Identify and select experts;
(2) Explain to the expert the nature of the problem and the elicitation procedure. Create awareness of biases in subjective judgements and explore these.
(3) Clearly define the quantity to be assessed and chose a scale and unit familiar to the expert.
(4) Discuss the state of knowledge on the quantity at hand (strengths and weaknesses in available data, knowledge gaps, qualitative uncertainties).
(5) Elicit extremes of the distribution.
(6) Assess these extremes: could the range be broader than stated?
(7) Further elicit and specify the distribution (shape and percentiles or characterising parameters).
(8) Verify with the expert that the distribution that you constructed from the expert’s responses correctly represents the expert’s beliefs.
(9) Decide whether or not to aggregate the distributions elicited from different experts (this only makes sense if the experts had the same mental models of the quantity for which a distribution was elicited).

Resources required
Typically performing a formal expert elicitation is a time and resource intensive activity. The whole process of setting up a study, selecting experts, preparing elicitation questions, performing expert training, expert meetings, interviews, analyses, writing rationales, documentation etc. can easily stretch over months or years.
The choice of whether to perform a formal or a more informal elicitation (NCRP, 1996) depends on the price one is willing to pay for more inscrutable and defensible results, and will be influenced by the relevance and controversies regarding the problem area.
One needs to have good interviewing skills and a reasonable understanding of the field under consideration. A good understanding of biases in subjective judgements by experts is required to avoid these biases to the maximum extent possible. Skills are needed to draft a good questionnaire or template for the elicitation. Training in elicitation techniques may be needed.

Figure: Seven step procedure for a formal expert elicitation (Knol et al., 2010).

Strengths and limitations
+ It has the potential to make use of all available knowledge including knowledge that cannot be easily
formalised otherwise.
+ It can easily include views of sceptics and reveals the level of expert disagreement on certain estimates.
− The fraction of experts holding a given view is not proportional to the probability of that view being
correct.
− One may safely average estimates of model parameters, but if the expert's models were incommensurate,
one may not average models (Keith, 1996).
− If differences in expert opinion are irresolvable, weighing and combining the individual estimates of
distributions is only valid if weighted with competence of the experts regarding making the estimate.
There is no good way to measure competence.
− The results are sensitive to the selection of the experts whose estimates are gathered.

Further reading:
A.B. Knol, P. Slottje, J.P. van der Sluijs, E. Lebret (2010). The use of expert elicitation in environmental health impact assessment: a seven step procedure, Environmental Health 9:19.

P. Slottje, J.P. van der Sluijs and A.B. Knol (2008). Expert Elicitation – Methodological suggestions for its use in environmental health impact assessments. RIVM Letter Report 630004001/2008, National Institute for Public Health and the Environment, Bilthoven, 56 pp.

References
Ayyub BM (2001) Elicitation of Expert Opinions for Uncertainty and Risks, CRC Press, Florida.

Hora SC (1992) Acquisition of Expert Judgement: Examples from Risk Assessment. Journal of Energy Engineering, 118, 136-148.

Meyer MA and Booker JM (1991) Eliciting and Analyzing Expert Judgement: A practical Guide, Academic Press, London

Morgan MG and Henrion M (1990) Uncertainty, A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis, Cambridge University Press.

Spetzler CS and von Holstein S (1975) Probability Encoding in Decision Analysis. Management Science, 22(3).

http://legacy.ncsu.edu/classes/ce456001/www/Background1.html