NUSAP analysis of the TIMER energy model

Introduction

This project implemented a novel approach to uncertainty assessment, known as the NUSAP method (Numeral Unit Spread Assessment Pedigree) to assess qualitative and quantitative uncertainties in the TIMER energy model, part of RIVMs IMAGE Model. We used the IMAGE B1 scenario produced for the IPCC as case study.

Objective

Develop a framework for uncertainty assessment and management including both quantitative and qualitative dimensions, and test and demonstrate its usefulness in integrated assessment models.
• What are key uncertainties in TIMER?
• What is the role of model structure uncertainties in TIMER?
• Uncertainty in which input variables and parameters dominate uncertainty in model outcome?
• What is the strength of the sensitive parameters (pedigree)?

Method

The set of methods by which we implemented NUSAP in this project include:
• A comprehensive checklist for model quality assurance;
• A meta-level analysis of the results of the six SRES energy models, to explore model structure uncertainties;
• The Morris algorithm for global sensitivity analysis;
• A NUSAP expert elicitation workshop to systematically assess the pedigree of sensitive parameters
• A diagnostic diagram to prioritise uncertainties by the combination of criticality (based on Morris) and parameter strength (based on pedigree).

Table 1 Pedigree matrix for parameter strength. Note that the columns are independent.

 
NUSAP expert elicitation workshop, Loosdrecht, June 12 and 13 2000

Results

The most sensitive model components turned out to be:
• Population levels and economic activity as main drivers;
• Variables related to intra-sectoral structural change;
• Progress ratios for technological improvements;
• Variables related to resources of fossil fuels (size and cost supply curves);
• Variables related to autonomous and price-induced energy efficiency improvement;
• Variables related to initial costs and depletion of renewables;

We assessed pedigree and value loading of these model components in a NUSAP expert elicitation workshop (see photo) with 18 participants. The elicitation was done in parallel in three groups of 6 experts each.

For some parameters we found reasonable consistency in pedigree scores across the group results, indicating a common view of the underpinnings of these parameters. For other parameters we found considerable disagreement within and across groups. An example result is presented in fig. 1.

Average scores for all five pedigree criteria over all parameters range from low (validation 1.1) to medium (empirical: 1.8, method 1.8, theory 2.0 and proxy 2.4, all on a scale from 0 to 4). The slightly higher score for theoretical understanding compared to empirical basis combined with the consistently low scores for validation nicely reflect the inherent theory ladeness of scenario studies, in this case based on not fully crystallised theory.

Findings for sensitivity and pedigree were combined in a diagnostic diagram (fig. 2). The Y-axis plots contribution to change in projected CO2 emissions found with the sensitivity anaysis. The X-axis displays normalized average pedigree scores for each variable. The error bars about these values (one standard deviation) reflect expert disagreement on pedigree scores. The scale goes from 1 at the origin to zero on the right, bringing the more `dangerous' variables in the top right quadrant of the plot.

Conclusions

• Our model quality assurance checklist proves a quick scan to flag major areas of concern and associated pitfalls in the complex mass uncertainties.
• The meta-level intercomparison of TIMER with the other SRES models gave us some insight in the potential roles of model structure uncertainties.
• Global sensitivity analysis supplemented with expert elicitation constitutes an efficient selection mechanism to further focus the diagnosis of key uncertainties.
• Our pedigree elicitation procedure yields a differentiated insight into parameter strength.
• The diagnostic diagram puts spread and strength together to provide guidance in prioritisation of key uncertainties.

Overall, the project demonstrated that the NUSAP method can be applied to complex models in a meaningful way. The method provides a useful means to focus research efforts on the potentially most problematic parameters while it at the same time pinpoints specific weaknesses in these parameters.

References

Full report of the case:
Jeroen P. van der Sluijs, Jose Potting, James Risbey, Detlef van Vuuren, Bert de Vries, Arthur Beusen, Peter Heuberger, Serafin Corral Quintana, Silvio Funtowicz, Penny Kloprogge, David Nuijten, Arthur Petersen, Jerry Ravetz. Uncertainty assessment of the IMAGE/TIMER B1 CO2 emissions scenario, using the NUSAP method. Dutch National Research Program on Climate Change, Bilthoven, 2002, 225 pp.

Short paper summarizing the case:
J. van der Sluijs, James Risbey, Serafin Corral Quintana, Jerry Ravetz, Uncertainty management in complex models: the NUSAP method, in: Integrated Assessment and Decision Support. Ed. A.Jakeman, A. Rizzoli. Proceedings of IEMSS 2002 Conference, 24-27 June 2002, Lugano, Switzerland. p. 13-18.

One pager
Poster version of the case

Published journal article reporting the case
Jeroen van der Sluijs, Matthieu Craye, Silvio Funtowicz, Penny Kloprogge, Jerry Ravetz, and James Risbey (2005) Combining Quantitative and Qualitative Measures of Uncertainty in Model based Environmental Assessment: the NUSAP System, Risk Analysis, 25 (2). p. 481-492.