News
Eliciting expert judgements for environmental risk analysis
Mark Burgman,
PhD, FAA, Adrienne Clarke Chair of Botany, Director, Australian Centre of
Excellence for Risk Analysis, School of Botany, University of Melbourne,
Parkville, 3010.
Expert judgements are a routine and necessary part of risk analysis. Full, detailed model-building and data acquisition are usually too slow, cumbersome and costly for routine application. We need experts to fill gaps in knowledge and data, to provide explanations of how ecological systems work, and to estimate probabilities of various outcomes. However, numerous studies have documented that experts may be susceptible to context, framing and cognitive and motivational biases. Research over the last 40 years in cognitive psychology has shown that perception of risk is filtered by memory, framing, availability, anchoring, ‘affect’ (intuitive, fast, mostly automatic response to a hazard, linked to experience and emotion) and other subjective factors. For example, overconfidence describes a situation in which the expert’s own confidence in their judgment or estimate does not correspond to the accuracy of that estimate. Research on the performance of nuclear risk engineers, geophysicists, ecologists and other professional groups has shown that most experts are overconfident in their ability to estimate quantities. In recent years, research into elicitation has focused on strategies to account for and overcome some of these cognitive biases.
Models depend on both parameter estimates, and ideas about cause and effect. We summarise our understanding of biological systems with simplified abstractions that capture the essential elements of the system. These conceptual models are embedded in our descriptions of cause-and-effect and lead us to form hypotheses, perform experiments and collect data. There are several direct and indirect techniques for eliciting uncertainties for quantities, frequencies and probabilities, and for eliciting conceptual models.
There is, however, considerable variation in procedures and formats. Direct methods involve simply asking an expert for the desired probabilities or frequencies. Indirect methods infer a judgement about a quantity from a person’s behaviour, using contexts such as gambles or pairwise comparisons. Language-based methods to detect and adjust for expert bias have various strengths and weaknesses. Andrew Speirs-Bridge and Fiona Fidler have been working with us to develop better ways of asking questions so that the information that experts give us is much more reliable.
Although experts may disagree on the conceptualization of a system or problem, there are surprisingly few tools for systematically resolving such differences. Terry Walshe is working with the research group to develop the application of formal methods for representing an expert’s beliefs about model structures, reconciling differences of opinion and exploring the importance of these differences for decisions on how we should manage a system.
There is broad scientific consensus on the chief stages of formal elicitation of quantities and scientific ideas of cause and effect. The key theme is that beyond a basic framework, the best approach is to structure the elicitation to suit the decision at hand. General guidelines may avoid the most serious and predictable psychological and motivational pitfalls. The ultimate objective of this research at the School of Botany is to create techniques for involving scientific experts in the creation of decision-support systems that have practical utility for environmental risk analysis, and that make the best use possible of the important knowledge that experts have.