Eliciting expert judgements for environmental risk analysis  

Mark Burgman, PhD, FAA, Adrienne Clarke Chair of Botany, Director, Australian Centre of Excellence for Risk Analysis, School of Botany, University of Melbourne, Parkville, 3010.  

Expert judgements are a routine and necessary part of risk analysis. Full, detailed model-building and data acquisition are usually too slow, cumbersome and costly for routine application. We need experts to fill gaps in knowledge and data, to provide explanations of how ecological systems work, and to estimate probabilities of various outcomes. However, numerous studies have documented that experts may be susceptible to context, framing and cognitive and motivational biases. Research over the last 40 years in cognitive psychology has shown that perception of risk is filtered by memory, framing, availability, anchoring, ‘affect’ (intuitive, fast, mostly automatic response to a hazard, linked to experience and emotion) and other subjective factors. For example, overconfidence describes a situation in which the expert’s own confidence in their judgment or estimate does not correspond to the accuracy of that estimate. Research on the performance of nuclear risk engineers, geophysicists, ecologists and other professional groups has shown that most experts are overconfident in their ability to estimate quantities. In recent years, research into elicitation has focused on strategies to account for and overcome some of these cognitive biases.

Models depend on both parameter estimates, and ideas about cause and effect. We summarise our understanding of biological systems with simplified abstractions that capture the essential elements of the system. These conceptual models are embedded in our descriptions of cause-and-effect and lead us to form hypotheses, perform experiments and collect data. There are several direct and indirect techniques for eliciting uncertainties for quantities, frequencies and probabilities, and for eliciting conceptual models.

There is, however, considerable variation in procedures and formats. Direct methods involve simply asking an expert for the desired probabilities or frequencies. Indirect methods infer a judgement about a quantity from a person’s behaviour, using contexts such as gambles or pairwise comparisons. Language-based methods to detect and adjust for expert bias have various strengths and weaknesses. Andrew Speirs-Bridge and Fiona Fidler have been working with us to develop better ways of asking questions so that the information that experts give us is much more reliable. 

Although experts may disagree on the conceptualization of a system or problem, there are surprisingly few tools for systematically resolving such differences. Terry Walshe is working with the research group to develop the application of formal methods for representing an expert’s beliefs about model structures, reconciling differences of opinion and exploring the importance of these differences for decisions on how we should manage a system.

There is broad scientific consensus on the chief stages of formal elicitation of quantities and scientific ideas of cause and effect. The key theme is that beyond a basic framework, the best approach is to structure the elicitation to suit the decision at hand. General guidelines may avoid the most serious and predictable psychological and motivational pitfalls. The ultimate objective of this research at the School of Botany is to create techniques for involving scientific experts in the creation of decision-support systems that have practical utility for environmental risk analysis, and that make the best use possible of the important knowledge that experts have. 

Need for Taxonomy – classify species or face extinction

The social licence and need for taxonomists have never been greater, yet this most vital of life science struggles for survival.  Professor Gerry Cassis (University of New South Wales) and Professor Pauline Ladiges (University of Melbourne)

The Australia, Wednesday October 31, 2007

In early 2004, there was a quarantine emergency, where Pakistan rejected our wheat shipment due to an alleged presence of spores of the fungus Tilletia indica that causes a disease called Karnal Bunt. This species of fungus is unknown in Australia, but there are other related species that are 'spore-look-alikes'. A National Diagnostic Protocol was in place, and wheat stores at Australian ports were sampled.  Fungal spores (the look-alikes) were found in a percentage of these stores but none was Tilletia indica when compared with herbarium reference collections. Taxonomic expertise saved the day and a $4 billion export trade resumed!

That story is just one of countless issues emerging from a natural world whose blood pressure is off the chart in the face of human resource consumption and habitat change. The growing crisis is reflected in media headlines that focus on what are now perennial environmental and biological issues - such as climate change, biodiversity conservation, water and natural resource management - and issues of the day, such as pulp mill approvals and equine flu.

So it's a very safe prediction that the life sciences will dominate in the 21st century. The government and corporate sectors are increasingly focused on the environment. Our needs for sustainable solutions are pressing on the scientific community to provide the best scientific information to allow for informed and balanced decisions. Increasingly radical adjustments are needed in the way we do business, such as what we see presently in the nexus between energy policy and climate change.

Yet, as scientists, what do we know? Quite a lot, but not nearly enough, would be the appropriate answer. Nowhere is this reflected more than in the vital science of taxonomy. Taxonomy is the discovery, description, identification and classification of organisms. This year is the 250th anniversary of the taxonomic discipline and the 'Darwin' of our science, the Swedish biologist Carolus Linnaeus, who invented this most powerful predictive science. It's not some quaint pursuit of obsessives but a fundamental science that underpins so much of the life sciences. Just ask the wheat farmers and our frontline biosecurity officers.

Consider two facts: first, we probably know less than a fifth of the Earth's species - less than 2 million of the predicted 10 million species are documented; second, these species are fundamental to our survival through the provision of food, clean air and water, healthy soils, our medicines.

Ready criticisms exist about the need for fully documenting the world's biota: who cares about all these species, for example, and do we really have time to find, count and organise them when environmental remediation is often urgent? But this would be like running trying to run a massive global supply business without having undertaken a stocktake.

In effect, the social licence and need for taxonomists has never been greater. Yet for all the demands on them, many readers would be surprised to know that taxonomists are under serious threat of extinction in Australia (Higher Education Supplement reference ***]. Within a decade our taxonomic workforce may be halved, to about 70 taxonomists, because more than 75% of them are over 45 years of age. This is nowhere near adequate when considering that more than 600,000 species exist in Australia, with less than 30% described, and that large parts remain unexplored. This would leave each Australian taxonomist 8,500 species to consider, which is an impossible task within any individual career.

From 2000 to 2004, government and higher education expenditure on taxonomy has fallen from $34million to $29 million, whereas the overall expenditure in biology as a whole has increased by more than 12% in these sectors. This perverse trend in taxonomic investment is most recently stated in the 2006 State of Environment Report -: "One very important issue that continues to get worse is a national decline in capacity in biological taxonomy. The situation in this field has become critical".

Within the past year, the Commonwealth Government has made a positive step forward in the funding of two taxonomic initiatives; Australia's Living Atlas (National Collaborative Research Infrastructure Strategy - $7.5 million over 5 years) and the National Taxonomy Hub (Commonwealth Environmental Research Facility - $5.5 million over four years). These are welcome developments but are token in relation to the task of comprehensively documenting Australia's biota and addressing the decline in university training, career pathways and the taxonomic workforce.

In response to this taxonomic capacity challenge, a seminal meeting was held recently at the Australian Museum in Sydney (October 4 and 5); the National Forum on Australian Taxonomy', where leading taxonomists attended from universities, herbaria and museums. This was initiated and coordinated by the peak Commonwealth taxonomic funding agency, the Australian Biological Resources Study. There was substantial agreement in the identification of the workforce shortfalls, societal relevance, and recommendations were presented to reverse current trends.

Strategies identified by speakers for moving forward include: development of university biodiversity training programmes (with the possibility of national coordination), research fellowships for early career researchers and greatly enhanced funding to target research in priority areas.

One remedy proposed at the forum was to adopt fully the digital age, with the implementation of taxonomic methodology on the web, so-called cybertaxonomy. Organisms know no political borders, as our karnal bunt example shows, and fundamentally their documentation is a global issue. Critical to the cybertaxonomy model is the development of research teams, akin to that in astronomy, which theoretically results in classifications that are more robust, predictive, and information is readily accessible. The need to expedite the documentation of the Earth's biota is critical, and in recognition of this, the US National Science Foundation in 2003 established a new program, the Planetary Biodiversity Inventory, to fund flagship projects and establish international research teams. Of the seven projects that have been funded, three have Australia at their core, with research hubs established at the University of New South Wales (plant bugs), University of Adelaide (parasitic wasps), and the Queensland and Western Australian Museums (goblin spiders), all of which work in collaboration with US counterparts. Funding for these projects are on a scale not seen before in taxonomy (more than $3 million per project).

One of the key attributes in this program is the training of the next generation of taxonomists, and it represents explicit reform and succession planning for taxonomy. This is largely absent in Australia's response to the taxonomy crisis, and is most apparent in the demise of the discipline in the tertiary sector, where few taxonomists are employed in Australian universities. In real terms there has been a 21% increase in funding for biological research in universities between 2001and 2004, but a 14% decrease for taxonomy.

The institutional mainstays for taxonomy are in Australia's museums and
herbaria, but their survival is also currently at stake and their capacity for training is constrained. Innovative approaches, such as joint taxonomic appointments between universities and museums/herbaria have been successful (e.g., South Australian Museum and the University of Adelaide), but leadership from the higher education sector is largely missing.

As a result of the National Forum of Taxonomy, a taskforce has been established to develop policy solutions to the taxonomy crisis in Australia. Succession planning in taxonomy is critical, and it has been estimated that a $50 million enhancement over the next four years would secure our capacity to deliver environmental information into the future. The taxonomic community has awoken and is ready to provide national leadership.

Professor Gerry Cassis (University of New South Wales, zoologist) and
Professor Pauline Ladiges (University of Melbourne, botanist)