Complexity in expert judgement

  • CERF futures - expert judgement imageWhen we don’t have enough data to make a decision or where analysis or interpretation requires expertise we don’t have (or don’t have enough of) ourselves we often consult a person deemed to be an expert. We use this judgement of expertise to decide who to listen to and it gives us confidence in their answer.
  • New research is beginning to challenge this idea. One study took a group of students and a group of ecological experts, who were asked to predict the outcomes of (in press) ecological experiments. Individuals were provided with the papers’ introduction and method and were asked to specify 80% confidence intervals for their answer.
  • The study showed that experts’ confidence intervals captured the truth 49-65% of the time while the students captured the truth 76% of the time.
  • Importantly, there was no correlation between performance and years of experience, self-assessment of expertise or publication record.
  • Additional research has shown that using the Delphi technique to elicit expert judgement from groups offers large improvements in performance. The process asks individuals for their judgements, presents all answers anonymously to the group, then individuals re-judge their answers. Again, domain experts perform no better than professionals from other domains in this task.

Implications and next steps: This evidence suggests that it is not a person’s ‘expertise’ in an area that makes them perform well. It is their overall ability to question their own judgement (uncertain people perform better), their knowledge seeking behaviour and their logic that may ultimately affect accuracy. The research offers important considerations for policy making. It may be important not to rely wholly on the opinions of a single expert, and to use robust methods to elicit this opinion, as is already the case in many instances. Additionally, it may be important to request that groups of experts make their confidence levels explicit as part of the process.

Diversity and distributions: http://tinyurl.com/c2w8ggh; Expert Knowledge and Its Application in Landscape Ecology (book): http://tinyurl.com/cwx6lma; Conservation Biology (method recommendations): http://tinyurl.com/cnua25l; Presentation: Delivered by Mark Burgman at Cranfield University 14th March 2013.

About Hayley Shaw

Knowledge Exchange Manager at Cranfield University's Centre for Environmental Risks and Futures (CERF). Sharing the latest news from Cranfield, and insights from across the industry. All things risk, environment, and the future.
This entry was posted in Consumer Attitudes and Behaviour. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s