The Paradox of Expertise
A previous essay, Why Smart People Defend Bad Ideas, continues to get feedback, despite the fact that I wrote it before I knew the concept of cognitive bias. People leave comments often and here’s a particularly interesting link (thanks CCJ).
From Studies in Intelligence (vol 47 no 1), a declassified CIA journal:
The Paradox of Expertise: the strengths of expertise can also be weaknesses. Although one would expect experts to be good forecasters, they are not particularly good at making predictions about the future. Since the 1930s, researchers have been testing the ability of experts to make forecasts. The performance of experts has been tested against actuarial tables to determine if they are better at making predictions than simple statistical models.
Seventy years later, with more than two hundred experiments in different domains, it is clear that the answer is no. If supplied with an equal amount of data about a particular case, an actuarial table is as good, or better, than an expert at making calls about the future. Even if an expert is given more specific case information than is available to the statistical model, the expert does not tend to outperform the actuarial table.
The full essay does explore advantages that experts have. It also discusses the paradox of using teams to balance against expert bias, the role of methodology, etc. The context of the essay is, you guessed it, CIA type intelligence gathering, but in reading this essay much of it applied to any kind of complex work.
Do you know of other attempts to quantify the value of expertise? Please leave a comment. Thanks.
Check out the work of another Phil. Tetlock has done quite a bit of work here over more than two decades.
Super Crunchers by Ian Ayres (http://www.supercrunchers.co.uk) takes a statistical look at just this topic and explores the overlap of big data with our ability to ask the right questions of this data to come up with better predictions than the so-called experts. Examples range from the price of fine wine to whether or not an airline should give you the remaining seat on the last plane home. It’s a fascinating read.
That is a fantastic book!