A previous essay, Why Smart People Defend Bad Ideas, continues to get feedback, despite the fact that I wrote it before I knew the concept of cognitive bias. People leave comments often and here’s a particularly interesting link (thanks CCJ).
The Paradox of Expertise: the strengths of expertise can also be weaknesses. Although one would expect experts to be good forecasters, they are not particularly good at making predictions about the future. Since the 1930s, researchers have been testing the ability of experts to make forecasts. The performance of experts has been tested against actuarial tables to determine if they are better at making predictions than simple statistical models.
Seventy years later, with more than two hundred experiments in different domains, it is clear that the answer is no. If supplied with an equal amount of data about a particular case, an actuarial table is as good, or better, than an expert at making calls about the future. Even if an expert is given more specific case information than is available to the statistical model, the expert does not tend to outperform the actuarial table.
The full essay does explore advantages that experts have. It also discusses the paradox of using teams to balance against expert bias, the role of methodology, etc. The context of the essay is, you guessed it, CIA type intelligence gathering, but in reading this essay much of it applied to any kind of complex work.
Do you know of other attempts to quantify the value of expertise? Please leave a comment. Thanks.