A previous essay, Why Smart People Defend Bad Ideas, continues to get feedback. People leave comments often and here’s a particularly interesting link (thanks CCJ).

From Studies in Intelligence (vol 47 no 1), a declassified CIA journal:

The Paradox of Expertise: the strengths of expertise can also be weaknesses. Although one would expect experts to be good forecasters, they are not particularly good at making predictions about the future. Since the 1930s, researchers have been testing the ability of experts to make forecasts. The performance of experts has been tested against actuarial tables to determine if they are better at making predictions than simple statistical models.

Seventy years later, with more than two hundred experiments in different domains, it is clear that the answer is no. If supplied with an equal amount of data about a particular case, an actuarial table is as good, or better, than an expert at making calls about the future. Even if an expert is given more specific case information than is available to the statistical model, the expert does not tend to outperform the actuarial table.

The full essay talks about the paradox of using teams to balance against expert bias, the role of methodology, etc. The context of the essay is, you guessed it, CIA type intelligence gathering, but in reading this essay much of it applied to any kind of complex work.

The full essay is here.