June 6, 2009

The Wisdom of the Crowd in One Mind

James Surowiecki's 2004 best-selling book, The Wisdom of Crowds, described the phenomenon that when estimates by a diverse group of people are averaged together, the resulting group answer is more accurate than the estimate of a typical member. The reason this is the case is that using a large sample of imperfect estimates tends to cancel out extreme errors and converge on the truth. For instance, when students are asked to estimate the temperature in a classroom the group answer (averaged estimates) will usually be more accurate than the estimate of any typical member.
Stefan Herzog and Ralph Hertwig have now published a study (The Wisdom of Many in One Mind) which shows it can be possible to benefit from the wisdom of crowd principle (averaging multiple erroneous estimates) as an individual. Herzorg and Hertwig had participants in their study make a first guess and then a second guess. Before making the second guess, however, the participants were given the following directions: “First, assume that your first estimate is off the mark. Second, think about a few reasons why that could be. Which assumptions and considerations could have been wrong? Third, what do these new considerations imply?... Fourth, based on this new perspective, make a second, alternative estimate.”
By averaging these two estimates significant accuracy gains were achieved which were about half size of the accuracy gains that would have been achieved by averaging with a second person. What is important to notice is that second estimates were not more accurate than first estimates. The accuracy gains were only realized after averaging the first and the second estimate.

5 comments:

  1. Interesting.
    At first, when I read the title of your post, I thought you were referring to something like the concept of "society of minds" by MInsky and a connectionist framework..
    but this is news to me, thanks!

    ReplyDelete
  2. Hi Paolo, thanks. Have you noticed this is again some kind of algorithm which enhances performance?

    ReplyDelete
  3. Actually no, I did not.
    But now I do! :)

    ReplyDelete
  4. You might also be interested in what are called "prediction markets." Apparently allowing people to bet on an outcome (and bet more if they feel they have an insider insight of some kind) results in incredibly accurate forecasting. It's similar to what you were saying, but better in a way because people who think they don't know don't bet and thus don't taint the data. That's a really quick into, but it's worth looking into. Thanks for the post.

    ReplyDelete
  5. Hi Tim, thank you. I did know about prediction markets; fascinating indeed, aren't they?

    ReplyDelete