Reducing Leader Overconfidence

Daniel Walters, Philip Fernbach, Craig Fox, and Steven Sloman have developed an intriguing and apparently quite effective technique to curb overconfidence.  They published their research in a paper titled, “Known Unknowns: A Critical Determinant of Confidence and Calibration."  Walters described the key findings and implications from this research in a web essay published by INSEAD,  where he serves on the faculty. 

The scholars describe a technique in which people are asked to "explicitly consider the missing pieces of information in a judgment." In other words, they try to identify and write down what they don't know about a situation, i.e. what are the key unknowns? They compared this methodology to another technique often recommended for improving decision-making effectiveness: devil's advocacy. Walters reports that identifying unknowns can be more effective than devil's advocacy. Why? In one of their experiments, they examine two situations: one setting in which people were overconfident and another where they were underconfident. They found that devil's advocacy reduces confidence in both settings. On the other hand, considering unknowns only reduces confidence in the situation where people were initially overconfident.  Devil's advocacy proves to be a "blunt instrument" in their words.  Here's Walters' summary of the research:

The third study allowed us to test whether considering the unknown reduced confidence or improved calibration. In many domains, people demonstrate underconfidence and are overly cautious. A true improvement in calibration would mean that considering the unknowns reduces confidence when people are overconfident, but not when people are well-calibrated or underconfident. In this study, participants answered two sets of general knowledge questions. The questions were divided into nine knowledge domains (e.g. state populations, calorie counts), for which participants varied in their level of overconfidence versus underconfidence. As in the second study, participants either considered the unknown, or considered the alternative (the devil’s advocate technique). Both interventions were compared with a group which had no prompting to ponder additional information. As we predicted, considering the unknowns only reduced confidence when it was misplaced (in overconfident domains), whereas playing devil’s advocate had an equal impact in the subject areas that encouraged overconfident and underconfident responses.

The research is intriguing.  I would offer two caveats.  First, I would argue that many organizational leaders display overconfidence much more frequently than underconfidence.  Second, I have come to believe that WHO plays the devil's advocate, WHEN they play that role, and HOW they serve as the devil's advocate matters a great deal.  Indeed, it is a blunt instrument, particularly if not used properly. However, with some care, the technique can be deployed with much success.