Overestimating What We Know: The Trap of Effortless Search

You would like to learn about a particular topic.  You have two options.  You can search online for information, which AI tools and search engines summarize quickly for you.  Alternatively, you could dig deeper, read an article or two, listen to a podcast, and... maybe even read a book on the subject!  How much do we rely on the easy route to knowledge, and how confident are we that we have become highly knowledgeable through our rapid search for information?

In her amazing book, Uncertain: The Wisdom and Wonder of Being Unsure, author and journalist Maggie Jackson explores this topic.  She writes:

After even a brief online search, information seekers tend to think they know more than they actually do, according to a decade of studies. In one set of five experiments, people were asked to study weighty topics, such as autism or inflation, before taking a quiz on the subject. Half the participants were told to find an online article on the topic, while others were simply given the same information without having to search for it.  People who searched online were far more overconfident going into the quiz.  In one round, they predicted that, on average, they would get two-thirds of the questions right, although they scored less than 50%. 

In contrast, people who had been given the information studied longer, absorbed more, and got about 60% of the questions right - about what they had expected.  Rarely if ever in life are just handed information. Searching and seeking are the human condition. But how we do so matters.  In the virtual realm, we seem to lose the ability to sense that we don't know, the starting point of discernment.  This false confidence blossoms even when people learn nothing from an online search, further studies show. By assuming we can know effortlessly, we close our eyes to our failings and so to chances to explore.  We run from the work of fully attuning to the here and now, finding in hubris a retreat from the challenges of facing up to reality as potent as that of outcome-oriented fear. 

This research suggests that we need to proceed with caution when we jump to conclusions based on a breezy online search or quick prompt on an AI tool such as ChatGPT, Gemini, or Claude.   Ask yourself: What do I actually know?  How deep and accurate is my knowledge?  Should I be making critical decisions based on this superficial knowledge?

Can Cold Hard Facts Mitigate Overconfidence Bias?

Do humans suffer from overconfidence bias?  You bet!  Most of us think that we are above average.  In fact, studies show that, "The most incompetent people, in whatever skill researchers ask about, tend to overestimate themselves by the widest margin. People who actually are above average are often pretty good at self-assessment, and some rock stars even mistake themselves as closer to average than they are." 

Many people have argued, however, that overconfidence can and should diminish if individuals are exposed to objective performance evaluation data.  Is that true?  Well, a new paper by Patrick Heck, Daniel Benjamin, Daniel Simons, and Christopher Chabris (published in Psychological Science) questions that conventional wisdom.   They studied over 3,000 tournament chess players from 22 countries.  In chess, each player has a rating that accurately reflects their probability of winning a contest.  In short, chess players have access to objective, accurate performance evaluation data.  Yet, overconfidence persists even in the face of cold hard facts! The scholars report:

"On average, participants asserted their ability was 89 Elo rating points higher than their observed ratings indicated—expecting to outscore an equally-rated opponent by 2:1. One year later, only 11.3% of overconfident players achieved their asserted ability rating. Low-rated players overestimated their skill the most and top-rated players were calibrated. Patterns consistent with overconfidence emerged in every sociodemographic subgroup we studied. We conclude that overconfidence persists in tournament chess, a real-world information environment that should be inhospitable to it."

Hubris gets the best of us at times, and it certainly affects business leaders in many situations.  My conclusion from this study is that we can't simply expect good outcome measures to mitigate overconfidence bias.  Pointing to the facts is not enough.  People's emotions matter, and their identity shapes how they will make sense of objective performance data.  As we give feedback or evaluate performance, we need to consider the likelihood that distorted perceptions of self-efficacy may not go away just by pointing to the numbers.  We have to appeal to people in ways that go beyond the data if we wish to help them reset their self-evaluations and improve based on our feedback.  

Super Bowl Coaches, The Curse of Expertise, and The Importance of Perspective Taking

Source: Sporting News

As we approach the Super Bowl this weekend, here's a quick look back at the history of NFL championship coaches.  The data are clear.  In football, championships are generally not won by coaches who were formerly superstar players.  33 coaches have won the 54 Super Bowls that have taken place. Several coaches have earned multiple championships, including Bill Belichick (6) and Chuck Noll (4). Of those coaches, only 1 man made the Pro Football Hall of Fame as a player (Mike Ditka). Only 2 men earned Pro Bowl status as players (Mike Ditka and his mentor, Tom Landry, who made it to one Pro Bowl as a punter for the New York Giants in the 1950s). None of the other Super Bowl winning coaches earned Pro Bowl status as a player. 

Why might great players not thrive as coaches in the league?  Multiple potential explanations exist.  However, I'll focus here on something called the curse of expertise.   Put simply, experts sometimes have a difficult time teaching much less experienced and accomplished people.  Why?  They forget what's it like to be in the novice's shoes.  They can't predict the types of challenges and problems that the novice will face when mastering a new skill.   In many cases, the expert may not even be fully aware of the "how" behind certain highly effective results.  It comes so naturally to them that they don't have a complete understanding of the process that leads to those successful outcomes.   

Moreover, a study by Yale psychologists Matthew Fisher and Frank Keil shows that experts think they can teach and explain effectively.  In other words, they often don't recognize the curse of expertise.  Fisher and Keil conducted a series of experiments demonstrating that, "Highly educated individuals tend to overestimate their ability to explain their own areas of formal expertise."   In short, they argue that the experts' "confidence is unwarranted."  

Managers in business face the same curse of expertise as they lead teams and organizations.  They may not be able to recognize the problems and challenges that front-line or entry-level workers face as they try to master their job.  They may be overconfident in their ability to teach their team members.  To overcome the curse of expertise, managers need to get better at perspective taking.  They need to be able to step into their employees' shoes.  They must ask open-ended questions to learn about the challenges their team members are facing.  They have to reflect back and recall some of the early failures and stumbles in their career.   They need to check in with their employees and test for understanding after explaining something to them.  How did they interpret what I said?  Did they understand my rationale, my thinking, my logic?  Finally, managers need to invite questions.  What more would you like to know?  How can I clarify what I have explained?  The better managers become at perspective taking, the more likely they are to overcome the curse of expertise.  

Underestimating our Creativity

We are all generally familiar with the notion that people tend to exhibit overconfidence in their abilities across a variety of domains. However, some recent research suggests that we are not at all overconfident when it comes to assessing the originality of our own ideas during the early stages of the problem-solving process. We underestimate our own creativity. Ella Miron-Spektor of INSEAD writes,

"The results of three studies we conducted on this little-researched facet of creativity were striking. We found that people systematically underestimate their originality – a defining characteristic of creativity – throughout the ideation process. Such bias in the evaluation of our originality stems from the belief that other people think like us."

Professor Miron-Spektor notes that individuals become overconfident when they invest a great deal of time, effort, and energy into refining an idea that they selected from among many they may have considered initially.  However, early on in the process of generating ideas, we may actually underestimate how novel our possible solutions are.   What, then, should managers do with this finding?  How should it affect their behavior as they lead team members in a problem-solving process?   Miron-Spektor writes:

As a manager, make it a point to invite your team members to share their ideas which may otherwise never see the light of day because they assume their ideas are not innovative enough. By acknowledging their creativity, you can reduce their bias, at least to some extent.

 For more about my own research on creativity in organizations, see the short video clip below: 

Are You an Overconfident Leader?

Dartmouth Professor Sydney Finkelstein, author of Superbosses: How Exceptional Leaders Master the Flow of Talent, has published a thought-provoking article in the Wall Street Journal this week. He poses four questions that leaders can ask themselves to determine whether they suffer from overconfidence. I think it's a useful list. Here are the four questions:

1.  How much time do I really spend listening?

2.  Do I originate most of the ideas?

3.  Do I often feel like I’m the smartest person in the room?

4.  Do I think of myself as indispensable to my business’s success?

Falling in Love with Rosy Scenarios

Source: Flickr

Optimism can be a powerful force in an organization.   Certainly, leaders need to be optimistic if they wish for the firm to accomplish ambitious goals and objectives.   We won't get anywhere if people are always saying, "We can't do that.  It's not realistic.  That won't work here."   Having said that, leaders sometimes fall in love with rosy scenarios.  They fail to question assumptions, and they don't look closely enough at worst-case scenarios.  David Breashears, the great mountaineer and accomplished filmmaker, once told me that some leaders are not really leaders; they are cheerleaders.  He said that they never talk about failure.   Breashears pointed out that you have to express confidence in your team, but you have to consider and prepare for failure scenarios.  If you don't, you could die on a mountain.  

My experience suggests that many leaders don't want to talk about failure.  They enjoy setting very ambitious targets and celebrate the notion of achieving the impossible.   However, the best leaders understand precisely the nature of the challenge that they are setting forth for the organization.  They might be setting aggressive targets, but they understand the risks.  They have probed each assumption, and they recognize where things can go wrong.  Researcher Gary Klein advocates the use of the pre-mortem, an exercise designed to imagine what failure might look like before you embark on a project, so that you can either build contingency plans or alter your initial strategy so as to enhance the odds of success.   Many leaders do not utilize pre-mortems, though they are highly effective in institutions such as the military.  

In today's Wall Street Journal, Thomas Gryta and Ted Mann have penned an in-depth look at the fall of GE, based upon extensive interviews with current and former employees.  At point, they describe former CEO Jeff Immelt's penchant for unbridled optimism.   They offer this anecdote:

Already the chief of GE’s largest business by sales, Bolze, 52 years old and square-jawed, was in the race to succeed Immelt, and he was about to add a huge new global portfolio of power plants and thousands of workers to his fiefdom.  Moving through the slides, Bolze came to the proposed annual sales growth rate of the power business: 5%.

There was ample reason for skepticism. Power had been struggling to meet targets, and its sales hadn’t grown that quickly in years. Global investment in new gas-fired power plants was slowing. Energy efficiency was on the rise. That meant future revenue from the highly profitable service contracts GE had signed was likely to fall, or at least to grow less quickly. Global gross domestic product, a reliable proxy for the power market, was below 4%.  It was a rosy assumption that cried out for interrogation, the very point of the formal review. As the room watched, Immelt gave the desk in front of him a confident slap.  

“Great, next page,” he said.

Immelt could be tough on executives in his own way in these briefings, but it wasn’t usually for being too optimistic. “Where’s the guy I used to know?” he would ask an underling who told him Immelt’s targets couldn’t be hit. When the mood soured, the tone changed. “Your people,” Immelt would say, “don’t want it bad enough.”

The story is rather disconcerting, but don't think that GE and Immelt are unique.  This story plays out in many organizations each day.  The most successful leaders become comfortable with managing a delicate balancing act.  They aim high, but they critically examine how failure might occur.  They uncover hidden risks, and they probe for faulty assumptions and problematic strategies.   They are healthy skeptics.  

Reducing Leader Overconfidence

Daniel Walters, Philip Fernbach, Craig Fox, and Steven Sloman have developed an intriguing and apparently quite effective technique to curb overconfidence.  They published their research in a paper titled, “Known Unknowns: A Critical Determinant of Confidence and Calibration."  Walters described the key findings and implications from this research in a web essay published by INSEAD,  where he serves on the faculty. 

The scholars describe a technique in which people are asked to "explicitly consider the missing pieces of information in a judgment." In other words, they try to identify and write down what they don't know about a situation, i.e. what are the key unknowns? They compared this methodology to another technique often recommended for improving decision-making effectiveness: devil's advocacy. Walters reports that identifying unknowns can be more effective than devil's advocacy. Why? In one of their experiments, they examine two situations: one setting in which people were overconfident and another where they were underconfident. They found that devil's advocacy reduces confidence in both settings. On the other hand, considering unknowns only reduces confidence in the situation where people were initially overconfident.  Devil's advocacy proves to be a "blunt instrument" in their words.  Here's Walters' summary of the research:

The third study allowed us to test whether considering the unknown reduced confidence or improved calibration. In many domains, people demonstrate underconfidence and are overly cautious. A true improvement in calibration would mean that considering the unknowns reduces confidence when people are overconfident, but not when people are well-calibrated or underconfident. In this study, participants answered two sets of general knowledge questions. The questions were divided into nine knowledge domains (e.g. state populations, calorie counts), for which participants varied in their level of overconfidence versus underconfidence. As in the second study, participants either considered the unknown, or considered the alternative (the devil’s advocate technique). Both interventions were compared with a group which had no prompting to ponder additional information. As we predicted, considering the unknowns only reduced confidence when it was misplaced (in overconfident domains), whereas playing devil’s advocate had an equal impact in the subject areas that encouraged overconfident and underconfident responses.

The research is intriguing.  I would offer two caveats.  First, I would argue that many organizational leaders display overconfidence much more frequently than underconfidence.  Second, I have come to believe that WHO plays the devil's advocate, WHEN they play that role, and HOW they serve as the devil's advocate matters a great deal.  Indeed, it is a blunt instrument, particularly if not used properly. However, with some care, the technique can be deployed with much success.   

Overconfidence in Founder-Led Firms

Bradley Hendricks and Mark Lang, and Kenneth J. Merkley evaluated 11,285 10-Ks reports filed by firms that went public between January 1, 1997 and December 31, 2013.  They have published a paper about a key distinction between the filings of founder-led firms and companies led by non-founders. The paper is titled, “Through the Eyes of the Founder: CEO Characteristics and Firms’ Regulatory Filings.”

Hendricks, Lang, and Merkley find a significant degree of overconfidence on the part of the founder-led firms.  They report, "Consistent with firms’ financial reports reflecting the founder’s optimism, we find that the 10-Ks for founder-led firms tend to be characterized by both more positive text and less negative text relative to 10-Ks for other firms."  

Prior research suggests that a positive tone in the 10K filing leads to a higher stock price.  However, the authors questioned how investors would react to the optimism conveyed by founder-led companies.  Perhaps investors would be attuned to the fact that entrepreneurs can exude overconfidence, and thus they would discount the optimistic statements made by founder-led firms.  What did they actually find?   Share prices rise in a similar fashion in the immediate aftermath of the 10K filing for both founder-led firms and those led by non-founders.  The authors state, "We find that investors do not appear to recognize founders’ tendency toward overoptimism and, instead, interpret tone for founder-led firms similarly to non-founder-led firms."  

What happens later on though?  For the full year after the 10K filing, the scholars found a negative relationship between the tone of the 10K and stock price performance for the founder-led firms.  No such negative relationship exists for the firms led by non-founders.   That finding supports the notion that investors mistakenly accept the overly optimistic pronouncements of founders initially, and the correction only takes place over time as they realize that the actual performance will not meet lofty expectations.   

What's the lesson here?  We need to analyze company pronouncements not simply in terms of the content, but in terms of the messenger as well.  Who is the company leader, and what is his or her background?  If he or she is an entrepreneur/founder, we might want to be a bit more careful about accepting their positive statements and projections.  

Overconfident CFOs

Strategy and Business reports today on a new study by Itzhak Ben-David, John R. Graham and Campbell R. Harvey.   They examined the accuracy of predictions by chief financial officers (CFOs).  They examined over 13,000 surveys of chief financial officers in 2011.  What did they find?  "Instead of hitting the right range 80 percent of the time, the CFOs were correct in only about 36 percent of the cases when predicting the market’s point total a year out, the authors found. Even during the least volatile periods in the sample, CFOs had only a 59 percent success rate."  

Psychologists, of course, have long known that human beings are subject to an overconfidence bias.  Interestingly, this study shows that the overconfidence extends to projections about company performance as well.  According to Strategy and Business, "CFOs who erred on market forecasts also tended to provide unrealistic estimates of returns on investment for their firm’s projects. These hubristic CFOs failed to anticipate volatility and risks, even when they could look to benchmarks like their firm’s return on invested capital as a basis for their predictions."  

What I do take away from this study?   In many companies, the senior management team looks to the CFO to be the "voice of reason" or the "force for restraint" in the face of ambitious executives who want to invest, grow, and expand.  This study shows that many CFOs may not be effective forces of reason and restraint. Instead, their overconfidence may add fuel to the fire when it comes to flawed strategies.  Far from being the cynic or the ultra-conservative person watching the pennies closely, some CFOs may dramatically underestimate the risks associated with certain investments.  Moreover, they may support rosy projections for the future. 

Overconfident CEOs and Earnings Forecasts

Executives must have confidence to succeed, but how much is too much?   How specifically does overconfidence affect organizational actions?   Wharton Professor Holly Yang and Iowa Professor Paul Hribar have written an interesting new paper regarding CEO overconfidence.   They examined 974 CEOs from Fortune 500 firms in the 2000-2007 time period.  They first examined the extent to which the CEOs were overconfident, based on descriptions and quotes in news articles, press releases, etc.  Then they examined those firms' earnings forecasts.  Yang describes their findings: "We found that if a CEO is classified as overconfident, then his or her chance of missing the forecast is 10% higher than for a manager who is less confident."  The authors wrote in their paper: "Overconfident CEOs are more likely to issue optimistically biased forecasts because they overestimate their ability to affect their financial results and/or they underestimate the probability of random events."