Advisers and Uncertain Advice

Celia Gaertig and Joseph Simmons have published a paper titled, "Do People Inherently Dislike Uncertain Advice?"   The authors focus on the longstanding research finding that individuals prefer confident to uncertain advisors.   Individuals generally don't want take advice from someone who does not appear self-assured.  Gaertig and Simmons extend this research by examining the question:  Do people exhibit an aversion to uncertain advice itself?

The scholars studied advice evaluation in a series of studies focused on issues such as sports predictions, finance, and weather.  Their findings proved remarkably consistent.   Indeed, people evaluate confident advisers more favorably than those who appear uncertain and hesitant.  However, people do not necessarily dislike uncertain advice.   Here is an excerpt from their paper: 

In eleven studies, we found that people do not inherently dislike uncertain advice. We observed this in studies of sports, weather, and stocks. We observed this in studies that operationalized uncertain advice as imprecision, as statements of numerical probability, and as statements of non-numerical uncertainty. And we observed this in studies in which people directly evaluated the advice and in studies that asked people to choose between an advisor who provided certain advice versus one who provided uncertain advice.  

This paper gives me comfort.  It shows that people are quite capable of sifting through advice, and recognizing the merits of advice couched in the form of probabilities.  We do not expect complete certainty.  That's a good thing, as I believe we should be skeptical of advice and predictions that seem to express absolute certainty.    We just might be more rational than some people think!  

How You Think vs. What You Know

Philip Tetlock wrote a terrific book last year titled Superforecasters.  In that book, he describes the  decades of research he has conducted on expert predictions. Put simply, he has found that experts don't make great prognosticators, despite their vast knowledge and experience. Knowing a great deal does not enable you to see the future. In fact, it can be a handicap at times, because you may be tied down by conventional wisdom, long-held assumptions, and pre-existing beliefs. In Tetlock's research, he has found that a small set of people have an uncanny ability to make accurate predictions. These people are not experts in the domain in which they made those forecasts. What enables them to predict so effectively? Tetlock conclues that the key is HOW THEY THINK, not what they know. The best forecasters exhibit the following characteristics:

• They are open-minded, reflective, and intellectually curious. 
• They acknowledge what they do not know. 
• They gather information from a wide variety of sources and question the validity of each source. 
• They enjoy pondering a range of diverse views, and they update their conclusions as facts change. 
• They treat beliefs as testable hypotheses rather than hard truths.

In my view, the same might be said about the best leaders.  They often distinguish themselves by how they think, not what they know.   The best leaders are reflective and intellectually curious, question the conventional wisdom, do not become wedding to existiing beliefs, admit what they do not know, critique the validity of information sources, and gather diverse opinions before making key decisions.  

Predicting Accurately: Focus on Outcomes First, Not the Details?

Theresa F. Kelly and Joseph P. Simmons have published a new study in the Journal of Experimental Psychology regarding our ability to make accurate predictions.   Kelly and Simmons conducted 19 experiments involving over 10,000 research subjects.  They studied people's ability to predict the outcomes of 724 professional sports contests.  Interestingly, they found that participants were less effective at predicting the outcomes of games if they also were trying to predict details about the contests (e.g. if they tried to predict the number of hits each team would get in a baseball game, while also trying to predict the final score of the game).   

Why might people become less effective in predicting the outcomes of games if they were also trying to forecast specific details about those contests?   The scholars explain, 

We believe that this happens because having people predict the details of an event makes them think about additional information that is unimportant for predicting other related outcomes; however, once this information is made accessible in memory, people are more likely to use it in their forecasts, decreasing the weight given to more important information. This suggests that a relatively simple way to improve predictions could be to take a top-down approach and start by first predicting the most general outcomes and then letting those forecasts guide predictions of more detailed outcomes. Importantly, this prescription is not intuitive, as most decision-makers feel compelled to try to think through all of the available details about an event before making their forecasts (Lovallo & Kahneman, 2003).

More on Making Good Predictions: An Interview with Philip Tetlock

Superforecasters: Foxes vs. Hedgehogs

You can't go anywhere, watch anything, or read any article these days without hearing from an expert or pundit predicting the future.  Should we pay much attention to those forecasters?  Well, according to research by Philip Tetlock and his fellow scholars at the Good Judgment Project, we might want to remain skeptical of such predictions.   They have found that many experts are not very accurate in their predictions.  On other hand, they discovered some folks who they term "superforecasters."  Who are these folks?  Here's an excerpt from a BBC story on Tetlock's research:

Roberts cites the Isaiah Berlin essay “The Fox and the Hedgehog” – a comparison often used by Tetlock himself – which divides thinkers into those “hedgehogs'” narrowly invested in a single topic and “foxes” with a wider, if shallower, range of experience. “Foxes” like him, Robert says, tend to be better forecasters. “They don't get attached to one particular narrative” and are able to adapt their viewpoints to incorporate any new information, unlike “hedgehog” thinkers, who often force new information into a pre-existing mental framework, or discard it if it seems to contradict their initial view.

In short, the problem with many experts is that they are hedgehogs.  They have deep expertise in a narrow domain, but as a result, they also have a lot of biases.   They have a strong attachment to certain pre-existing beliefs.  The foxes are more open-minded.  What else characterizes the superforecasters?

1.  An ability to ask good questions

2.  An ability to assimilate new information, even if it contradicts their initial view

3.  An ability to use the scientific method to pose and test hypotheses

4.  A reliance on data and not just intuition

5.  A willingness to acknowledge their own lack of knowledge in certain areas 

Can Experts Predict the Next Great Startup Success Story?

Scholars Erin Scott, Pian Shu, and Roman Lubynsky have written a fascinating new paper about startups.  They examined a dataset of 652 ventures from MIT's Venture Mentoring Service (VMS).   The service attempts to match startups with mentors.  The mentors receive data about a variety of startup ideas.  They must decide what they think about the ideas without having an opportunity to review information about the founders or to meet the team in person.   The researchers then examined how many of these startups went on to have their products commercialized successfully.  

Overall, the more highly rated ideas did have a better chance of being commercialized.  However, that was not the case for all types of startups.  They grouped the ventures in terms of high R&D intensity industries (i.e. life sciences, energy) and low R&D intensity industries (i.e. software, consumer products).  Highly evaluated ideas tended to be more likely to be commercialized successfully in the high R&D intensity group, but no such relationship was found in the low R&D intensity group.  HBR's Walter Frick explains this finding: 

Think of it this way: if the venture “idea” includes patent-protected technology in an industry with high entry costs, it’s going to be easier to determine that the venture has commercial potential. For web and mobile ventures, which are less likely to have intellectual property, and where entry costs are lower, it’s harder to know up front whether a venture will have a real, sustainable competitive advantage.

Finally, the researchers examined whether experts were better at predicting success.   Frick writes, "The researchers checked to see if “expert” mentors were any better at picking ideas than the group overall. They looked at mentors with experience in the venture’s industry, as well as mentors with a PhD. Neither group was any better at predicting which ideas would succeed."  

Improving Your Predictions

A great deal of research shows that experts are often not very accurate in their predictions.  In this video, Wharton marketing professor Barbara Meller discusses how we can improve our forecasting abilities.  It's worth taking a look.