Don't Overestimate Your Powers of Prediction

This week is the NFL Draft. Here's my analysis of 15 years of data on 1st round picks at QB. Conclusion: It's very hard to predict success. True in other fields too. Lessons for us all: don't overestimate powers to predict; don't misattribute forecasting success/failure based on few data points.

Predicting Accurately: Focus on Outcomes First, Not the Details?

Theresa F. Kelly and Joseph P. Simmons have published a new study in the Journal of Experimental Psychology regarding our ability to make accurate predictions.   Kelly and Simmons conducted 19 experiments involving over 10,000 research subjects.  They studied people's ability to predict the outcomes of 724 professional sports contests.  Interestingly, they found that participants were less effective at predicting the outcomes of games if they also were trying to predict details about the contests (e.g. if they tried to predict the number of hits each team would get in a baseball game, while also trying to predict the final score of the game).   

Why might people become less effective in predicting the outcomes of games if they were also trying to forecast specific details about those contests?   The scholars explain, 

We believe that this happens because having people predict the details of an event makes them think about additional information that is unimportant for predicting other related outcomes; however, once this information is made accessible in memory, people are more likely to use it in their forecasts, decreasing the weight given to more important information. This suggests that a relatively simple way to improve predictions could be to take a top-down approach and start by first predicting the most general outcomes and then letting those forecasts guide predictions of more detailed outcomes. Importantly, this prescription is not intuitive, as most decision-makers feel compelled to try to think through all of the available details about an event before making their forecasts (Lovallo & Kahneman, 2003).

Improving Your Predictions

A great deal of research shows that experts are often not very accurate in their predictions.  In this video, Wharton marketing professor Barbara Meller discusses how we can improve our forecasting abilities.  It's worth taking a look. 

Should You Trust the Expert?

On the Washington Post website, Darden Business School Professors Yael Grushka-Cockayne and Kenneth C. Lichtendahl Jr., ask the provocative question: Is it better to trust the best expert, or the average of a group of experts?  They examine this question in the context of economists forecasting economic growth for the Wall Street Journal.  They remind us about the concept of "the wisdom of crowds" described so eloquently by James Surowiecki of The New Yorker.  Then, they describe research by two Duke professors, Rick Larrick and Jack Soll:

Rick Larrick and Jack Soll, business professors at Duke University, have shown that when given a chance to do so, people often prefer to rely on experts. In laboratory experiments, they found that where experts disagreed, people would deem the “most able” among them and trust that individual’s judgment more.  Despite this perception, the average forecast often outperforms the best individual’s forecast. Such outperformance happens when forecasts bracket the true result. 

In their column for the Washington Post, the two Darden Professors examine past data from the Federal Reserve Bank of Philadelphia, an institution that surveys economists and relies on averaging of the individual forecasts.  They found that, "The crowd beat the expert in 63 percent of the 40 quarters. Not surprisingly, any two forecasters in the survey often bracketed the truth, bracketing on average 28 percent of the time."   

No surprises here... we have known about the wisdom of crowds for some time.  What is fascinating is that people tend to want to rely on the individual expert.  They prefer the expert over the crowd... absolutely the wrong strategy.