What Happens When We Can Quantify Some Aspects of a Decision But Not Others?

Do we make biased decisions because we are obsessed with quantifying our decision analysis? Linda Chang, Erika Kirgios, Sendil Mullainathan, and Katherine Milkman have published an interesting new study titled, "Does counting change what counts? Quantification fixation biases decision-making."  They asked the question: "Do people decide differently when some dimensions of a choice are quantified and others are not?"

The scholars conducted a series of experiments.  Each decision that the research subjects encountered involved some tradeoffs.  Some dimensions of the tradeoffs were described quantitatively and others qualitatively.  The results of the experiments demonstrated that people tended to prefer the alternatives about which numerical data was offered, rather than qualitative information.  Thea authors explain that, "This 'quantification fixation' is driven by the perception that numbers are easier to use for comparative decision-making."  

The scholars argue that we face many decisions in business and in life in which some dimensions of the tradeoffs can be quantified, but others simply cannot.  The qualitative information may be rich and useful though.  The numbers may tell only a portion of the story.  Think about a manager facing a decision about a brand extension.  Numbers may be readily available demonstrating the potential for sales growth, market share increases, and profitability enhancement.  On the other hand, the risks around brand dilution may be more readily described in qualitative fashion.  Do managers pay less attention those very real brand dilution risks simply because they can't easily produce numbers about how dilution may arise and impact the business?

The scholars conclude, "Those who structure decision contexts ignore quantification fixation at their peril. As quantification becomes increasingly prevalent, people may be pulled away from valuable qualitative information toward potentially less diagnostic numeric information."

Can Data Analytics Make Your Product Boring?

We live in an age in which companies are investing heavily in data analytics and algorithms.  Clearly, these analyses are often very helpful in making better decisions.  However, I'd like to pose a critical question as leaders ponder the impact of analytics in their organizations.  Can analytics make your product or service boring?  Could it lead to strategy convergence in your industry?  In other words, might an intense focus on analytics by all competitors lead to less much product differentiation?

Consider the case of the National Basketball Association (NBA).   In the 1986 NBA season, teams averaged 3.3 three-point attempts per game.  League MVP and champion Larry Bird attempted 194 attempts for the entire season.   That number led the entire league.  Last year, teams averaged 35.1 three-point attempts per game.  26 players attempted more than 500 three-point shots during the season.   In 1986, games involved a wide array of shooting.  Teams had powerful low-post players such as Hakeem Olajuwon and Kevin McHale.  They had players who could slash to the basket and others who had perfected the mid-range jumper.  Today, the game involves an overwhelming number of three-point shots and very little low-post play.  In the 1980s, some teams played a bruising, slower, physical game.  Others played a run-and-gun, fast-break game.  Today, nearly all teams rely heavily on long-distance shooting. 

Fans can disagree over whether today's game is better than the 1980s version of NBA basketball. However, it is difficult to argue that today's game is more differentiated than the 1980s version.  Most assuredly, teams all play a much more similar style of play today.  Strategy convergence has taken place at an incredible rate.   What's driven this change?  Analytics.  The data clearly say that teams should take a high number of three-point shots.  It simply makes a great deal of sense if you want to win. 

What's the lesson for business leaders?  Analytics may drive what seem like clearly better decisions.  Yet, it might just lead to strategy convergence, which may be harmful in the long run.  Your product may become much more similar to those of your competitors.  Distinctiveness falls, and in the long run, that may actually harm profitability for all rivals.  

Big Data and The Analytics Paradox

Kellogg School Professor Eric Anderson describes an interesting phenomenon that he calls the “Analytics Paradox.”  Anyone grappling with how their firm employs data science as part of their business strategy should become familiar with this concept.  Anderson explains:  

A young firm starts out making many mistakes. Eager to improve, they collect lots of data and build cool new models,” he says. “Over time, these models allow the young firm to find the best answers and implement these with great precision. The young firm becomes a mature firm that is great at analytics. Then one day the models stop working. Mistakes that fueled the models are now gone and the analytic models are starved."

Anderson offers an example of the analytics paradox.  Imagine that a firm provides two-day delivery services.    They consider using data analytics to help make an important decision: whether or not to offer one-day delivery services to customers.  You might be hard-pressed to answer that question using analytics.  Why? An effective organization becomes proficient at executing two-day delivery.  If the firm can't meet two-day delivery deadlines on a regular basis, processes and systems are changed.  Employees who can't meet the two-day delivery schedule get admonished or even dismissed.  In short, a firm that is very effective at execution will drive all variability out of its "production" system.  Yet, without variability, you will find it very difficult to use analytics to drive improved decision making. 

What can you do to conquer the analytics paradox?    Put simply, you need to inject variability into your system by promoting thoughtful and systematic experimentation.   The use of experiments enables you to test different models and systems, and in so doing, generate the type of data that can be analyzed effectively to enhance decision making.  

What's the "Optimal" Failure Rate at Netflix?

When Orange is the New Black, House of Cards, and Crown became mega-hits for Netflix, many people credited the analytics capabilities of the company. Mining the customer data had enabled the firm to project the type of original programming that would be highly successful. By this logic, Netflix would achieve a lower failure rate on new shows than the major television networks. After all, broadcasters such as CBS and NBC cancel a substantial share of their new shows each year, some after only a few episodes. 

On the recent Netflix earnings call, many investors were pleased to hear about strong subscriber growth at the firm. However, some investors came away concerned about the amount of spending taking place as the firm acquires or develops new content. Moreover, some observers and analysts have expressed concern about the recent cancellations of some new Netflix original shows. Tom Huddleston Jr. reported on the company's reaction to this criticism in a recent Fortune article:

Meanwhile, also on the Monday earnings call, Netflix's chief content officer Ted Sarandos defended the company's recent cancellations of a handful of expensive, but underperforming, original series. "The more shows we have, the more likely in absolute numbers that you’ll see cancellations, of course," Sarandos said. The executive compared Netflix's recent spate of cancellations—including big-budget series like The Get Down and Sense8—to traditional TV networks that cancel nearly one-third of their new shows after their first seasons. Netflix, he said, has renewed 93% of its original series. With respect to the shows that Netflix opted not to renew, Sarandos argued: "If you’re not failing, maybe you’re not trying hard enough."

This quote from Sarandos raises a fascinating question.  What is the "optimal" failure rate at Netflix?  Surely, we would like the failure rate to be lower than the broadcast networks.  We would like to see the company reaping the benefits of its analytics capabilities.  At the same time, no one should want Netflix's failure rate on original programming to be zero.  We want the firm to take some chances in hopes of landing some surprising breakthrough hits.  Hopefully, the firm isn't simply guessing or drawing on the intuition of the "creatives" in the business.  We would like to see them engaging in "enlightened" experimentation, using big data to guide them while still taking some risks.   If they balance data mining and risk-taking in an effective way, the failure rate won't be zero, but it will be much lower than their broadcast and cable competitors.  

Combining Intuition and Analysis

We live in an age of big data.  People have become enamored with the use of analytics to solve complex problems and to make better decisions.  I'm a big believer in analytics, but I hope we don't forget that intuition does play an important role in decision-making processes.   In the video below, I explain briefly what intuition is and how it works.  

How can we combine intuition and analysis to help us make better choices?  Here are four ways that the two modes of thinking can complement one another.  

1. Use analysis to check your intuition, but not simply to justify decisions that have already been made 
2. Use intuition to validate and test the assumptions that underlie your analysis 
3. Use analysis to explore and evaluate intuitive doubts that emerge as your prepare to make a decision 
4. Use the intuition of outside experts to probe the validity of your analysis

Why Data Analytics Professionals Need to be Good Storytellers

Kellogg Insights has a good article this week, in which they feature comments from various executives about hiring data analytics professionals. It's a hot field, making it a tough fight for great talent. Still, companies have to find the right talent. That means more than hiring folks who can crunch the numbers. Leslie Hampel, Director of Global Strategies at Starbucks, explains: 

What I am looking for is someone who can bridge the gap. Can you do the math? That’s important, but can you pull the story out of the math? Particularly for the next 10 years or so, as we work our way through this current generation of CEOs who don’t understand algorithms for the most part. They are making decisions from a very different place. How do you show them that you have applied the analytic rigor, then help tell the story so that they feel comfortable investing millions, if not billions, of dollars in this idea? It really becomes about storytelling. 

I concur wholeheartedly.  When I was a graduate student, I taught introductory economics at Harvard.   I also participated in the interviewing and training process for new teaching fellows in this course.  Plenty of talented doctoral students in economics applied for these positions.  Some of them could not tell a story though. They could draw the graphs and write the equations on the board, but they could not explain the intuition and the logic in a way that others could understand easily.   The same logic applies with data analytics professionals.  They have to be able to persuade others, some of whom are not able to digest the math easily.   Telling a story with the numbers is crucial.  

Netflix, Analytics, and Original Programming

Many former and current students have asked me about Netflix's decision to offer original programming.  They always want to know, "Does vertical integration make sense for Netflix or not?"  They know that vertical integration has not always worked out in entertainment industry (think the breakup of Viacom/CBS and the unfortunate consequences of the AOL-Time Warner merger).  I ran across this CNBC interview with Netflix CEO Reed Hastings today (thank you, Professor Jay Rao, for pointing me in this direction!).   Hastings commented on the company's original programming:

Hastings attributed the success of original programming such as "Orange is the New Black" and "House of Cards" to Netflix's powerful data analytics.  "We are just a learning machine. Every time we put out a new show, we are analyzing it, figuring out what worked and what didn't so we get better next time," Hastings added.

Hastings' comments suggest that vertical integration may make a great deal of sense in this case, because Netflix can increase the odds of success with its original programming due to data analytics.  How powerful can data be in this case?  Well, if think about it, Netflix has been invested in "big data" since its inception in the late 1990s (long before big data became a common term).   From the beginning, Netflix did not want to focus on new releases. It wanted to be able to offer a deep library, and then use data to recommend lesser known titles to people.  Now, it's taking that data analytics to a whole new level, by using information it has compiled for over fifteen years to develop original programming.  As we all know, the failure rate for new shows can be quite high.   If Netflix can reduce that rate, even just by a small margin, it can improve the economics of programming substantially.  

Big Data in Formula 1 Racing: Don't Overwhelm Humans

Fortune has a fascinating article about how Formula 1 teams are using the internet of things and data analytics to win auto races.   According to the article, 

"These machines, each valued at more than $9 million (a steering wheel alone is worth $77,000 or so) are more than just pricey contraptions capable of whizzing around the track at more than 200 miles per hour. They are also intelligent, thanks to the many dozens of sensors fastened to them. Each sensor communicates with the track, the crew in the pit, a broadcast crew on-site, and a second team of engineers back home in Europe."  

The team then uses predictive algorithms to help them understand how the car will perform under certain track conditions.  These data guide key decisions.  However, they are careful not to put too much on the driver's plate.  After all, he or she is concentrating on many factors while driving at a very high rate of speed.   The team doesn't want to overwhelm the "cognitive capacity" of the driver.  In other words, they have to boil all that data down to a few key items about which they want to make the driver aware.  

That description sounds quite similar to how a great football coach operates.  They conduct extensive analysis of the opponent, breaking down game film and evaluating data about the strengths and weaknesses of that team.  The coaches then build game plan.  However, they have to keep the ultimate plan simple enough so that players can make fast decisions on the field.  They want them to still act instinctively and not be overwhelmed by too much information.   Managers in all types of enterprises should take note.   We want data analysis to guide people's decisions, but we have to keep in mind the cognitive capacity of those individuals.  We have to be able to boil down all that data to a few key principles and recommendations that they can implement effectively and quickly. 

Big Data Will Not Solve All Our Problems, May Mislead Us

Microsoft Research's Kate Crawford has a terrific blog for HBR about big data.  In the post, she discusses the hype regarding big data, and she talks about the hidden biases that we must be aware of when analyzing large data sets:

Data and data sets are not objective; they are creations of human design. We give numbers their voice, draw inferences from them, and define their meaning through our interpretations. Hidden biases in both the collection and analysis stages present considerable risks, and are as important to the big-data equation as the numbers themselves. 

Crawford has some terrific examples of biases in data sets.  For instance, she talks about how the Twitter data after Hurricane Sandy offers a distorted view of the storm.  Why?  As the storm progressed, people in the hardest hit areas ran out of battery power on their cellphones.  Thus, they stopped tweeting.  Folks in Manhattan, where the storm was significant, but not as devastating, engaged in much more Twitter activity.  Moreover, people in the lowest income groups are not as well represented on Twitter, because many do not own smartphones.  As she writes, "We can think of this as a "signal problem": Data are assumed to accurately reflect the social world, but there are significant gaps, with little or no signal coming from particular communities."

The lesson is clear.  Begin your big data project by asking:  How was the data collected?  What populations are overrepresented?  What populations are underrepresented?  Beyond that, you should ask:  Who collected and assembled the data set?  Do they have an agenda?  Are they biased in any way?  Often, the biggest bias in big data has nothing to do with access to technology or underrepresented populations.  Instead, the most significant bias lies inside the mind of the person assembling the data.  Their agenda clouds the process of data collection. 

How Analytics Can Help You Improve Quality and Reduce Costs

I found a terrific example of the use of "Big Data" in Fast Company magazine this month.   The article by Farhad Manjoo describes a situation at Washington Hospital Center.   ER doctors became concerned that many patients returned to the hospital just a short time after being discharged.  A computer scientist at Microsoft Research began to investigate.  He wanted to identify some triggers that would predict whether a patient would be readmitted.  Specifically, he was looking to help doctors identify some predictors that might not otherwise receive much attention by ER physicians and nurses.  He analyzed more than 300,000 ER visits.    Among other things, he discovered that the length of a patient's stay in ER tended to be a good predictor of readmission.  If a patient stayed in the ER for more than 14 hours, they were likely to return to the hospital within a few weeks.  Similarly, if the patient's chart mentioned the word "fluid" at some point, that seemed to predict readmission quite well too.  

This story illustrates how companies can use analytics to help them understand how to improve the quality of customer service, as well as to reduce costs.  Take an automobile dealer.   They conduct repair and maintenance on thousands of cars per year.  A fair number of those cars return shortly after a repair or maintenance appointment, because something is not working correctly or hasn't been done to the customer's satisfaction.   An automobile dealer could analyze the data from thousands of those cases, and it could try to identify the predictors of return visits.  If they could identify a few solid predictors, then they could try to intervene to reduce those return visits.  Those interventions could improve quality and customer satisfaction, while reduce costs (since every return visit is costly).   Many service businesses could apply a similar logic and use analytics to achieve positive results.   Can your company benefit from such an approach?  

Making the Novel Seem Familiar

I've just finished reading New York Times journalist Charles Duhigg's new book, The Power of Habit.  I enjoyed it a great deal (full disclosure: Charles was my student 10 years ago at Harvard Business School).   Duhigg has a terrific chapter focusing on the predictive analytics function at many corporations, including Target.  Many of you probably read an excerpt from the book which appeared in the New York Times.  That excerpt discussed how Target tried to determine which female shoppers were pregnant so as to begin marketing key products to them before their babies were born.  As we all know, predictive analytics has become a crucial area of focus for many companies.  They can't seem to find enough talented folks who can mine data, conduct sophisticated analyses, and distill key insights.  

Duhigg makes a key point though.  Simply identifying people who are likely to want to purchase your product is not enough.  You have to entice them by "making the novel seem familiar."   It turns out that we are more likely to adopt a new habit if the behavior seems familiar to us.  Duhigg uses the example of a new song.  We tend to listen to music that has some key similarities to music heard often on the radio.   We tend not to listen to dramatically different songs.  Thus, radio stations sandwich such new songs between two tunes that are very familiar.  That tactic entices us to give that new song a shot.  Similarly, Target doesn't just come right out and bombard those pregnant women with a ton of ads and coupons for baby products.  They have come to realize that these women may not want retailers to know that they are pregnant, or they might be alarmed that a company could have figured this out.  Thus, retailers sandwich such targeted marketing between ads and coupons for other products unrelated to pregnancy and children.  They make the novel seem familiar.  It turns out that this tactic often works when it comes to getting us to adopt new habits, in life and in business.

Big Data, Diapers.com, and the Importance of Analytics

Several days ago, the New York Times published an article titled, "The Age of Big Data."  The newspaper described how companies will need many more data analysts who can "help businesses make sense of an explosion of data — Web traffic and social network comments, as well as software and sensors that monitor shipments, suppliers and customers — to guide decisions, trim costs and lift sales."   The article cited a McKinsey Consulting study which predicted that the United States will need 140,000-190,000 more employees with “deep analytical” expertise" in the coming years. 

As an example of the importance of big data, consider the online retailer Diapers.com (owned by Amazon).   Forbes writer Meghan Casserly describes the firm's use of big data in an article published on the magazine's website.  The company has built powerful proprietary algorithms over the past few years based on tons of transactions.  These algorithms predict what customers are likely to buy in the future, how much they will spend, and whether they will be profitable for the firm.  The company's strategy focuses on building loyal customers who purchase low margin baby supplies initially, and then buy higher margin items such as car seats, strollers, and the like in the future.  The algorithms not only help predict purchasing patterns, but they enable Diapers.com and its sister sites to market appropriately to different customers.  Perhaps most importantly, the firm can identify which customers will be profitable for the firm.  Thus, they can spend their time catering to the most profitable customers, rather than wasting marketing expenditures on consumers who will be a drain on resources. 

Every company should be thinking about how it can use algorithms to drive performance.  Analytics can be used in a myriad of ways.   However, building a strategy based on big data requires the right talent.  Therefore, firms need to begin thinking carefully about how they will attract, develop, and retain the talent needed to collect and analyze the huge volumes of data that now exist.  Universities need to think about how to educate people for these roles, as demand will be strong.  We need to do more than educate people in mathematics and statistics though.  We need analysts who can understand business models and strategies, and who have a deep understanding of consumer behavior too.   The best analysts will be those who can marry statistical knowledge with a broader understanding of the entire organizational system.