Big Data AND Consumer Anthropology

We shouldn't be choosing between leveraging big data to make decisions vs. conducting in-depth qualitiative research about our customers.  We should be using both methodologies, as they are often quite complementary.  As market researcher and Kellogg Professor Gina Fong states, "“Big data’s having a real moment right now. Numbers don’t lie—and big numbers sometimes give people more confidence in the data. But they only give you the what. Consumer anthropology and qualitative research are there to address the why behind people’s choices.”

Qualitative research can be most effective when we study people in their natural environment - retail stores, homes, workplaces, public parks, mass transit stations, etc. The key is to see what they do, rather than relying simply on what they say. After all, as Margaret Mead once said, "What people say, what people do, and what they say they do are entirely different things.”  Fong offers a wonderful example of how qualitative research can deepen our insights into consumer behavior and even prevent us from jumping to the wrong conclusions. Kellogg Insight describes her powerful example:

For example, a colleague of Fong’s loved eating avocados, but he had stopped purchasing them in the grocery store. Looking solely at patterns in the data, the store’s brand team might conclude that he and others like him didn’t like avocados anymore, or that avocados had become too expensive. Armed with this information, the team might react by discounting avocados or displaying them in another part of the store.  But his motivation for passing up avocados was more complicated.

“He really liked avocados, but he didn’t know how to pick one when it was perfectly ripe,” Fong says. “Either the avocado would ripen too fast and spoil, or it would take too long to ripen, and he would forget about it. This was frustrating to him, so he just punted the whole activity.”  On a recent store visit, he saw a display with avocados categorized in three groups: “ripe today,” “ripe in a couple of days,” and “ripe in four to five days.” This has led to him once again buying avocados.

Which Music Do We Prefer? Interesting New Research

https://freesvg.org/

We know that Spotify, Apple, and others have developed sophisticated algorithms to recommend music to listeners. While many customers know precisely what they would like to listen to, others are open to discovery. Spotify distinguishes between those listeners with a "closed" vs. "open" mindset. The algorithms are especially important for those with an open mindset. 

Now we have a new academic study that suggests an interesting way to predict the music we will enjoy. David Greenberg, Sandra Matz, Andrew Schwartz, and Kai Fricke have published a paper titled, "The Self-Congruity Effect of Music" in the Journal of Personality and Social Psychology.  They write: 

Across three studies we show that people prefer the music of artists who have publicly observable personalities (“personas”) similar to their own personality traits (the “self-congruity effect of music”)... Our findings are largely consistent across two methodological approaches to operationalizing an artist’s public personality: (a) the public personality as reported by the artist’s fans, and (b) the public personality as predicted by machine learning on the basis of the artist’s lyrics.

The scholars also show powerful evidence of a gender effect in music preferences.  They write:

The present article also provides the first evidence that listeners tend to prefer music from artists of the same sex, which had been previously theorized but not studied empirically (Hagen & Bryant, 2003). That is, the gender-fit between the listener and artist was found to be a significant predictor of musical preferences.

So, the next time you say yourself, "I love that artist's music."  Ask yourself, what do you know about that performer?  Why do you like that artist?   For companies, of course, the implications are profound.  Can they use this finding to refine their algorithms, without intruding on our personal privacy?  What if listeners were willing to disclose more about their personality in order to discover more music they would enjoy?   

Blending Big Data and Design Thinking Works Best

Bryant University Professor Lori Coakley and I have published a new article in the American Management Association Quarterly (Fall 2018 issue) titled, "The Human Center of Design Thinking."  In this short essay, we argue that many firms stumble because they focus on a technology in search of a problem.  Put another way, they wield a hammer in search of a nail... rather than seeking out unmet human needs, pain points, and frustrations that must be alleviated.  Moreover, managers sometimes think large datasets, derived from surveys and purchase histories, can provide them all the insight they need about customers.  They are sorely mistaken.   Finally, we explain why we sometimes empathize poorly with our customers, and we offer some tips on how to do so more effectively. 

Big Data and The Analytics Paradox

Kellogg School Professor Eric Anderson describes an interesting phenomenon that he calls the “Analytics Paradox.”  Anyone grappling with how their firm employs data science as part of their business strategy should become familiar with this concept.  Anderson explains:  

A young firm starts out making many mistakes. Eager to improve, they collect lots of data and build cool new models,” he says. “Over time, these models allow the young firm to find the best answers and implement these with great precision. The young firm becomes a mature firm that is great at analytics. Then one day the models stop working. Mistakes that fueled the models are now gone and the analytic models are starved."

Anderson offers an example of the analytics paradox.  Imagine that a firm provides two-day delivery services.    They consider using data analytics to help make an important decision: whether or not to offer one-day delivery services to customers.  You might be hard-pressed to answer that question using analytics.  Why? An effective organization becomes proficient at executing two-day delivery.  If the firm can't meet two-day delivery deadlines on a regular basis, processes and systems are changed.  Employees who can't meet the two-day delivery schedule get admonished or even dismissed.  In short, a firm that is very effective at execution will drive all variability out of its "production" system.  Yet, without variability, you will find it very difficult to use analytics to drive improved decision making. 

What can you do to conquer the analytics paradox?    Put simply, you need to inject variability into your system by promoting thoughtful and systematic experimentation.   The use of experiments enables you to test different models and systems, and in so doing, generate the type of data that can be analyzed effectively to enhance decision making.  

Avoiding Analysis Paralysis

University of Chicago Professor Sanjog Misra commented to the business school's magazine about the problem of analysis paralysis.  Professor Misra offered an interesting perspective on how to avoid this classic decision trap.  He teaches a class focused on the use of algorithms and data analysis in marketing.  Misra argued that you have to be very clear about the question you are trying to answer BEFORE you begin your analysis.   Moreover, he advocated for more clarity about the answer you are trying to achieve.  Misra recommended trying to sketch out the parameters of an idea answer before you start evaluating the data.  He explained:

This isn’t a totally new idea, just new to analytics. In the business world, one wouldn’t want to put out a call for proposals with no details about what he or she is looking for. We wouldn’t want to wade through a million proposals to decide what suits our needs. That would be silly. Instead, when you put out a call for proposals or a purchase order, you typically outline a very detailed specification of what you want. Similarly, it’s worthwhile speccing out the “answer” you are looking to find. You don’t go around aimlessly.  One of my interpretations of Peter Kennedy’s 10 commandments about data analysis is, “Thou Shall Not Fish.” That’s something I emphasize in my classes. Of course, sometimes mining for data is actually what’s required. So if the objective is to fish, then you should be fishing. If it isn’t, the commandment applies.

What's the "Optimal" Failure Rate at Netflix?

When Orange is the New Black, House of Cards, and Crown became mega-hits for Netflix, many people credited the analytics capabilities of the company. Mining the customer data had enabled the firm to project the type of original programming that would be highly successful. By this logic, Netflix would achieve a lower failure rate on new shows than the major television networks. After all, broadcasters such as CBS and NBC cancel a substantial share of their new shows each year, some after only a few episodes. 

On the recent Netflix earnings call, many investors were pleased to hear about strong subscriber growth at the firm. However, some investors came away concerned about the amount of spending taking place as the firm acquires or develops new content. Moreover, some observers and analysts have expressed concern about the recent cancellations of some new Netflix original shows. Tom Huddleston Jr. reported on the company's reaction to this criticism in a recent Fortune article:

Meanwhile, also on the Monday earnings call, Netflix's chief content officer Ted Sarandos defended the company's recent cancellations of a handful of expensive, but underperforming, original series. "The more shows we have, the more likely in absolute numbers that you’ll see cancellations, of course," Sarandos said. The executive compared Netflix's recent spate of cancellations—including big-budget series like The Get Down and Sense8—to traditional TV networks that cancel nearly one-third of their new shows after their first seasons. Netflix, he said, has renewed 93% of its original series. With respect to the shows that Netflix opted not to renew, Sarandos argued: "If you’re not failing, maybe you’re not trying hard enough."

This quote from Sarandos raises a fascinating question.  What is the "optimal" failure rate at Netflix?  Surely, we would like the failure rate to be lower than the broadcast networks.  We would like to see the company reaping the benefits of its analytics capabilities.  At the same time, no one should want Netflix's failure rate on original programming to be zero.  We want the firm to take some chances in hopes of landing some surprising breakthrough hits.  Hopefully, the firm isn't simply guessing or drawing on the intuition of the "creatives" in the business.  We would like to see them engaging in "enlightened" experimentation, using big data to guide them while still taking some risks.   If they balance data mining and risk-taking in an effective way, the failure rate won't be zero, but it will be much lower than their broadcast and cable competitors.  

Algorithm Aversion: How Can We Overcome It?

Algorithms can help us make better decisions in a variety of situations.  However, human beings tend to have an aversion to using algorithms.  They trust their gut more than the computer, even though the algorithms may lead to better decisions. Knowledge@Wharton reports on a stream of fascinating research by Cade Massey, Joseph Simmons, and Berkeley J. Dietvorst.  They found that you can persuade people to use algorithms if you give them a choice as to whether to use the algorithm or not.  In other words, don't force them to use it; make them feel a sense of control.  That will help convince them to choose the algorithm.  However, many people stop using the algorithm after some period of time, because they become frustrated with the mistakes that the computer makes.  Of course, the computer might make fewer mistakes than a human using intuition, but people don't recognize that possibility. Instead, they fixate on the mistakes and lose faith in the algorithm. Simmons points out, "People want algorithms to be perfect and expect them to be perfect, even though what we really want is for them to simply be a little better than the humans."

The scholars also found that you could persuade people to use the algorithms if you gave them an ability to adjust the computer's recommendation slightly.   Of course, the algorithm's predictions and recommendations become less accurate when humans intervene in this manner.  However, the researchers found that you only have to give people an ability to adjust the algorithm slightly to enhance adoption.   Providing them an ability to adjust more substantially does not increase adoption more than offering a slight adjustment possibility.  Thus, you might be willing to tolerate a bit of degradation in the algorithm's accuracy simply because giving people some sense of control increases adoption of the computer-assisted decision-making system.  For more on this research, see the video below in which Massey and Simmons are interviewed about the research.  

Combining Intuition and Analysis

We live in an age of big data.  People have become enamored with the use of analytics to solve complex problems and to make better decisions.  I'm a big believer in analytics, but I hope we don't forget that intuition does play an important role in decision-making processes.   In the video below, I explain briefly what intuition is and how it works.  

How can we combine intuition and analysis to help us make better choices?  Here are four ways that the two modes of thinking can complement one another.  

1. Use analysis to check your intuition, but not simply to justify decisions that have already been made 
2. Use intuition to validate and test the assumptions that underlie your analysis 
3. Use analysis to explore and evaluate intuitive doubts that emerge as your prepare to make a decision 
4. Use the intuition of outside experts to probe the validity of your analysis

Using Big Data to Create Better Presentations

Matt Abrahams has written a good article for Stanford Business School's Insights website. He describes the research conducted by Noah Zandan, founder and CEO of Quantified Communications. Zandan's company has used analytics to identify the key attributes of the most effective presentations by speakers and leaders of various kinds. Zandan's conclusions are based on the study of more than 100,000 presentations. Abraham's article is filled with great suggestions for improving your public speaking. Here are a few of the key points (excerpts below):

Avoid hedging language. Qualifying phrases such as “kind of” and hesitant language like “I think” can be beneficial in interpersonal communication, where they invite contribution and adjust your status relative to the person with whom you are conversing. But in contexts like presenting in public, they can reduce your credibility.

Vocal elements include volume, rate, and cadence. The keys to vocal elements are variation and fluency. Think of your voice like a wind instrument. You can make it louder, softer, faster, or slower. We are wired to pay attention to these kinds of vocal change, which is why it is so hard to listen to a monotonous speaker. In fact, even just a 10% increase in vocal variety can have a highly significant impact on your audience’s attention to and retention of your message.

Vital elements capture a speaker’s true nature — it is what some refer to as authenticity. For authenticity, Zandan’s team has found that the top 10% of authentic speakers were considered to be 1.3 times more trustworthy and 1.3 times more persuasive than the average communicator. Authenticity is made up of the passion and warmth that people have when presenting.  Passion comes from exuding energy and enthusiasm.

Netflix: Geography, Age, Gender Not Good Predictors

David Morris wrote an article this week for Fortune titled, "Netflix: Geography, Age, and Gender are 'Garbage' for Predicting Taste." Morris writes, 

"'Geography, age, and gender? We put that in the garbage heap,' VP of product Todd Yellin said. Instead, viewers are grouped into “clusters” almost exclusively by common taste, and their Netflix homepages highlight the relatively small slice of content that matches their taste profile. Those profiles could be the same for someone in New Orleans as someone in New Delhi (though they would likely have access to very different libraries)."

Why is this statement so fascinating?  To me, it speaks directly to Netflix's competitive advantage.  If geography, age, and gender were, in fact, accurate predictors of viewers' preferences, then Netflix would have a far less formidable advantage over rivals.  Why?  Well, those variables are easy to identify and measure.  Others can get access to that data quite easily and build predictive algorithms using that information.  However, if more accurate predictive algorithms involve data that are not as publicly available, then Netflix has a key advantage.  In other words, if the predictive power rests with variables that come from proprietary data that Netflix has collected about us, then the sustainability of Netflix's advantage over competitors rises substantially.  The same holds true for any company trying to take advantage of "big data" to develop predictive algorithms.   The key is to unearth variables that matter, but hopefully, variables that are not easily identified and measured by others.  

Big Data in Formula 1 Racing: Don't Overwhelm Humans

Fortune has a fascinating article about how Formula 1 teams are using the internet of things and data analytics to win auto races.   According to the article, 

"These machines, each valued at more than $9 million (a steering wheel alone is worth $77,000 or so) are more than just pricey contraptions capable of whizzing around the track at more than 200 miles per hour. They are also intelligent, thanks to the many dozens of sensors fastened to them. Each sensor communicates with the track, the crew in the pit, a broadcast crew on-site, and a second team of engineers back home in Europe."  

The team then uses predictive algorithms to help them understand how the car will perform under certain track conditions.  These data guide key decisions.  However, they are careful not to put too much on the driver's plate.  After all, he or she is concentrating on many factors while driving at a very high rate of speed.   The team doesn't want to overwhelm the "cognitive capacity" of the driver.  In other words, they have to boil all that data down to a few key items about which they want to make the driver aware.  

That description sounds quite similar to how a great football coach operates.  They conduct extensive analysis of the opponent, breaking down game film and evaluating data about the strengths and weaknesses of that team.  The coaches then build game plan.  However, they have to keep the ultimate plan simple enough so that players can make fast decisions on the field.  They want them to still act instinctively and not be overwhelmed by too much information.   Managers in all types of enterprises should take note.   We want data analysis to guide people's decisions, but we have to keep in mind the cognitive capacity of those individuals.  We have to be able to boil down all that data to a few key principles and recommendations that they can implement effectively and quickly. 

Beyond Big Data

Here's a terrific short video from the University of Chicago's Booth School of Business.  The video explores the latest trends in consumer research.

Big Data Will Not Solve All Our Problems, May Mislead Us

Microsoft Research's Kate Crawford has a terrific blog for HBR about big data.  In the post, she discusses the hype regarding big data, and she talks about the hidden biases that we must be aware of when analyzing large data sets:

Data and data sets are not objective; they are creations of human design. We give numbers their voice, draw inferences from them, and define their meaning through our interpretations. Hidden biases in both the collection and analysis stages present considerable risks, and are as important to the big-data equation as the numbers themselves. 

Crawford has some terrific examples of biases in data sets.  For instance, she talks about how the Twitter data after Hurricane Sandy offers a distorted view of the storm.  Why?  As the storm progressed, people in the hardest hit areas ran out of battery power on their cellphones.  Thus, they stopped tweeting.  Folks in Manhattan, where the storm was significant, but not as devastating, engaged in much more Twitter activity.  Moreover, people in the lowest income groups are not as well represented on Twitter, because many do not own smartphones.  As she writes, "We can think of this as a "signal problem": Data are assumed to accurately reflect the social world, but there are significant gaps, with little or no signal coming from particular communities."

The lesson is clear.  Begin your big data project by asking:  How was the data collected?  What populations are overrepresented?  What populations are underrepresented?  Beyond that, you should ask:  Who collected and assembled the data set?  Do they have an agenda?  Are they biased in any way?  Often, the biggest bias in big data has nothing to do with access to technology or underrepresented populations.  Instead, the most significant bias lies inside the mind of the person assembling the data.  Their agenda clouds the process of data collection. 

Disney MagicBands: This Could be Magical... and Expensive!

Source: Fast Company

Disney has used RFID technology to develop MagicBands - a wristband that serve as a guest's theme park ticket, hotel key, and payment system throughout DisneyWorld.   When I read about this project at Fast Company's website, I thought to myself... "These MagicBands could be REALLY expensive for parents."  It's going to make spending money that much easier.  I speak from firsthand experience.  We just stayed at DisneyWorld's Contemporary Resort a few weeks ago.   The cards you receive upon check-in serve as hotel room keys and park tickets, and they can be used to pay for food and gifts throughout the parks.  I found myself a bit less hesitant about spending money with that card in hand, as opposed to having to pull cash out each time we bought something.  However, as Fast Company writer Mark Wilson points out, these new MagicBands offer more than a way to dig deeper into customer wallets.  Here's an excerpt:

But the cashflow aspect is only one aspect of Disney’s new service. When you begin to consider the potential of wearing a wireless ID around your wrist, all sorts of natural, customized interactions will become possible. Imagine a child meeting Mickey Mouse, and after sharing a warm hug, Mickey actually wishing them a happy birthday by name. There’s a digital handshake going on here, of course, but it’s totally imperceptible. All a child is left wondering is, “How did Mickey know me … and that it was my birthday?!?” Animatronics will see a similar personalization, so the otherwise obtuse talking robots can specifically acknowledge the people standing in front of them.  At the same time, MagicBands enable a deeper level of data collection for Disney. They’ll be able to track someone through the entire park--to see their kingdom as a complex interaction model--finding trends in preferences and habits that can no doubt be monetized. Do people who meet Cinderella buy more princess apparel? Do those who eat the cheese fries for lunch go back to the hotel to take naps?

Talk about Big Data and Personalization all wrapped up into one incredible initiative!   Immediately, one thinks about privacy issues, but Wilson points out that parents will be able to establish some privacy controls on the wristbands.   The bigger issue is how Disney uses the wristbands.  I think that the real opportunity here lies in focusing first on enhancing the guest experience.  If this new tool can make the guests enjoy their stay even more, then the money will follow.  People will spend more if they are more engaged and more satisfied with the experience, and if the wristbands can be used to minimize some of the guest's typical problems at the park (consider how mobile apps have helped people find rides with the shortest wait times).   If the experience is enhanced, guests will stay longer at the park, interact with characters and exhibits in a more meaningful way, and avoid some of the frustrations that they typically encounter at the park.  When that occurs, the improved profits will come.

How Analytics Can Help You Improve Quality and Reduce Costs

I found a terrific example of the use of "Big Data" in Fast Company magazine this month.   The article by Farhad Manjoo describes a situation at Washington Hospital Center.   ER doctors became concerned that many patients returned to the hospital just a short time after being discharged.  A computer scientist at Microsoft Research began to investigate.  He wanted to identify some triggers that would predict whether a patient would be readmitted.  Specifically, he was looking to help doctors identify some predictors that might not otherwise receive much attention by ER physicians and nurses.  He analyzed more than 300,000 ER visits.    Among other things, he discovered that the length of a patient's stay in ER tended to be a good predictor of readmission.  If a patient stayed in the ER for more than 14 hours, they were likely to return to the hospital within a few weeks.  Similarly, if the patient's chart mentioned the word "fluid" at some point, that seemed to predict readmission quite well too.  

This story illustrates how companies can use analytics to help them understand how to improve the quality of customer service, as well as to reduce costs.  Take an automobile dealer.   They conduct repair and maintenance on thousands of cars per year.  A fair number of those cars return shortly after a repair or maintenance appointment, because something is not working correctly or hasn't been done to the customer's satisfaction.   An automobile dealer could analyze the data from thousands of those cases, and it could try to identify the predictors of return visits.  If they could identify a few solid predictors, then they could try to intervene to reduce those return visits.  Those interventions could improve quality and customer satisfaction, while reduce costs (since every return visit is costly).   Many service businesses could apply a similar logic and use analytics to achieve positive results.   Can your company benefit from such an approach?  

Big Data, Diapers.com, and the Importance of Analytics

Several days ago, the New York Times published an article titled, "The Age of Big Data."  The newspaper described how companies will need many more data analysts who can "help businesses make sense of an explosion of data — Web traffic and social network comments, as well as software and sensors that monitor shipments, suppliers and customers — to guide decisions, trim costs and lift sales."   The article cited a McKinsey Consulting study which predicted that the United States will need 140,000-190,000 more employees with “deep analytical” expertise" in the coming years. 

As an example of the importance of big data, consider the online retailer Diapers.com (owned by Amazon).   Forbes writer Meghan Casserly describes the firm's use of big data in an article published on the magazine's website.  The company has built powerful proprietary algorithms over the past few years based on tons of transactions.  These algorithms predict what customers are likely to buy in the future, how much they will spend, and whether they will be profitable for the firm.  The company's strategy focuses on building loyal customers who purchase low margin baby supplies initially, and then buy higher margin items such as car seats, strollers, and the like in the future.  The algorithms not only help predict purchasing patterns, but they enable Diapers.com and its sister sites to market appropriately to different customers.  Perhaps most importantly, the firm can identify which customers will be profitable for the firm.  Thus, they can spend their time catering to the most profitable customers, rather than wasting marketing expenditures on consumers who will be a drain on resources. 

Every company should be thinking about how it can use algorithms to drive performance.  Analytics can be used in a myriad of ways.   However, building a strategy based on big data requires the right talent.  Therefore, firms need to begin thinking carefully about how they will attract, develop, and retain the talent needed to collect and analyze the huge volumes of data that now exist.  Universities need to think about how to educate people for these roles, as demand will be strong.  We need to do more than educate people in mathematics and statistics though.  We need analysts who can understand business models and strategies, and who have a deep understanding of consumer behavior too.   The best analysts will be those who can marry statistical knowledge with a broader understanding of the entire organizational system.