Showing posts with label experimentation. Show all posts
Showing posts with label experimentation. Show all posts

Monday, April 14, 2025

Can AI Enable Us To Focus More on the Work We Love to Do?

Source:  Doyenhub Software Solution

Most of the attention on AI these days seems to focus on how it can make us more productive, as well as how it may even displace some workers whose tasks can now be automated.   However, a new study examines a slightly different question.  It explores whether AI can change the nature of work.  Might we engage in different activities as a result of the impact of AI, and might we even have more time to do what we truly love to do? 

Manuel Hoffmann and his colleagues have published a new working paper titled "Generative AI and the Nature of Work."  They examined the results of a natural experiment associated with the use of GitHub Copilot, a generative AI tool for software developers.  The findings illustrate an interesting shift in the work that developers were doing.  First, the generative AI tool enabled software developers to spend more time coding and less time on administrative tasks related to project management.  Second, they found that developers engaged in more exploratory work with the introduction of this AI tool.  In other words, developers conducted more experiments and spent less time on established projects.  The scholars summarize the key results as follows:

Copliot eligible developers engage with an additional 15 new repositories on average relative to ineligible peers. Beyond simply interacting with a new set of repositories, we also find evidence that generative AI enables developers to gain exposure to a wider range of technologies. In Panel B, we can see that eligible developers increase their cumulative exposure to new programming languages by 21.79% relative to the baseline. We also estimate a version of this cumulative programming language exposure measure weighted by the median salary reported by software developers who use that language.21 Access to Copilot induces developers to experiment with programming languages that command a 1.41% higher salary relative to a baseline of $119,371 (an increase of $1,683).

I found this paper to be quite thought provoking.  It makes perfect sense to me.  If AI is making us more productive, then what are doing with that new time have on our hands? Are we just doing more work in a given period of time, or are we sometimes using that "new time on our hands" to engage in innovation work?  Are we learning new skills, trying out new ideas, creating new products and services, and inventing better work processes?   These outcomes may not only be beneficial for the organization, but rewarding and fulfilling for us personally.  


Monday, June 10, 2024

Companies Learning from Their Histories


Fortune's Phil Wahba has written an excellent article titled "From Tide Pods to Coach bags, how Fortune 500 companies use museums of their hits and misses to drive success."  He documents how business leaders have developed company museums and assigned individuals to serve as corporate historians.  Many firms derive great benefit from these efforts to preserve and highlight important facets of their histories.  What are some of the key uses of these company museums?

1.  The museums keep track of substantial failures, enabling the the firms to heed the lessons of those setbacks in the future.  Moreover, sometimes companies can resurrect failed projects, find new uses for old technology, and simply find that the time is now right for something that may have been ahead of its time.  Documenting and highlighting failures, and not just successes, also sends an important message regarding the culture.  Employees come to understand that intelligent failures are acceptable, and even encouraged, because they represent the type of experimentation that can lead to breakthrough innovation. 

2.  The museums enable product developers to tap into past designs for inspiration, as the Wahba article illustrates by pointing out that designers at Coach enjoy looking back at the bags that were fashion hits in previous decades.  

3.  Executives can dig into the artifacts and records to examine how leaders addressed similar challenges in the past.  Wahba writes that Coca-Cola executives dug into records from the 1918 pandemic when the COVID-19 virus swept across the globe in 2020.  They sought to understand how the company responded then, and what lessons might be applicable in the 21st century.  

4.  Perhaps most importantly, these museums enable companies to highlight the values that they hope will endure at the company.  What aspects of the company culture do they want to highlight for current employees?  How can they demonstrate the company's commitment to making life better for customers, and not just producing profits?  The museums have a role in telling the story of the founders and giving employees a sense of the impact that the organization has made on people's lives.  

In short, history matters.  Companies have much to learn from their past, and investing in telling the story of past success and failure can be incredibly valuable.   It's so important to examine the good and the bad, because people learn very effectively when they can compare and contrast success and failure.  

Tuesday, June 01, 2021

Sustaining a Culture of Experimentation After the Pandemic Subsides

Source: Fortune.com 

Did your company shift and adapt in an agile manner during the early months of the pandemic?  Were you impressed with how the organization was willing to try new things? Did this period of time mark a break from past situations all too often characterized by "analysis paralysis" and slow, bureaucratic planning processes?   Many organizations can answer "yes" emphatically to all three questions.  Now the challenge is to maintain this culture of experimentation and agility as the pandemic subsides.  That will be difficult for many firms.  Some will backslide into past practices.  In an article for Fortune, Mark Hoplamazian,  CEO of Hyatt, argues that we can't go back to business as usual.  We have to continue to embrace the culture of experimentation that has been essential for firms to survive and even thrive during the pandemic.  He writes: 

As the COVID-19 pandemic evolves into a chronic yet more manageable state, business leaders might be tempted to return to the comfort of business as usual. That would be a mistake.

While some organizations were stopped in their tracks by the onset of the pandemic, those that welcomed experimentation were a step ahead in their ability to manage through it. Accustomed to being agile and listening to their stakeholders, they had a kind of strategic muscle memory that allowed them to smoothly embrace new ideas, behaviors, and practices.

But here’s the tricky thing about experimentation: It means being comfortable with some experiments not working out. Continuously learning and modifying is a critical piece of the puzzle.

It’s no secret that the pandemic brought the travel and hospitality industries to a screeching halt, and it’s my firm belief that Hyatt’s focus on listening, testing, learning, and adapting is what led us through the pandemic and put us on a clear path to recovery.

Saturday, August 05, 2017

Failure Has Become Fashionable

Everyone is talking about becoming more tolerant of failure these days.  You read it everywhere.  Failures are really learning opportunities.   You learn more from failure than success.  You have to be willing to fail if you wish to innovate.  Experimentation entails some failures; without them, you won't create anything truly bold.  You have to make it ok to fail.  

Is it really true? Should failure be acceptable in your organization?  HBS Professor Amy Edmondson has noted that there is a spectrum of reasons for failure.  Some failures are truly preventable.  They result from people knowingly deviating from procedures or not paying close enough attention to specifications.  These types of failures are blameworthy events.  They are not acceptable.  At the other end of the spectrum are intelligent failures.  These failures result from the experimentation process.   In these situations, people are trying to learn through hypothesis testing or exploratory experiments.   These types of failures, according to Edmondson, are praiseworthy events.  

Are all experiments praiseworthy?  Of course not.  Some experiments are well-designed and carefully  implemented.  Others are hastily arranged and not well-designed.  You want to encourage people to make smart bets.  They should be engaging in enlightened, disciplined trial and error in the iterative process.  They shouldn't just be haphazardly trying new things.  Failures that emerge from a well-designed experiment are praiseworthy events.  We should not send the message, though, that all testing is good testing.  We should encourage people to put some thought into how they execute tests, experiments, and prototypes.  Are they designed in such a way to elicit highly useful user feedback?  Was data collected in an unbiased way, or were people simply trying to confirm what they already believed?  Did people pick the right sample, and were appropriate controls chosen if a hypothesis was  being tested?  These types of questions need to be asked when people fail during the iterative process.   We want to tolerate certain types of failure because we want to maximize learning. However, poorly designed tests and experiments do not lead to effective learning.  

Tuesday, July 18, 2017

What's the "Optimal" Failure Rate at Netflix?

When Orange is the New Black, House of Cards, and Crown became mega-hits for Netflix, many people credited the analytics capabilities of the company. Mining the customer data had enabled the firm to project the type of original programming that would be highly successful. By this logic, Netflix would achieve a lower failure rate on new shows than the major television networks. After all, broadcasters such as CBS and NBC cancel a substantial share of their new shows each year, some after only a few episodes. 

On the recent Netflix earnings call, many investors were pleased to hear about strong subscriber growth at the firm. However, some investors came away concerned about the amount of spending taking place as the firm acquires or develops new content. Moreover, some observers and analysts have expressed concern about the recent cancellations of some new Netflix original shows. Tom Huddleston Jr. reported on the company's reaction to this criticism in a recent Fortune article:

Meanwhile, also on the Monday earnings call, Netflix's chief content officer Ted Sarandos defended the company's recent cancellations of a handful of expensive, but underperforming, original series. "The more shows we have, the more likely in absolute numbers that you’ll see cancellations, of course," Sarandos said. The executive compared Netflix's recent spate of cancellations—including big-budget series like The Get Down and Sense8—to traditional TV networks that cancel nearly one-third of their new shows after their first seasons. Netflix, he said, has renewed 93% of its original series. With respect to the shows that Netflix opted not to renew, Sarandos argued: "If you’re not failing, maybe you’re not trying hard enough."

This quote from Sarandos raises a fascinating question.  What is the "optimal" failure rate at Netflix?  Surely, we would like the failure rate to be lower than the broadcast networks.  We would like to see the company reaping the benefits of its analytics capabilities.  At the same time, no one should want Netflix's failure rate on original programming to be zero.  We want the firm to take some chances in hopes of landing some surprising breakthrough hits.  Hopefully, the firm isn't simply guessing or drawing on the intuition of the "creatives" in the business.  We would like to see them engaging in "enlightened" experimentation, using big data to guide them while still taking some risks.   If they balance data mining and risk-taking in an effective way, the failure rate won't be zero, but it will be much lower than their broadcast and cable competitors.  

Friday, June 26, 2015

The (Not-So) Hidden Value of Pixar's Short Films

The Wall Street Journal reports this week on Pixar's purpose in making short films.  For instance, Lava is the short film that runs before the newest Pixar feature film, Inside Out.    The short films are a wonderful opportunity for experimentation.   As a result, they help fuel the creative process at the highly successful film studio.  Here's one excerpt from the article:  

[Making short films] is a wonderful opportunity to step out and do something you may not be able to do on a feature, really take some chances,” said Mr. Murphy, a senior animator at the studio. “I think each short really represents the tastes of the individual that directed them.” 

The article highlights one additional value from making these short films.   It enables Pixar to develop and evaluate its talent.  Providing young directors and animators an opportunity to create a short film is a relatively inexpensive way to see who might be ready to work on a feature film.  Moreover, it enables Pixar to grow and develop those high potentials.   They learn by doing in an environment in which they can take some risks.  Every firm should be looking for opportunities to experiment as a means of fostering innovation, but they should also be examining how these experiments can be a useful means of nurturing and testing younger talent. 

Monday, September 08, 2014

Fast Company article: Embracing Failure

Rachel Gillett has written an article for Fast Company titled, "What The Hype Behind Embracing Failure Is Really All About."  She has included several of my comments. The article examines the popularity of the concept of "tolerating failure" - is it overdone? Is it a fad?  What's the true meaning of the concept of embracing failure?  Here is the link to the article. 

Friday, November 01, 2013

Intuit Founder Scott Cook: Experimentation vs. Powerpoint

In a Fast Company interview, Intuit founder Scott Cook talks about a key lesson he learned about decision-making processes.   He regrets that too many decisions are made through a series of meetings, where people present their ideas to folks at higher levels in the organizational hierarchy.  Multiple reviews take place, as senior people scrutinize the proposals generated from below.  Powerpoint dominates these meetings.  Yet, despite all this review and critique, many poor decisions get made.  Cook advocates for a shift from decision-making by persuasion and Powerpoint to decision-making by experimentation.  Here's an excerpt:

There's a pattern here. Both companies make much better decisions because they don't rely on hierarchy, PowerPoint, persuasion. They're making decisions based on real experiments. So I said, wait a minute. Whenever reasonable, let's move from decisions by persuasion to decisions by experiment.  Three things happen. One, you make better decisions because it's actually real consumers or real production methods that aren't based on theory or a PowerPoint. It's based on real results. That's one.  Two, you enable your most junior people to test their best ideas, and when in you're doing PowerPoint presentations, whose ideas are most likely to get lost?  The third is, you get surprises more often, and surprises are a key source of innovation. You only get a surprise when you are trying something and the result is different than you expected, so the sooner you run the experiment, the sooner you are likely to find a surprise, and the surprise is the market speaking to you, telling you something you didn't know.

Cook does a very nice job here of explaining three key virtues of a shift toward experimentation.  I often advise executives that experiments can be a very effective tool for resolving key differences of opinion.  I encourage leaders to stimulate constructive debate.  We make better decisions if we encourage dissenting views.  However, sometimes we then find ourselves in a difficult spot.  An impasse emerges, and it's not clear how to resolve the differences of opinion that have emerged.  In those instances, how can we manage the conflict constructively?  One way to do so is through experimentation.  Rather than continuing to hold meetings and debate the issue, we can shift toward experimentation. Ask yourself: What are some of the key assumptions and hypotheses at the core of the arguments being made?  What could we do to test some of the key assumptions and hypotheses that have been put forward here?  Experiments, then, can become a way to manage conflict constructively and advance a decision-making process that is stuck in a stalemate.  

Tuesday, July 30, 2013

A Culture of Experimentation at Intuit

If you are interested in moving your organization toward a more effective, lower cost method of developing new products and services, take a look at this terrific presentation by Kaaren Hanson, VP of Design Innovation at Intuit. 

Monday, July 01, 2013

Starbucks: Handcrafted Sodas?

The Wall Street Journal reports that Starbucks is testing handcrafted sodas at locations in Atlanta, Georgia, and Austin, Texas.   The article states that the flavors of soda include spiced root beer and lemon ale.  The sodas are created using a carbonation machine in the Starbucks stores, and a grande size drink sells for $2.95 at these locations.

What do we make of this newest product line extension for Starbucks?  As the article explains, the move clearly constitutes another attempt to drive sales at off-hours, i.e. in the afternoon and evening.  Moroever, it offers another way to drive same-store sales growth, a key metric for any restaurant or retailer.   

The move does come with some risks.  Naturally, customers must enjoy the product, or it could harm the Starbucks brand.  In addition, continued menu extensions add complexity to the operations of a particular location.  Can Starbucks maintain speed and customer service as the menu expands?   With food items expanding as well, Starbucks must watch the issue of complexity carefully.  The good news:  Starbucks understands the power of experimentation.  It is testing the concept in two locations, and presumably, it's gathering a great deal of data.  That data include customer feedback, barista input, and information about the impact that the new products are having on the speed of service.   What happens if customers love the soda, but service slows?  That could be tricky.  At that point, Starbucks will work to streamline operations as much as possible to increase throughput.   They have done quite a bit of that type of work in the past few years.  Still, I notice considerable differences in speed across locations.  Part of the difference can be accounted for by the fact that some store layouts are clearly not as conducive to streamlined, efficient operations.   Part of that is due to local differences in the way locations are managed.  

One final point about the notion of testing and experimentation.  Sometimes, companies find that such small tests go quite well, but national roll-outs then falter.  Why?  The tests are not truly representative of what will happen during a full-scale expansion.  Senior managers supervise the tests very carefully. Extra resources are deployed.  The test becomes something more than a test... it's actually a "proving ground" or "demonstration" rather than a true controlled experiment.  

Friday, October 12, 2012

How Big Companies Act Like Startups

Soren Kaplan wrote a terrific article recently for Fast Company.   Kaplan explained how some large companies try to act more like startups in an effort to jumpstart innovation.  He outlines four key practices/principles that invigorate innovation:

1. Follow customers home:  In other words, conduct anthropological research.  Go into customer homes and observe them in their natural settings using various products and services.

2.  Tap outside collaborators:  Bring in experts and innovators from outside the firm to bring fresh perspectives, explain their research, and educate folks about how things are done elsewhere.

3.  Stay small:  Don't think in terms of bet-the-company type projects.  Think instead in terms of low cost, rapid experimentation.

4.  Use the best, invent the rest:  Creating a new product or service doesn't mean inventing the entire thing from scratch.  Often, you can combine off-the-shelf components with proprietary innovations to develop a unique new product.

Friday, June 15, 2012

Is YouTube An Oppportunity For Terrific Advertising Experiments?

Alex Konrad has written an article for Fortune titled, "Pepsi, Brewing up viral magic."   The article describes how Pepsi debuted a new "Uncle Drew" ad for Pepsi Max on YouTube.   According to the article, "In the five-minute clip, 'Uncle Drew' amuses, then mesmerizes, a pick-up basketball game and its fans with crossovers and dunks unbecoming of a white-bearded, paunch-carrying old man, and only possible because the true identity of 'Uncle Drew' was a carefully disguised young basketball star, the clip's writer-director Kyrie Irving."  (Irving is the former Duke player and current NBA Rookie of the Year who plays for the Cleveland Cavaliers).   What's interesting is where the ad went next... it is now appearing on television as a 30-second spot during the first few games of the NBA Finals between the Miami Heat and the Oklahoma City Thunder (go Thunder!). 

That progression is somewhat unique, going from YouTube to television.   I find it very interesting though, and I think more firms should emulate this strategy.  This story proves that firms can and should use YouTube not just as part of a social media marketing strategy... They should think of YouTube as a land of experimentation.   YouTube offers an inexpensive way to experiment with new ad strategies.  The cost of failure is minimal, and even the ads that don't become viral sensations can be "useful failures" in that they may provide powerful learning opportunities.  

If firms are to use YouTube as a powerful mechanism for low cost, low risk, fast experimentation, then they need to have clear methods of evaluating these experiments.  Konrad's article explains that Pepsi had just such a method of evaluation:

"Pepsi Max brand team member Sam Duboff, who led creation and development of the piece, says 'Uncle Drew' had to satisfy three major metrics in order to justify its adaptation into a television segment. 'Uncle Drew' had to keep viewers engaged, hit the brand's target demographic, and generate its own legs through word of mouth. With 80% of viewers watching through the 4-minute mark, a 82.1% male viewer group that skewed towards the brand's core 25-44 age group, and over 5 million views from embedded YouTube players suggesting the viewer watched over a media site or Facebook, Duboff and his team hit all three."