Testing Your Assumptions

You've used AI and/or Excel to build a beautiful model that examines the ROI of a new venture, project, or initiative.  The result? A strong case for investment, because the ROI is quite good.  You are ready to make your case to senior executives and/or investors.  How can you stress test your model?  The most important thing you can do is to test your assumptions. After all, garbage in, garbage out. If you build a model based on false assumptions, you are in big trouble.  Of course, confirmation bias afflicts many of us as we make decisions. We may be picking and choosing assumptions to deliver a rosy picture, rather than making more realistic presumptions about the future.  

How do we decide which assumptions warrant the most attention and should be tested vigorously?  Jon Fjeld was a long-time tech industry executive, and he now teaches at Duke's Fuqua School of Business.  He has a simple method for determining which assumptions need to be scrutinized first.  He argues that three factors are critical:

1. Severity:  How big is the impact on the project if the assumption is not true?

2. Probability: How likely is it that the assumption is not true?

3. Cost:  How expensive and time consuming is it to test the assumption?

Fjeld uses these three factors to create a simple ratio that can be used to rank your assumptions.  His equation is: (Severity x Probability)/Cost.   The higher the ratio, the more important it is that your prioritize the testing and validation of that assumption.  While we don't actually have a clear way to quantify these three factors, the concept of this ratio makes good sense.  By thinking about these three factors, and their relationship to one another, we can do a better job of deciding how we want to test, experiment, and prototype before making a big bet.  

A Stubborn Attachment to Existing Beliefs

Source: Rand Corporation

In 1962, Roberta Wohlstetter published a fascinating book titled, "Pearl Harbor: Warning and Decision."   She examined why officials did not perceive the threat of a potential attack in Hawaii appropriately.   She opened her book by writing, "It would be reassuring to believe that Pearl Harbor was just a colossal and extraordinary blunder.  What is disquieting is that it was a supremely ordinary blunder." Later, in assessing the decision making of key leaders, she notes how many of them held a strong pre-existing belief that the Japanese would never attack the United States. That belief clouded their assessment of data and signals, and ultimately led to misguided judgments. She wrote, "Apparently human beings have a stubborn attachment to old beliefs and an equally stubborn resistance to new material that will upset them."  Truer words have never been written. 

I'm reminded of this quote when I think about an organization I've been studying for quite some time.  This once very successful organization made a series of strategic decisions more than a decade ago based on a core set of beliefs and assumptions.   Those beliefs and assumptions were consistent with the strong preferences of senior leaders - i.e., they aligned quite well with the interests and aspirations of top managers and board members.  Years later, quite a bit of evidence suggests that some of those assumptions were invalid.  Yet, leaders remain stubbornly attached to those beliefs.   They haven't been willing to confront the data, question those beliefs, and change course.  

Poor governance practices have exacerbated the problem, as board members prefer not to question the assumptions or the strategy, given that some of the initiatives were "pet projects" in which they were emotionally invested.  Less than desirable outcomes and results are constantly rationalized away, much like a retailer that explains away its poor financial results during a particular quarter by pointing to "weather conditions" that reduced traffic to its stores.  Funny how quarterly financial reports never seem to talk about favorable weather conditions bolstering sales and profits!  

Perhaps the most frustrating element of this story is that the stubborn attachment to existing beliefs has not only led to an unwillingness to change direction or abandon misguided plans, but has led to a "doubling down" on failed policies.  The sunk cost trap is clearly at play.   The organization finds itself throwing good money (and effort) after bad.  The problem, as is often the case, is not simply the prior investment of financial resources.  It's the huge emotional sunk cost; it's the unwillingness to admit when one is wrong.  

Whenever you witness the sunk cost trap in action, you should not only pay attention to the wasted resources as good money is thrown after bad.  One should also pay close attention to the opportunity costs. What opportunities have been missed and what investments have NOT been made because so many resources continue to be allocated based on a stubborn attachment to long-held beliefs and assumptions?  In this particular organization I've studied, the opportunity costs in many ways have been far more substantial than the actual expenditures wasted in support of invalid beliefs and assumptions.  

Has your organization found itself in this predicament?  How can you as a leader help to identify implicit and explicit assumptions and beliefs that need to be tested, challenged, and validated?  How can you protect against the stubborn attachment that Wohlstetter wrote about so eloquently nearly sixty years ago?

Declare Your Unstated/Hidden Assumptions

Source: Wikimedia

Managers many a number of crucial assumptions when they make strategic decisions or craft strategic plans.   No strategy rests simply on a body of facts or evidence.  Ambiguity and unpredictability about the future always reigns, and therefore, managers must build their plans based on certain assumptions. Of course, we often don't identify these assumptions explicitly.  They remain hidden.  Declaring your assumptions enables them to be discussed and debated.  They can be tested and probed critically.  Otherwise, managers may take certain assumptions for granted, treating them as truth, when they may actually be quite shaky.   Several years, Mark Hollingworth wrote a good article for Ivey Business Journal, in which he explained the six primary benefits of including a statement of your assumptions any strategic plan: 

1. Inclusion facilitates the analysis of any organization’s business plan by a financial institution, venture capitalist or angel investor. The risk of making a bad investment will be reduced if the investors understand and share the strategic assumptions of the organization’s management team.
2. Differences in points-of-view about strategic assumptions are the source of many of the conflicts that arise between investors and company management – and within a management team itself. Strategic assumptions represent the shared values, beliefs and vision of the management team. Demanding that they be included in a strategic plan will force management teams to hold the difficult internal conversations required and that allow them to uncover, challenge, and capture their shared assumptions.
3. Knowing they need to exit a strategic planning process with a complete, shared set of strategic assumptions forces a management team to use a much more rigorous strategic planning process.
4. Face-to-face, it is very difficult for most people to defend strategic assumptions which are ungrounded or that they do not believe or share.
5. Developing and debating strategic assumptions with groups of employees is an excellent way to gain buy-in and commitment to the organization. Having to declare and justify the assumptions upon which a plan is built means that it is difficult for a CEO to impose his or her views. With increased levels of employee buy-in, there is a greater probability that the strategic plan will actually be implemented.
6. By presenting strategic assumptions for rigorous debate and analysis, the probability is minimized that investors, employees, management and any other stakeholders will waste time, money and energy on trying to implement plans that have little chance of generating the promised results.

Not All Failure is Smart Failure

Source:  Flckr

Several months ago, long-time executive and now Duke Professor Jon Fjeld published an article for Sloan Management Review titled, "How to Test Your Assumptions."   He begins the article by questioning the "fail fast" approach embraced by many entrepreneurs in recent years.   He notes that many startups do not understand the lean startup philosophy, and they are implementing it in a haphazard and ineffective way.  He writes: 

The enthusiasm surrounding the “lean startup methodology” and its many offshoots has created a mindset that entrepreneurs should just launch, failing early and often — iterating, to use startup parlance. But failure alone does not teach. If there are an infinite number of bad ideas, eliminating one gets us no closer to a good idea. Rather, the businessperson contemplating a new venture must begin by evaluating factors that have to be true for the venture to succeed. He or she also must model these factors in a way that allows for reasonable testing. For example, the assumption that people will buy a product for the asking price is a big one; it would take a full launch to completely validate this. Therefore, the entrepreneur must split big assumptions into discrete, manageable assumptions that can be tested at a level of detail allowing for efficient learning.

He makes a terrific point.  Not all failure is instructive.  At times, you can fail, but find yourself unable to distinguish among many causal factors to which that failure can be attributed.  In fact, these types of failures can be enormously costly, because individuals can derive incorrect attributions easiliy, and then make the wrong kinds of changes in hopes of averting failure in the future.  Failing fast works best when you can test your ideas in ways that enable you to identify what went wrong and WHY it went wrong.   Sometimes, two or three possible causes should be considered, and another test may be conducted to determine which of these factors led to the failure.  

Clinging to Old Mistaken Assumptions

Why do leaders cling to assumptions that are outdated, and/or long since proven untrue by reams of data?   Many reasons exist for this cognitive trap.  However, one common reason is that flawed decisions and strategies have resulted from those mistaken assumptions.  Leaders don't want to acknowledge those failures, and thus, they somehow convince themselves that the assumptions are actually true despite the evidence to the contrary.   They rationalize away the data that demonstrates the failed outcomes of strategic choices that have been made.  

Of course, leaders also confuse facts with assumptions all the time.  Consider the following statement:  We need X units of volume to cover our fixed costs.  That seems like quite a reasonable statement, presumably grounded in the data about revenues, contribution margin, and the like.  However, the statement contains a powerful hidden assumption.  It presumes that those fixed costs are set in stone, forever unchanged.  What if the organization could reduce those fixed costs over the longer term?  Of course, that would change the breakeven volume level, perhaps by a considerable degree.  Asking the what if question is critical.  It often opens up new possibilities.  

Of course, asking the what if questions about critical assumptions can be threatening to leaders.  Consider the example above once again.  Asking the what if question might mean initiating a dialogue about a pet project or favorite strategic initiative endorsed or perhaps even created by the leader.   Eliminating that project or initiative might reduce fixed costs considerably, and change the breakeven level.  However, the leader doesn't want to have a debate about that possibility.  Thus, they never ask the what if question about this assumption.