Reflect & Learn: Post-Project Evaluation of Initial ROI Analysis

Source: Praxis

Upon completion of a project, have you ever gone back to evaluate the initial return on investment analysis used to justify the decision to proceed with that initiative?  This type of post-project evaluation can be very instructive.   Did the project deliver the type of return that was expected?  Why or why not?  Did the project fail, or were the expectations regarding the benefits and returns simply overinflated?  

The purpose of this anlaysis should not be to blame people for misguided cost/benefit analysis, or to punish those who did not deliver results in line with expectations.  Instead, spotting errors and and deriving lessons from this type of reflection can enable an organization to enhance the quality of its decisions moving forward.   ROI analyses will improve, enabling managers to make more informed, higher quality decisions.  

What types of issues and errors should one look out for when evaluating results relative to that initial analysis?  Here are a few questions to ask: 

1. Were we experiencing confirmation bias as we collected and analyzed the data?  
2. Did we make fail to engage a devil's advocate to challenge our forecasts and predictions?  
3. Did we make some assumptions that were overly optimistic?   Did we back into certain assumptions simply to make the return on investment seem attractive?
4. Were the initial investment figures unrealistically low?   Did we have to make substantial additional investments that probably should have been anticipated at the outset?  
5. Did we fail to account for opportunity costs when considering the attractiveness of this opportunity?
6. Were people hesitant to challenge or critique the analysis because they perceived the investment to be a "pet project" of the CEO or another senior leader?

Why Don't We Reflect & Learn?

Most leaders acknowledge the value of learning from past experience.  Some organizaitons have established highly regarded best practices for deriving lessons learned from past projects.  For instance, the U.S. Army pioneered the After-Action Review process, and it has documented the substantial benefits derived from the systematic use of this lessons learned methodology.  Still, most leaders don't spend nearly enough time faciltating these types of activities or empowering their people to engage in this type of work.  Why?  What seems to be preventing these powerful learning experiences from occurring in organizations?  The usual answer is time.  We hear responses such as the following:   "We would love to conduct postmorterms, but who has the time to perform that work?"  "We are always rushing off to implement the next set of plans."  "No one rewards you for taking time out to review and learn from past experiences."  

In my view, time represents a significant challenge, but these types of explanations often mask a deeper problem in the organization.  What are the true impediments to engaging in after-action reviews?

1.  Blame culture.  People fear that the process will degenerate into a finger-pointing exercise, rather than a true learning experience.  In these types of organizations, people fear talking about mistakes and how to learn from them, because they don't want to be accused of being part of the problem.  

2.  Lack of systemic thinking.  In many organizations, explanations of past failure tend to be individualistic, i.e. who was the rotten apple that needed to be thrown out of the bunch?  Learning organizations embrace systemic explanations for past success and failure, i.e. was the barrel rotten, and therefore, did it spoil some of the apples?  Systemic explanations for failures do not preclude managers from holding people accountable for negligent or irresponsible acts.   However, they help managers understand multiple factors that may contribute to success and failure. 

3.  Attribution error.   Psychologists have shown that people often attribute others' failures to character flaws, lack of expertise, or other internal deficiencies.  However, we explain our own failures quite differently.  We blame external events or factors beyond our control.  Such distortions in our attributions may explain why we do not engage in lessons learned exercises as frequently as we should. 

4.  Low leader self-awareness.  In some enterprises, leaders have tried to facilitate such after-action reviews, and the efforts have not been fruitful.  Thus, they choose not to spend time performing them again.  However, these leaders often are not aware that their presence and influece during the after-action review discouraged candid dialogue.  Thus, the group did not generate powerful lessons learned, as people refrained from discussing the tough issues.  Leaders sometimes lack the awareness to recognize how their behavior and presence may have distorted the dialogue during an after-action review process.  

Startup Funeral

Kevin Galligan is one of the organizers of an event called Startup Funeral that takes place in Manhattan.  A group of young professionals get together to hear the stories of startup failures.  The point is for entrepreneurs to share the lessons from an in-depth postmortem analysis of a startup failure.  At the same time, the event is supposed to be fun.... a party, according to Galligan.  Another organizer, Valerie Lisyansky, tells Fortune magazine: “It’s equally as important to be successful as it is to understand your failure, understand what happened, and educate community.”   Publicly sharing postmortems has become more commonplace in the startup community.   People want others to hear the lessons that they have learned the hard way.  

Is this type of event taking the entire "celebrate failure" movement a bit too far?   I don't think so, though I would note that the article refers to the fact that many entrepreneurs back out after initially committing to present at such events.  I think the key to a successful sharing of lessons learned from a startup postmortem is a safe environment.   I'm not sure an open invitation public party is the most safe environment.   You want to have a place where people are comfortable putting themselves out there and taking an interpersonal risk. I also think it's important to hear the perspective of multiple people involved in a startup, not simply the founder(s).   Multiple perspectives can shed light on causes of the failure that are not apparent to the founder(s), or that the founder(s) aren't ready to acknowledge.