Scholars Erin Scott, Pian Shu, and Roman Lubynsky have written a fascinating new paper about startups. They examined a dataset of 652 ventures from MIT's Venture Mentoring Service (VMS). The service attempts to match startups with mentors. The mentors receive data about a variety of startup ideas. They must decide what they think about the ideas without having an opportunity to review information about the founders or to meet the team in person. The researchers then examined how many of these startups went on to have their products commercialized successfully.
Overall, the more highly rated ideas did have a better chance of being commercialized. However, that was not the case for all types of startups. They grouped the ventures in terms of high R&D intensity industries (i.e. life sciences, energy) and low R&D intensity industries (i.e. software, consumer products). Highly evaluated ideas tended to be more likely to be commercialized successfully in the high R&D intensity group, but no such relationship was found in the low R&D intensity group. HBR's Walter Frick explains this finding:
Think of it this way: if the venture “idea” includes patent-protected technology in an industry with high entry costs, it’s going to be easier to determine that the venture has commercial potential. For web and mobile ventures, which are less likely to have intellectual property, and where entry costs are lower, it’s harder to know up front whether a venture will have a real, sustainable competitive advantage.
Finally, the researchers examined whether experts were better at predicting success. Frick writes, "The researchers checked to see if “expert” mentors were any better at picking ideas than the group overall. They looked at mentors with experience in the venture’s industry, as well as mentors with a PhD. Neither group was any better at predicting which ideas would succeed."