Research labs are an old idea. Edison’s Menlo park lab in the 1870s had most of the elements innovation labs and R&D groups try to emulate today. But that doesn’t stop companies today from starting from scratch, assuming they’re the first to try. In researching the book the Myths of Innovation I studied how research labs in established companies work, and why they fail.

Here’s why in simple terms:

• Ideas are easy. It’s not hard to find interesting ideas. They’re everywhere. Many corporate R&D (research and development) groups and universities know how to develop an idea into a prototype. But developing a prototype into a complete, and marketable product is much harder. A research lab at a large company could discover perpetual motion and it would still be hard to get release it as a product.
• Creating an R&D group, or Innovation Team, alienates everyone else. This is the killer – the limits of sociology kills innovation more often than the limits of technology. The jargon is technology transfer, which means “how do we get ideas from the R&D group into products?” Product teams are very busy with their own projects and have little natural interest in wanting to follow someone else’s plans. R&D, by design, sets goals unapproachable for product team’s 3/6/12 month horizons and wonder why they’re ignored 90% of the time, while product teams look at R&D and think they’re insane (10 year horizons? Will we be alive then?). Managers love to couch this problem with jargon like “innovation pipelines” and “invention waves” but they’re missing the real challenge – they don’t see the problem as sociology. R&D groups and product teams naturally resent each other, primarily because the leaders of each group have very different belief systems about what is valuable. Until those issues are addressed little else matters: great ideas will go nowhere.
• R&D never has a service arm. One obvious way to build camaraderie with product teams is to dedicate 10% of R&D as a service to Product team engineers. What problems are too big for release schedules, but too tactical for typical R&D ambitions? That’s the sweet spot where technology transfer starts and grass roots level bonds can be made. Any R&D manager who invests in closing the culture gap buys trust that will help when R&D has a breakthrough idea that ordinarily would scare product teams. The problem is that R&D ego would see this role as a concession to product team “superiority”, which is why its never been done.
• The politics are broken. Conflicts start at the VP level. R&D is directed at satisfying VPs, and R&D managers bet that when they deliver a great innovation the VP, or CEO, will do the radical shift from old ideas to new ones. There are often big egos at work in anyone interested in leading an innovation lab and the hubris is hard to hide. They underestimate how hard politically these moves are and how rarely they happen. VPs may have the most power but often that means they are least likely to adopt change since they have the most to lose (it’s likely rising middle managers on product teams that are at the leverage point of power, ambition and risk taking, not their boss).
• Everyone wants to feel creative. Would you want to be told to only work on other people’s ideas? Any R&D group faces the challenge of being in the ivory tower: they’re resented as soon as they walk in the door, especially if they throw their PhD’s and research pedigrees around at engineers with different values. Innovation is a social process that smart motivated people want to participate in – if you propagate the belief that only special people in special roles can do it, something is broken (see questioning VPs of Innovation).
• At the end of the day someone has to make the bet. Good R&D minimizes risk but never eliminates it. It produces explorations and potentials, but someone has to say “I’m betting my successful and profitable division on new thing X”. That’s a hard thing to say: I’ve never said it, have you? Most R&D managers despite their confidence have never said it either and part of why they’re “in research” is their lack of aptitude or interest in business thinking. In the end every failed R&D effort has a Product VP, or CEO, who was unwilling to take the risk, either for good reasons (the ideas were not worthy) or bad (they didn’t believe or didn’t have the the courage). No one likes to risk their career based on someone else’s bet.

References:

• This Booz-Allen report evaluates R&D spending and supports the notion that pure R&D matters less, in a business context, than transfer.
• Mark Stefik’s book Breakthrough: Stories and Strategies of Radical Innovation was the best R&D management book I found. It’s interview-centric to a fault and not the typical list of do’s and don’ts found in most management books – it attempts to provide a framework for how leaders of R&D should think about innovation with many supporting stories.
• There is a shortage of good books on managing R&D – trade secrets or unwillingness to write about failure? Who knows. The closest thing to a recommended reference I found was The human side of managing technological research edited by Ralph Katz, a collection of papers that covers performance, process, invention and human nature.
• The primary reason research labs fail is that successful innovation depends on variables beyond our control – you can do everything right and still not deliver the breakthrough everyone expects – a theme explored in depth in the Myths of Innovation.