Why research labs fail at innovation

Research labs are an old idea. Edison’s Menlo park lab in the 1870s had most of the elements hi-tech R&D groups try to emulate. But that doesn’t stop every wave of innovators from starting from scratch, attacking the challenges of innovation as if they were the first to try.

In researching the book the Myths of Innovation I studied how research labs in established companies work, and why they fail.

There are many reasons for failure – foremost that creating new things is hard for anyone, anywhere – but I posit  labs succeed or fail because of their relationships with product teams, rather than their technological or creative brilliance.

Here’s why in simple terms:

  • Ideas are easy. It’s not hard to find interesting ideas. They’re everywhere. There are many R&D groups and universities with lots of good ideas. Hell, there are many programmers or marketers with good ideas and given time and some funds they can develop them into impressive prototypes. It’s certainly hard work but it’s the easier part – it’s getting new ideas into products and out to customers that’s hard (for established companies). An R&D lab at a large company could discover perpetual motion and it would still be very hard to get it out into a product.
  • Creating an R&D group alienates everyone else. This is the killer – the limits of sociology kills innovation more often than the limits of technology. The jargon is technology transfer, which means “how do we get ideas from the R&D group into products?” It’s the least exciting challenge but the most important. If the ideas never impact products what was the point? R&D sets goals unapproachable for product team’s 3/6/12 month horizons and wonder why they’re ignored 90% of the time, while product teams look at R&D and think they’re insane (10 year horizons? WTF). Managers love to couch this problem with jargon like “innovation pipelines” and “invention waves” but its BS – they don’t see the problem as sociology. R&D groups and product teams naturally resent each other. Until those issues are confronted little else matters: great ideas will go nowhere.
  • R&D never has a service arm. One obvious way to connect R&D to products, or to build good relationships, is to dedicate 10% of R&D as a service to Product team engineers. What problems are too big for release schedules, but too tactical for standard R&D budget? That’s the sweet spot where technology transfer starts. Any R&D manager who invests in closing the culture gap buys chips to spend when R&D has a breakthrough idea that ordinarily would scare the pants off product teams. The problem is that R&D ego would see this role as a concession to product team “superiority”, which is why its never been done.
  • The politics are broken. Conflicts start at the VP level. R&D is directed at satisfying VPs, and R&D managers bet that when they deliver a great innovation the VP will move from old ideas to new ones. They underestimate how hard politically these moves are and how rarely they happen. In the meantime line and middle managers, the people most interested in change, are ignored. VPs may have the most power but often that means they are least likely to adopt change since they have the most to lose. It’s the younger, middle ranked people that have the most ambition for change.
  • Everyone wants to feel creative. Would you want to be told to only work on other people’s ideas? Any R&D group faces the challenge of being in the ivory tower: they’re resented as soon as they walk in the door, especially if they throw their PhD’s and research pedigrees around at engineers with different values. Innovation is a social process that smart motivated people want to participate in – if you propagate the belief that only special people in special roles can do it, something is broken.
  • At the end of the day someone has to take the bet. Good R&D minimizes risk but never eliminates it. It produces explorations and potentials, but someone has to say “I’m betting my successful division on new thing X”. That’s a hard thing to say: I’ve never said it, have you? Most R&D managers despite their confidence have never said it either. In the end every failed R&D effort has a Product VP who was unwilling to take the risk, either for good reasons (the ideas were not worthy) or bad (they didn’t believe or didn’t have the the courage). No one likes to risk their career based on someone else’s bet.

I admit this is a simplified view: there are more complexities than I suggest (R&D groups have a wide range of possible goals) and I’ve worked with, but never in, R&D groups. So there, I confess, I’m a product side innovation zealot (Not really, but if you hated this post feel free to call me this especially if it’s less offensive than whatever you had in mind).


  • This Booz-Allen report evaluates R&D spending and supports the notion that pure R&D matters less, in a business context, than transfer.
  • Mark Stefik’s book Breakthrough: Stories and Strategies of Radical Innovation was the best R&D management book I found. It’s interview-centric to a fault and not the typical list of do’s and don’ts found in most management books – it attempts to provide a framework for how leaders of R&D should think about innovation with many supporting stories.
  • There is a shortage of good books on managing R&D – trade secrets I guess. The closest thing to a recommended reference I found was The human side of managing technological research edited by Ralph Katz, a collection of papers that covers performance, process, invention and human nature.
  • The primary reason research labs fail is that successful innovation depends on variables beyond our control – you can do everything right and still not deliver the breakthrough everyone expects – a theme explored in depth in my book.

10 Responses to “Why research labs fail at innovation”

  1. kavak yelleri

    the blog Tomorrow’s Trends i came across Scott Berkun’s blog and the interesting post Why research labs fail at innovation. It turns out that Scott have released a book on the subject “The Myths of Innovation

  2. Jay Oza

    This is a good post.

    I can relate to this since I worked in the Bell Labs technology transfer group and it failed because it did not have someone who could connect it from the R&D to few customers.

    I learned a lot from that experience and feel that innovation is very difficult unless it is separated from the main business and is operated completely differently than the regular business.

    Lot of companies fall into this trap and fail.



  1. Why research labs fail at innovation…

    There are many reasons for failure – foremost that creating new things is hard – but I posit that labs succeed or fail most often because of their relationships with product teams and not their technological brilliance or war chest of breakthrough idea…

  2. […] Why research labs fail at innovation – More interesting insight from Scott. Worth scanning. The referenced Booz-Allen report evaluating R&D spending seems interesting too. (This must hurt – “Patents don’t always lead to profits. In a separate analysis, Booz Allen found no relationship between the number of patents issued to an organization and its performance.”) […]

  3. […] Why research labs fail at innovation (scottberkun.com) Nice list of ideas for R&D environments. The one I like best is the ‘R&D never has a service arm’. This is something I’ve seen: there never seems to be time and money to solve ‘general grievances’. (tags: innovation scottberkun management research organisation) […]

Leave a Reply

* Required