Why designers fail: the report

Update: I presented an updated version of this talk at An Event Apart Seattle 2008: newer slides here.

I recently announced a study on why designers fail – exploring the reasons why designers, and people who work with designers, believe designers don’t achieve the results they desire. I presented the results as UIE 13 last week, and as promised here is a summary. Prize winners will be announced soon.

Top line summary

The survey consisted of 41 issues, divided into three categories: Psychological, Skill and Organizational. Each participant ranked each issue on a scale from 1 to 5, with 5 meaning the issue was highly significant in explaining why designers fail, and 1 meaning least significant (3 was identified as a neutral value).

The 389 survey respondents self identified as (rounded up #s):

Designer 34%
Project manager 17%
Programmer / Tester 12%
Usability engineer 10%
Group manager 7%
Business / Marketing 4%
Documentation 1%
Other 17%


The top 16 issues, ranked by average scores were:

People in non-design roles making design decisions 4.18
Managers making design decisions w/o design training 4.14
Designers don’t seek enough data before designing 3.92
No time is provided for long term thinking 3.81
Not receptive to critical feedback 3.69
Lack of awareness of the business fundamentals 3.66
Only lip-service is paid to “User centered design” 3.64
It’s never made safe to fail or experiment 3.62
Designer’s power diluted by too many cooks 3.60
Over-reliance on one kind of design style 3.54
Poor collaboration skills 3.51
Poor persuasion / idea pitching skills 3.49
Poor communication skills 3.49
Poor understanding of domain 3.48
Pressure to use first solution, not a good solution 3.45
Big Ego / Expects others to cater to their whims 3.41

Average scores per grouping

The average scores for groupings showed little different in weightings: there was no single grouping of issues that proved to be significantly more important in explaining why designers fail.

Organizational issues: 3.37
Skill issues: 3.15
Psychological issues: 3.11

Managers vs. Individual contributors

(Note: these charts are quick and dirty. If you have a pretty design stick and know how to use it, happy to share the data so you can make better charts).

One research question of the study was to see how individual contributors and managers varied in their thinking on failure. 49% of those surveyed identified as playing a lead or management role.

The results showed extremely high correlation between the opinion of individuals and managers.

Designers vs. non-designers

Another research question was how designers and non-designers results would compare. The results showed only minor variance in how designers and non-designers view causes of failure.

The results showed extremely high correlation between the opinion of designers and non-designers on why designers fail.

Conclusions

  • Many top reasons for failure are not typically considered design issues, such as collaboration skills, persuasion skills, and receiving critical feedback.
  • General consensus on top issues: managers, non-managers, designers and non-designers all had highly similar scores.
  • Nearly half of all respondents took time to write in additional issues and thoughts. There was a great deal of interest in discussing this topic further.

Background and Disclosures

  • This survey was designed primarily for qualitative use and as a basis for further discussion and research. I’m sure there are flaws and bias in the study design but I believe this study is valuable anyway.
  • The issue list was based on 3 things: my own experiences managing UX design training for Microsoft from 1999-2002, many years of debating this question at drunken design conference receptions, and this discussion on a previous blog post.
  • The issue descriptions listed above were modified to fit in the post – see the actual survey if you’re suspicious of leading questions or other survey bias issues.
  • The survey was distributed via this blog, the iXda mailing list and the pmclinic list.
  • A PDF version (95k) of the actual survey can be downloaded.

Here is a PDF version of the full UIE 13 talk on Why designers fail (6MB PDF). Update: A presented an updated version of this talk at An Event Apart Seattle. Newer slides here.

I’m looking for folks to help continue this research. If you’re interested leave a comment.

54 Responses to “Why designers fail: the report”

  1. Chaminda

    Like to participate furture servery activities you are doing…I cant download PDF version as it times out. You might want to fix that let us know.

    Reply
  2. Steve Baty

    Scott,

    Could you clarify the two comparison charts for me (Manager v Individual; designer v non-designer) and confirm that the Individual and Non-designer roles were exclusive sub-sets of the full data set? That is, there was no overlap between the designer/non-designer data; or the manager/individual data sets?

    The correlation between responses seems remarkably strong.

    Steve

    Reply
  3. Scott

    Steve:

    Good question. Each individual was forced to provide two pieces of data:

    1) are you a designer
    2) are you a manager

    The two charts (Designer vs non, Manager vs. non) shown include ALL of the survey responses, simply divided into two piles based on the above pieces of data.

    So yes, there was overlap. You could identify yourself as a designer and a manager and have your data show up on both of those charts.

    Just for sanity I’ll go back and make sure I didn’t do anything really stupid, but the data is different, just not significantly so.

    Reply
  4. Tony

    The PDF you linked at the end of the article won’t open in Preview.app

    Reply
  5. Scott

    Tony: Not sure how to help you. The PDF works fine for my two PCs. I used PDF995 to create the PDF and I’ve never had complaints before.

    Reply
  6. Barb

    Scott

    We struggle with this issue nearly every day and would love to be a part of the research.

    Thanks!

    Reply
  7. Lee

    Interesting results Scott.

    I didn’t have time to complete the survey unfortunately, but some of the key findings certainly correlate with my own experience.

    The first two findings though seem extremely similar to me – non-designer making design decisions, and managers making design decisions. Aren’t these the same thing in many cases?

    Anyway, apologies if this has been discussed before, but it seems like these two issues are common causes of problems.

    Congratulations on the good work anyway, and it’ll be interesting to see where you can take this!

    Reply
  8. David Pashley

    Interesting post, but your graphs let you down. They shouldn’t be line graphs, as the X axis is not a continuous value, but categories. A bar chart is a better graph to use.

    Reply
  9. Bianca

    I am interested in helping with continued research.

    Reply
  10. Scott

    Hi David: The main goal of those two charts was to compare results across all 41 issues, and after experimenting with different charts the range of differences was easiest to see with a line graph, especially at the 400 or so pixels that fit in a blog post. Technically you are correct, a line chart implies non-discrete data, but I chose ease of use for understanding the chart, over technical accuracy.

    Frankly, I found all the charting tools I could get my hand on totally sucked. Excel in particular. If anyone pointed me at a good charting tool, I’d update the charts with something that looked less like it was made by an 8th grade science teacher.

    Reply
  11. Dorian Taylor

    Scott, going over your list, I see a few patterns emerge of larger undercurrents to these problems. I’ve taken the liberty of grouping them together and attaching remarks.

    People in non-design roles making design decisions 4.18
    Managers making design decisions w/o design training 4.14
    Not receptive to critical feedback 3.69
    Designer’s power diluted by too many cooks 3.60
    Big Ego / Expects others to cater to their whims 3.41

    Design is often associated with power because the team ends up doing what the designer prescribes. This may end up in the hands of the manager, a senior programmer, or in contemporary Agile circles some kind of consensus. In the former cases there is a conflict of interest, and in the latter there are several. A way to defuse this effect is to officially recognize design as orthogonal to, rather than above, other production-oriented work, and assign the job of enforcing conceptual integrity to one brain only.

    Designers don’t seek enough data before designing 3.92
    It’s never made safe to fail or experiment 3.62
    Over-reliance on one kind of design style 3.54
    Pressure to use first solution, not a good solution 3.45

    I’m not sure what it is that compels us to expect that we can concoct a design in a vacuum and then unveil the completed artifact to an audience and have it work perfectly the first time. I blame TV.

    Lack of awareness of the business fundamentals 3.66
    Only lip-service is paid to “User centered design” 3.64
    Poor understanding of domain 3.48

    I think the designer is easily confused as to what their role is. Design really is a muddy term. I’ve met far too many artists (or programmers, or managers) that consider themselves designers, and I’ve met far too many designers that understand themselves to be engineers (or marketers, or CEOs, or whatever). To me, design essentially boils down to conceptual integrity, and therefore a designer is the one in charge of that.

    Poor collaboration skills 3.51
    Poor persuasion / idea pitching skills 3.49
    Poor communication skills 3.49

    I see these elements as being endemic to professional life.

    No time is provided for long term thinking 3.81

    And this one as being endemic to business.

    I could slice these points a few other ways, but I’d be here all day! Cheers.

    Reply
    • Ryan

      Dorian, your statements here are incredibly well put. This has been the saving grace of this article, for me at least, because the data is a little hard to draw conclusions from as presented.

      Reply
  12. Georgia

    I’m interested in helping with the continued research.

    Reply
  13. Jenn_lee_ca

    Just reviewed the results. And from experience, I think it boils down to:

    Poor leadership with a poor vision.

    All strategic work should stem from having a clear vision. And all actions (or tactics) should support the strategy. In plain language, are the things that you do, the services that you provide, the products that you build consistent with your vision?

    Does leadership reward the values that they want in a public manner?

    Do the leaders make it crystal clear on the vision of the organization? Do employees know how they help contribute to the vision?

    Do the leaders seek to understand the customer? Do they empower their employees (and reward their employees) to talk in various formats to customers?

    Does the company walk the talk?

    I think design fails because everyone is racing towards some place but no one understands why they are racing and what the track looks like.

    Reply
  14. Pedro Ornelas

    Scott,

    As with many things in life, it’s dificult to isolate an independent variable for a cause/effect relationship.. probably the combination of some of those at the same time produce failure (non-linear relationship).

    I know this is an obvious observation, but the truth it that we still rely on linearity to draw conclusions on complex systems! Just look at what happened to the financial system!

    my 2 cents

    Reply
  15. Lynn Cherny

    Office 2007’s version of Excel would have helped you make prettier charts. If you want me to, I’ll do them for you. I also think the numeric averages need bar charts.

    Yes, I’m volunteering to help with further work as I said before-!

    Reply
  16. greg

    Scott,

    Ditto on the bad presentation of the discontinuous data in the charts. If the point is to compare results across two groups, I would be inclined to aggregate the data, and highlight the specific results with larger intergroup differences.* As to charting software, I used to use Sigmaplot when I was in working in research, but don’t know if it’s still around.

    I missed the original survey, but I’m interested in the question itself. Who says designers fail? Why do people assume that they do? How do we mesaure design success or failure? Do designers “fail” because the expectations put on them are unrealistic? Using “meaningful criteria”(TM), how frequently *do* they fail?

    *PS. I don’t know Edward Tufte and haven’t (yet) read his book…

    Reply
  17. Gagan Diesh

    Hi Scott:

    I am wondering if the skewed nature of the study responses, is a problem? Would the graphs for example not be more meaningful/different, if we looked at equal representation between designers and business stakeholders?

    I think it would be interesting to also use design to do some data visualizations around this study if it produced more complex data ponts that have not been shown above!

    Thanks, I support studies that help bridge the gap in design communication, management and buy-in.

    Reply
  18. Scott (admin)

    Gagan:

    I dont know what skewed nature you’re referring to. You have to be more specific. Do you think the survey respondents were skewed? The data?

    Reply
  19. Ian Swinson

    Hi Scott,
    Thanks for kicking off this research and discussion. I’ve been working in design for 15 years and have encountered so many situations that set up designers for failure, and so many designers who, regardless of the situation, set themselves up for failure.

    Right now my biggest concern is creating a highly successful multidisciplinary design team in a fast-paced, agile, rapid iteration software environment. We’re constantly hiring so we have the added challenge of training seasonal professionals and complete newbies.

    Consider me interested.

    ian

    Reply
  20. uxdesign.com

    Design means making decisions. This can put designers and managers, if their relationship isn’t already grounded in trust and mutual professional respect, in immediate, if not explicit, conflict. Inexperienced and immature managers (and above) get a charge out of playing de facto art director. This is like telling your doctor how to operate on you. No good can come of it.

    I wholeheartedly agree with Dorian T. Oct. 21. Great comments. I would add that confusion (lack of distinction) between art and “design” is partly to blame, as well as TV.

    The best arbiter of conflict between a designer and manager is a user. Choose them wisely, and use their feedback more wisely. Distinguish between what they need versus want. Same for the client.

    Designers are idea peddlers. We have a responsibility to persuade. Language shapes our thoughts. As with colors and color combo’s: “The difference between the right word and almost the right word is the difference between a lightening bug and lightening.” -Mark Twain.

    I’ve come to see the “soft skills” of facilitating collaboration, communication, empathy, patience, campaigning on design’s behalf, creating a good environment for design, etc., as even more necessary than the techniques and methods and tool using “hard skills” we must also bring to bare. Turns out the soft skills are harder. Maybe just from lack of good training? Oh, we’re back to TV again.

    “Managers Who Love Design and the Designers Who Don’t Love Them:
    http://uxdesign.com/ux-management/article/managing-design/17

    Reply
  21. Mark

    Reason #8 on the top 15 issues for “Why designers fail:”

    It’s never made safe to FAIL or experiment.

    So if it is made safe for us to fail, then suddenly we stop failing. I find this really funny for some reason. It is really cracking me up. Don’t get me wrong, I get the concept, it’s just the wording that sounds so funny. Make it safe for me to fail and I won’t fail so much. Oh I’ll still come up with designs that fail but now we’ll call that a “learning exercise” instead of a failed design.

    No wonder people don’t get designers…

    Reply
  22. steve kennedy

    Scott:
    Great job!
    I am a designer with my own practice and teach design and typography at the college level. I am very much interested in continuing this discussion especially as it relates to students. Stressing the fundamentals of research, brainstorming, critique, business classes for design students, collaboration, presentation, and communication skills would help eliminate or mitigate most of these numbers to skew much lower.
    As for the following problems:
    People in non-design roles making design decisions
    Managers making design decisions w/o design training
    No time is provided for long term thinking
    Designer’s power diluted by too many cooks

    These are historical problems designers have never had control over. Having the savvy and experience when NOT to take on a project or when to fire your client usually are skills learned OJT.

    Reply
  23. Tara

    Scott,
    I sat in on this lecture at UIE a week or so ago – I am fascinated with this topic. I’d love to help out!
    Tara

    Reply
  24. Tired of DesignSnobs

    Gee, how predictable . . . your replies are mostly from designers and SURPRISE, they think that it’s others that mess up their work.

    Wow, what next . . . story on dog biting man?

    From a non-designer’s perspective, designers taking too much damned time on a ‘pencil’ when they were given specific instructions to ‘give me something I can just show the client’, is what messes things up most.

    LISTEN, snobs, you’ll be better off.

    Reply
  25. Arun

    I ‘m not sure if this has been mentioned but my observations has been that designs (including mine) fail most often due to insufficient time developing it.

    Most of the time up-front design time is all that is available and this is too short especially if the designers are less skilled.

    But even after the designs are made, if there is no flexibility in changing it (as the designer themselves if tasked at implementing it often do) or no time available to do so, the designs would stand a change of making a bigger-bad impact than otherwise.

    Edge cases and known subtleties are often hidden to the maintainers or 3rd parties who often take it up, as it so often happens in big corporate places. Continous improvement is un-avoidable even with world class designs, which are i feel sure to show as history continous improvements behind them. No one really makes systems that are world class at 1.0 these days.

    Reply
  26. Rick Spencer

    I’ve been on both sides of the equation (long career in usability, slightly less long career in Engineering Management since then).

    I am not at all surprised that the design community places the blame for failure outside of themselves. I would have done exactly the same when I was a practitioner.

    Engineering staff would love to have competent design peers, as I fortunately do in my current job. However, I have worked on and with Ux teams that impose more cost than benefit to the people tasked with delivering on time, on budget.

    Some of the common things that I have seen that limited design effectiveness are:
    1. Design is lagging behind development, tweaking designs for problems that are considered solved by the development teams but not producing timely solutions to problems being solved at the moment.
    2. Design does not grasp the problem domain, but thinks they do.
    3. Designs that are not appreciably better than what is produced by non-design professionals. Remember, fully 50% of designers are below average in skill for their field.

    Reply
  27. nemo

    Can’t believe no one has pointed out the “Top 15 reasons” list has 16 entries.

    Reply

Pingbacks

  1. All That Sparkles……

    I hope you’ve enjoyed the recent interviews with Chad and Kristin … I’m lining up a new set of interesting people/topics for early spring, so be on the lookout. Right now, though, there’s a more pressing issue to discuss: laundry…….

  2. […] ?? ??????????? MX’09, ??????? ???????????? ???????? Adaptive Path, ?? ?????? ?????? ??? ???????? ????????, ??????? ?????? ??? ?? ???? ??????????? ??????? ???????? ? ???? ?????????? ???????. ????????, ??? ??? ?????? ?????? ???????? ?????? ?????? ??????? «Why designers fail & what to do about it». […]

  3. […] I also noted the discussion in my UsabilityBlog post of last week. In another happy surprise for me, Scott dropped by and responded on UsabilityBlog, saying: The curious thing is why this fairly old idea (specialists need to persuade) has such a hard time gaining traction among the UI/IX/HCI community. And oddly, it’s seems really hard to sell the UI community on the point of view you offer. Do you have any theories as to why this is? I did a study of sorts on designers about why they fail to explore this question among the design community. But I have thought about doing a similiar thing for the usability/HCI side of things: http://www.scottberkun.com/blog/2008/why-design… […]

Leave a Reply

* Required