Why Designers Fail: The Report
Why designers succeed and fail in organizations is mostly a matter of opinion as there’s little data to support theories or make claims. To help ground discussions on this subject I ran a research study on why designers fail – exploring the psychological, skill and organizational reasons why designers, and people who work with designers, believe designers don’t achieve the results they desire (PDF of survey questions here).
Update: this research was originally presented at UIE 13, and also at MX ’08, An Event Apart Seattle ’08 and ’11. The latest version of the slides can be downloaded here, and this is the written sumation.
Top line summary
The survey consisted of 41 issues, divided into three categories: Psychological, Skill and Organization. Each participant ranked each issue on a scale from 1 to 5, with 5 meaning the issue was highly significant in explaining why designers fail, and 1 meaning least significant (3 was identified as a neutral value).
The 389 survey respondents self identified as (rounded up #s):
| Designer | 34% | |
| Project manager | 17% | |
| Programmer / Tester | 12% | |
| Usability engineer | 10% | |
| Group manager | 7% | |
| Business / Marketing | 4% | |
| Documentation | 1% | |
| Other | 17% |
The top 16 issues, ranked by average scores were:
| People in non-design roles making design decisions | 4.18 |
| Managers making design decisions w/o design training | 4.14 |
| Designers don’t seek enough data before designing | 3.92 |
| No time is provided for long term thinking | 3.81 |
| Not receptive to critical feedback | 3.69 |
| Lack of awareness of the business fundamentals | 3.66 |
| Only lip-service is paid to “User centered design” | 3.64 |
| It’s never made safe to fail or experiment | 3.62 |
| Designer’s power diluted by too many cooks | 3.60 |
| Over-reliance on one kind of design style | 3.54 |
| Poor collaboration skills | 3.51 |
| Poor persuasion / idea pitching skills | 3.49 |
| Poor communication skills | 3.49 |
| Poor understanding of domain | 3.48 |
| Pressure to use first solution, not a good solution | 3.45 |
| Big Ego / Expects others to cater to their whims | 3.41 |
Average scores per grouping
The average scores for groupings showed little different in weightings: there was no single grouping of issues that proved to be significantly more important in explaining why designers fail.
| Organizational issues: | 3.37 | |
| Skill issues: | 3.15 | |
| Psychological issues: | 3.11 |
Managers vs. Individual contributors
(Note: these charts are quick and dirty. If you’d like the raw data to generate better ones, just ask).
One research question of the study was to see how individual contributors and managers varied in their thinking on failure. 49% of those surveyed identified as playing a lead or management role.
The results showed extremely high correlation between the opinion of individuals and managers.
Designers vs. non-designers
Another research question was how designers and non-designers results would compare. The results showed only minor variance in how designers and non-designers view causes of failure.
The results showed extremely high correlation between the opinion of designers and non-designers on why designers fail.
Conclusions
- Many top reasons for failure are not typically considered design issues, such as collaboration skills, persuasion skills, and receiving critical feedback.
- General consensus on top issues: managers, non-managers, designers and non-designers all had highly similar scores.
- Nearly half of all respondents took time to write in additional issues and thoughts. There was a great deal of interest in discussing this topic further.
Background and Disclosures
- This survey was designed primarily for qualitative use and as a basis for further discussion and research. I’m sure there are flaws and bias in the study design but I believe this study is valuable anyway.
- The issue list was based on 3 things: my own experiences managing UX design training for Microsoft from 1999-2002, many years of debating this question at drunken design conference receptions, and this discussion on a previous blog post.
- The issue descriptions listed above were modified to fit in the post – see the actual survey if you’re suspicious of leading questions or other survey bias issues.
- The survey was distributed via this blog, the iXda mailing list and the pmclinic list.
- A PDF version (95k) of the actual survey can be downloaded.
You can download the slides from from my talks on this at UI13 and an An Event Apart Seattle.
I’m looking for folks to help continue this research. If you’re interested leave a comment.
Like to participate furture servery activities you are doing…I cant download PDF version as it times out. You might want to fix that let us know.
Scott,
Could you clarify the two comparison charts for me (Manager v Individual; designer v non-designer) and confirm that the Individual and Non-designer roles were exclusive sub-sets of the full data set? That is, there was no overlap between the designer/non-designer data; or the manager/individual data sets?
The correlation between responses seems remarkably strong.
Steve
Steve:
Good question. Each individual was forced to provide two pieces of data:
1) are you a designer
2) are you a manager
The two charts (Designer vs non, Manager vs. non) shown include ALL of the survey responses, simply divided into two piles based on the above pieces of data.
So yes, there was overlap. You could identify yourself as a designer and a manager and have your data show up on both of those charts.
Just for sanity I’ll go back and make sure I didn’t do anything really stupid, but the data is different, just not significantly so.
The PDF you linked at the end of the article won’t open in Preview.app
Tony: Not sure how to help you. The PDF works fine for my two PCs. I used PDF995 to create the PDF and I’ve never had complaints before.
Scott
We struggle with this issue nearly every day and would love to be a part of the research.
Thanks!
Interesting results Scott.
I didn’t have time to complete the survey unfortunately, but some of the key findings certainly correlate with my own experience.
The first two findings though seem extremely similar to me – non-designer making design decisions, and managers making design decisions. Aren’t these the same thing in many cases?
Anyway, apologies if this has been discussed before, but it seems like these two issues are common causes of problems.
Congratulations on the good work anyway, and it’ll be interesting to see where you can take this!
Interesting post, but your graphs let you down. They shouldn’t be line graphs, as the X axis is not a continuous value, but categories. A bar chart is a better graph to use.
I am interested in helping with continued research.
Hi David: The main goal of those two charts was to compare results across all 41 issues, and after experimenting with different charts the range of differences was easiest to see with a line graph, especially at the 400 or so pixels that fit in a blog post. Technically you are correct, a line chart implies non-discrete data, but I chose ease of use for understanding the chart, over technical accuracy.
Frankly, I found all the charting tools I could get my hand on totally sucked. Excel in particular. If anyone pointed me at a good charting tool, I’d update the charts with something that looked less like it was made by an 8th grade science teacher.
Scott, going over your list, I see a few patterns emerge of larger undercurrents to these problems. I’ve taken the liberty of grouping them together and attaching remarks.
People in non-design roles making design decisions 4.18
Managers making design decisions w/o design training 4.14
Not receptive to critical feedback 3.69
Designer’s power diluted by too many cooks 3.60
Big Ego / Expects others to cater to their whims 3.41
Design is often associated with power because the team ends up doing what the designer prescribes. This may end up in the hands of the manager, a senior programmer, or in contemporary Agile circles some kind of consensus. In the former cases there is a conflict of interest, and in the latter there are several. A way to defuse this effect is to officially recognize design as orthogonal to, rather than above, other production-oriented work, and assign the job of enforcing conceptual integrity to one brain only.
Designers don’t seek enough data before designing 3.92
It’s never made safe to fail or experiment 3.62
Over-reliance on one kind of design style 3.54
Pressure to use first solution, not a good solution 3.45
I’m not sure what it is that compels us to expect that we can concoct a design in a vacuum and then unveil the completed artifact to an audience and have it work perfectly the first time. I blame TV.
Lack of awareness of the business fundamentals 3.66
Only lip-service is paid to “User centered design” 3.64
Poor understanding of domain 3.48
I think the designer is easily confused as to what their role is. Design really is a muddy term. I’ve met far too many artists (or programmers, or managers) that consider themselves designers, and I’ve met far too many designers that understand themselves to be engineers (or marketers, or CEOs, or whatever). To me, design essentially boils down to conceptual integrity, and therefore a designer is the one in charge of that.
Poor collaboration skills 3.51
Poor persuasion / idea pitching skills 3.49
Poor communication skills 3.49
I see these elements as being endemic to professional life.
No time is provided for long term thinking 3.81
And this one as being endemic to business.
I could slice these points a few other ways, but I’d be here all day! Cheers.
Dorian, your statements here are incredibly well put. This has been the saving grace of this article, for me at least, because the data is a little hard to draw conclusions from as presented.
I’m interested in helping with the continued research.
Just reviewed the results. And from experience, I think it boils down to:
Poor leadership with a poor vision.
All strategic work should stem from having a clear vision. And all actions (or tactics) should support the strategy. In plain language, are the things that you do, the services that you provide, the products that you build consistent with your vision?
Does leadership reward the values that they want in a public manner?
Do the leaders make it crystal clear on the vision of the organization? Do employees know how they help contribute to the vision?
Do the leaders seek to understand the customer? Do they empower their employees (and reward their employees) to talk in various formats to customers?
Does the company walk the talk?
I think design fails because everyone is racing towards some place but no one understands why they are racing and what the track looks like.
Scott,
As with many things in life, it’s dificult to isolate an independent variable for a cause/effect relationship.. probably the combination of some of those at the same time produce failure (non-linear relationship).
I know this is an obvious observation, but the truth it that we still rely on linearity to draw conclusions on complex systems! Just look at what happened to the financial system!
my 2 cents
Office 2007’s version of Excel would have helped you make prettier charts. If you want me to, I’ll do them for you. I also think the numeric averages need bar charts.
Yes, I’m volunteering to help with further work as I said before-!
Scott,
Ditto on the bad presentation of the discontinuous data in the charts. If the point is to compare results across two groups, I would be inclined to aggregate the data, and highlight the specific results with larger intergroup differences.* As to charting software, I used to use Sigmaplot when I was in working in research, but don’t know if it’s still around.
I missed the original survey, but I’m interested in the question itself. Who says designers fail? Why do people assume that they do? How do we mesaure design success or failure? Do designers “fail” because the expectations put on them are unrealistic? Using “meaningful criteria”(TM), how frequently *do* they fail?
*PS. I don’t know Edward Tufte and haven’t (yet) read his book…
Hi Scott:
I am wondering if the skewed nature of the study responses, is a problem? Would the graphs for example not be more meaningful/different, if we looked at equal representation between designers and business stakeholders?
I think it would be interesting to also use design to do some data visualizations around this study if it produced more complex data ponts that have not been shown above!
Thanks, I support studies that help bridge the gap in design communication, management and buy-in.
Gagan:
I dont know what skewed nature you’re referring to. You have to be more specific. Do you think the survey respondents were skewed? The data?
Hi Scott,
Thanks for kicking off this research and discussion. I’ve been working in design for 15 years and have encountered so many situations that set up designers for failure, and so many designers who, regardless of the situation, set themselves up for failure.
Right now my biggest concern is creating a highly successful multidisciplinary design team in a fast-paced, agile, rapid iteration software environment. We’re constantly hiring so we have the added challenge of training seasonal professionals and complete newbies.
Consider me interested.
ian
Design means making decisions. This can put designers and managers, if their relationship isn’t already grounded in trust and mutual professional respect, in immediate, if not explicit, conflict. Inexperienced and immature managers (and above) get a charge out of playing de facto art director. This is like telling your doctor how to operate on you. No good can come of it.
I wholeheartedly agree with Dorian T. Oct. 21. Great comments. I would add that confusion (lack of distinction) between art and “design” is partly to blame, as well as TV.
The best arbiter of conflict between a designer and manager is a user. Choose them wisely, and use their feedback more wisely. Distinguish between what they need versus want. Same for the client.
Designers are idea peddlers. We have a responsibility to persuade. Language shapes our thoughts. As with colors and color combo’s: “The difference between the right word and almost the right word is the difference between a lightening bug and lightening.” -Mark Twain.
I’ve come to see the “soft skills” of facilitating collaboration, communication, empathy, patience, campaigning on design’s behalf, creating a good environment for design, etc., as even more necessary than the techniques and methods and tool using “hard skills” we must also bring to bare. Turns out the soft skills are harder. Maybe just from lack of good training? Oh, we’re back to TV again.
“Managers Who Love Design and the Designers Who Don’t Love Them:
http://uxdesign.com/ux-management/article/managing-design/17
Reason #8 on the top 15 issues for “Why designers fail:”
It’s never made safe to FAIL or experiment.
So if it is made safe for us to fail, then suddenly we stop failing. I find this really funny for some reason. It is really cracking me up. Don’t get me wrong, I get the concept, it’s just the wording that sounds so funny. Make it safe for me to fail and I won’t fail so much. Oh I’ll still come up with designs that fail but now we’ll call that a “learning exercise” instead of a failed design.
No wonder people don’t get designers…
Scott:
Great job!
I am a designer with my own practice and teach design and typography at the college level. I am very much interested in continuing this discussion especially as it relates to students. Stressing the fundamentals of research, brainstorming, critique, business classes for design students, collaboration, presentation, and communication skills would help eliminate or mitigate most of these numbers to skew much lower.
As for the following problems:
People in non-design roles making design decisions
Managers making design decisions w/o design training
No time is provided for long term thinking
Designer’s power diluted by too many cooks
These are historical problems designers have never had control over. Having the savvy and experience when NOT to take on a project or when to fire your client usually are skills learned OJT.
Scott,
I sat in on this lecture at UIE a week or so ago – I am fascinated with this topic. I’d love to help out!
Tara
Interesting looking, albeit expensive, report:
http://www.cbponline.com/product_p/research-troubledprojects.htm
Gee, how predictable . . . your replies are mostly from designers and SURPRISE, they think that it’s others that mess up their work.
Wow, what next . . . story on dog biting man?
From a non-designer’s perspective, designers taking too much damned time on a ‘pencil’ when they were given specific instructions to ‘give me something I can just show the client’, is what messes things up most.
LISTEN, snobs, you’ll be better off.
I ‘m not sure if this has been mentioned but my observations has been that designs (including mine) fail most often due to insufficient time developing it.
Most of the time up-front design time is all that is available and this is too short especially if the designers are less skilled.
But even after the designs are made, if there is no flexibility in changing it (as the designer themselves if tasked at implementing it often do) or no time available to do so, the designs would stand a change of making a bigger-bad impact than otherwise.
Edge cases and known subtleties are often hidden to the maintainers or 3rd parties who often take it up, as it so often happens in big corporate places. Continous improvement is un-avoidable even with world class designs, which are i feel sure to show as history continous improvements behind them. No one really makes systems that are world class at 1.0 these days.
Interesting similarly themed piece:
http://www.metafilter.com/80355/Assume-failure
I’ve been on both sides of the equation (long career in usability, slightly less long career in Engineering Management since then).
I am not at all surprised that the design community places the blame for failure outside of themselves. I would have done exactly the same when I was a practitioner.
Engineering staff would love to have competent design peers, as I fortunately do in my current job. However, I have worked on and with Ux teams that impose more cost than benefit to the people tasked with delivering on time, on budget.
Some of the common things that I have seen that limited design effectiveness are:
1. Design is lagging behind development, tweaking designs for problems that are considered solved by the development teams but not producing timely solutions to problems being solved at the moment.
2. Design does not grasp the problem domain, but thinks they do.
3. Designs that are not appreciably better than what is produced by non-design professionals. Remember, fully 50% of designers are below average in skill for their field.
Can’t believe no one has pointed out the “Top 15 reasons” list has 16 entries.