By Scott Berkun, May 2001

At the heart of design and engineering is critical thinking. The ability to separate what is worthwhile from what isn’t is the hallmark of the best in many fields, from film directors to project managers, programmers to designers. This skill can be learned but, because of its independence from the world of technology, it’s often forgotten in our industry. It’s clear, however, that most failures in engineering, design, or usability are caused by poor decisions at higher levels. Developers and designers should all have some awareness of what good critical thinking looks like and the process used to generate it. Critical thinking surfaces in three places in a Web or software development effort: planning, idea generation, and project management. This issue focuses on planning, but I’ll cover the others in future issues, provided there’s interest.

The most common failure of development projects is the inability to correctly define the problem. If the goal is vague, it’s impossible to know whether it’s been solved it or not. And even if the goal is well defined, it may be the wrong goal for the situation in which the design will be used. A well-built machine gun won’t help you repair a flat tire. So these two kinds of failure, vague goals and the wrong goals, have nothing to do with technical acumen. If you can’t prevent these kinds of failures, even the best developers or designers in the world will not succeed. You may write great code or create wonderful designs, but if you don’t solve the right problem, your efforts are wasted.

Understanding the Problem

The first step towards critical thinking is to take an objective view on the nature of the problem space. As a developer or designer, you are incredibly biased about the value of what you’re doing. You’re on the inside, looking out, and cannot possibly see your creations the way outsiders will. To get your bearings, you need to triangulate information from multiple sources. The viewpoint of a developer, manager, single important customer is of little value in isolation. Get the bird’s-eye view and as many alternative views as you can find. Make sure to talk directly with the people the design will be made for, and not take people’s word for what the problems are. Think of yourself as a politician trying to help your citizens. Would you only consider the opinions of the lobbyists? To do well, and to see things how they really are, you will have to go out of your normal way of doing things.

An additional challenge is that how you approach your customers will impact the kind of information you get from them. If you unintentionally bias your questions, which everyone does unless they’ve been trained not to, you’ll get unreliable information. The skill of observing and understanding customers is complex and is the primary reason first-rate Web and software teams have usability engineers. You can pick up the basics, but if you’re intent on doing great work, invest in a professional.

When researching the problems you need to solve, keep the following questions in mind:

• Who are our customers? What skills and knowledge do they have?
• What different sources of data can we use to understand their experience?
• What goals and tasks will they use our product or Web site to complete?
• What assumptions are we making and how can we verify them?
• What sources of data do we have? (Usability studies and heuristic evaluations are good places to start.)

If you aren’t confident in your answers to these questions, you’re not done yet. This information is the foundation of your development work. Make sure you have a solid understanding of your customers before you continue. If it’s your first release, use whatever information you can find about competitors and their users.

Analysis

As a developer, there are an infinite number of problems you could try to solve or new features you could add. So having an idea isn’t enough to warrant building it. Many problems are not worth solving, given the investment required and their limited payoff. Sometimes a solution for one problem creates two new problems. So good judgment implies the ability to separate what should be done from what could be done. After collecting data, the next step towards good judgment is analysis. You have to cultivate the information and create an assessment for where to invest.

With the data collected from customers and other sources, distill that information into one-sentence bullets of specific problems. These sentences should be written with respect for the customer perspective. For example, “Resize edit box width to 15 characters” is not a problem. But “It is too difficult to type in long search terms” is. The difference is dramatic. You rarely want to define the solution and the problem at the same time: You’ll usually miss the real problem. In this example, there may be many other ways to solve the problem of search terms, including changing the size of the edit box. But if you are too narrow, you’ll never see the alternatives. Good engineering is all about understanding the alternatives.

For each problem statement, provide supporting information. Include details about which customers have the problem, how it was reported, and even potential ideas for solutions. Perhaps only a certain subset of users have the problem, or it only occurs in certain situations. Say as much as you need to allow others to confirm or challenge your assumptions. If you’re the only one who has seen all of the supporting information (usability studies, market research, etc.), make it available to them. The more open you are about the sources, the less likely people will be to suspect them.

If you work on a team, create a short document that lists all of the problems, and include the supporting information with each one.

Example problem statements:

• It is difficult to navigate from one section of the Web site to another.
• Users have to wait too long for the software to load.
• Our security error messages are difficult to understand.
• The registration page has too many questions, and users often abandon it.
• Finding a specific product on the site index is too hard to complete.

Note: There is a fine line between a user problem and a functional bug. I draw the line based on two characteristics. A problem is a bug if 1) the problem represents a failure of the code to function as it was intended; 2) there is an obvious, simple solution. For example: The failure of a dropdown to list all 50 states is a bug, the inability of users to find the dropdown, or use it to complete a task, is a problem.

Bringing It Together: Lists and Priorities

There is a magic in ordered lists. The act of taking a list of items, and ranking them by priority, defines a release. Without clear priorities, teams fight and argue over what things should be done and what things should be cut. The work involved in setting priorities should be easier with the research you’ve done, but it’s always a challenge.

The best process for creating an ordered list is refinement. Someone on the team works through the list and ranks them based on his or her sense of the overall project—the first item in the list being the most important, the last item the least important. Then members of the team should be able to review that list in draft form and comment on it, allowing the owner to revise and update the list. At some point decisions are made and directives are set. If you’re working alone, it’s usually good to find someone you trust to review your priorities. Again, multiple points of view, filtered by one wise person, tend to bring out the best thinking.

Setting priorities requires the ability to evaluate on at least three criteria: Schedule, Team, and Business. There may be a predefined schedule set for the project, which limits the size and scale of the work that can be done. A problem that is likely to require rewriting half the code-base should not be attempted on a small release cycle.

The makeup and nature of a team defines what kinds of work should be done. What other commitments does the team have? Is there a designer or usability engineer on the team? What skill with Web or UI design does the team have? Last, and most important, are business considerations. What are the revenue goals for this project? Who are the competitors? What advantages will solving certain problems give you? What partnerships can you use?

There may be other considerations involved in your project, but whatever they are, they should be defined before prioritizing the list. The clearer the considerations, the easier it is to prioritize the problems. If you recognize new constraints half way through ordering the list, you’ll have to start over and reevaluate.

Once you have an ordered list in place, you can draw a line somewhere in the list, and separate the important items from the less important ones. How many items make it into the first bucket depends on your schedule.

Where’s the Fun Part?

Once the goals are set and the problems to be solved have been identified, the fun begins. With a framework and direction, engineers and designers should have free rein in generating ideas that solve the problems. Time should be planned to investigate different alternatives that might solve the problems and to run usability studies on prototypes to see if they actually improve things for customers. Then at a defined point in the schedule, potential solutions are evaluated, and the ones worthy of a full development effort, and commitment into the release, are continued. The rest are cut or postponed to a future release. This can work in a similar way to the problem definition process, where certain criteria are set, and a review process takes place.

The irony is that the initial work of defining problems is liberating, not confining. As long as you are working within important problem spaces, you’re guaranteed to be going in a good direction. Provided your problem statements are broad enough, there should be lots of innovative and creative ways to solve them. And even if you can’t completely solve an important problem, a partial solution to the right problem is better than a complete solution to the wrong one.

For more on critical thinking, read part two on idea generation, and part three on project management.

---