The data death spiral
Whenever data is misused as the only means for making decisions, a death spiral begins. Blind faith in data creates a false confidence that overwhelms reason. Decision makers forget their wisdom and wait for numbers, fueling an addiction to unnecessary, biased and distracting data. Cowardly decision makers thrive in unnecessary information, their fears can hide there, and the data can be used to the wise with blankets of endless numbers and facts of dubious value.
Data can be good: it can dispels ignorance and raises questions. But data doesn’t make decisions or find good ideas: people do. As soon as you discount people in the name of data, the downward spiral begins.
You know you’re in a data death spiral if:
- Confusion over what the hard part is. If you are working on anything challenging, data will only define the landscape. It won’t dictate a particular strategy: good data has different interpretations. If you present data to justify a decision you are emphasizing some data over others: which is an act of creativity and opinion, not fact and figure. Opinion is everywhere and is most dangerous when it’s hiding behind spreadsheets.
- No opinions are considered without data. Opinions are good if they come from smart thoughtful experts. If you are in a world where you, as an expert, can’t make obvious improvements without 10 pages of supporting material, guess what happens? Nothing happens. People spend all their time defending the obvious and the scale of work, and the energy to improve, drops dramatically.
- Creatives are hiding in the corner. Good designers and creatives know how to make things. Lots of things. They experiment and explore. When used properly they are a piston of progress. But if the landscape is data and analysis obsessed, creative types are relegated to refinement work, the least leveraged use of their talents. The team gets a fraction of their possible value.
- No one questions data. Not all data is the same. To collapse data down into “70% of our customers prefer green” requires several layers of simplification of the truth. Which customers? What alternatives were they shown (were there just 5 kinds of green)? How many customers were there? If no one every questions the data and understands its nuances the gates are open for people to throw more misinterpreted data at each other. You need data quality not just data volume.
How death spirals start
The spiral begins with ignorance. Leaders confuse the collection of research with thought. People who throw more data (not better) at problems are rewarded and others follow. Soon no idea can be suggested without a data armory. Meetings are data battlegrounds. Or worse, data massacres. When someone says “Morale is low. People are crying in the halls. We must do something.” another says “but where is your data?” and the conversation ends.
As the spiral bottoms out, people are proud of numbers and slide decks: not the results.
Steps to preventing death spirals
- Separate good data from bad. Good data self describes its biases. A smart person uses data with consideration for what things the data can’t tell you, and they include this understanding in their story. Encourage people to play devil’s advocate on their data and make data inquiry part of the discussion any time data is presented. Diffuse data obsession.
- Data and ideas are partners. Make sure developments happen both ways: ideas that are pulled out from data, and ideas that come from minds, with supporting data found later. At its best it’s an itterative process: you have ideas, you research data, you refine ideas, you research more, etc.
- Instincts matter. The Gladwell book Blink explores how our instincts serve us: are the instincts of your group serving you? Layers of process and data bury our hearts. Some people have better instincts about certain decisions and you want to harness that talent, not dilute it. It can be fine to say “Fred has the best perspective on this. It’s his decision.” And allow Fred to define how much or how little research is needed.
- Let go of the fear. Many people collect data to defend their choices should things go wrong. When bad things happen they point to the data and say “See! We did the right thing!” despite the results. This kind of paranoid, self-protective thinking weighs on a team: instead of ensuring success, people are protecting their asses. Most of us can smell it when a leader is in this mode and it puts the entire project on its heels. Insurance is for the birds: A good manager earns the trust of his team and lets them know that even in failure, he’ll take responsibility for what was done.
- Kill contrived group think meetings. Data obsession and consensus-driven teams go hand in hand. It’s on big, cowardly teams with leaders who demand everyone agree before decisions are made that people show up armed to the teeth with handouts and marathon presentations. Not everyone needs to agree to everything. Define who is in the best position to decide and give them the power. If necessary, pick a handful of people who disagree to go off and fight it out in private, reporting back when a decision has been made.
- People are better than data. Much of a person’s talent is difficult to capture in numbers. Their experiences, instincts and passions all factor into what makes them good at what they do. As a manager, I want to make sure I have channels open to these assets, channels that don’t require the same kind of data in order for me to listen. If I shut these things out, I’m denying myself the benefits of true expertise. If I’m working on something hard, and with tight deadlines, the more good decisions I can have my team make without days of research, the better off our team will be.
- The Danger of Faith In Data
- How to lie with statistics
- Chapter 8, How to make good decisions, from Making Things Happen
Really enjoyed this post and in 11 years of info management consulting, couldn’t agree more. Gathering good data, making a business case, thinking through a project approach, etc. are all important and have their place, but I’ve experienced how the search for consensus can lead to a lack of results and a staff sapped of energy and momentum who can’t remember what it was they were once so good at and passionate about. Data is important for establishing where you’ve been and where you are now, but it takes insight and vision to figure out where you’re going, to separate tactical from strategic, and to guide the energy of project staff. None of that is a job for data.
Infusing geekdom with humanity? Unexpected and welcome.
I came to read the cross-topical essay on software suckage(recommended by friend Bryan Zug), and was impacted greatly by this post.
Thanks Scott. This gives me some supportive data to use in my next project meeting. (just jokes man)
Great post, spot on observations and very well written. Thank you!
I want to plaster this all over my bosses offices. Shame is, they’d think they’re already doing all these things.
I think it’s all about CYA. Companies try to reduce risk and improve effectiveness by looking at data, and that is good. What happens though is that looking at data is not a substitute for good judgment, and when decision-making processes are data-driven rather than judgment driven, you’re going to have pain.
JohnO: I know what you mean. For any new information to be accepted the person reading has to open minded, at least a little, that there’s a better way to do be doing what they do.
How you get people who have trouble seeing clearly to see is the real challenge underneath all of this advice and expertise we kick around.
Excellent piece, timeless issue, and interesting to see how it plays out in different cultures. This piece on Google’s design practices is an interesting example; on the one hand, the company has access to immediate feedback and uses it; on the other, recourse to “taste” doesn’t seem to be a part of the process.
“When a company is filled with engineers, it turns to engineering to solve problems. Reduce each decision to a simple logic problem. Remove all subjectivity and just look at the data. Data in your favor? Ok, launch it. Data shows negative effects? Back to the drawing board. And that data eventually becomes a crutch for every decision, paralyzing the company and preventing it from making any daring design decisions….Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better.”
Great post, I can tell you put a lot of thought into this regularly. I especially like your last point – “People are better than data.” Very important to remember. :) We’ll post a link to this entry from our blog on Friday!