Why Job Interviews are Flawed
Why does anyone believe they are good at interviewing people? If you ask experienced managers you’ll often her a set of well worn pet theories about how they interview people. Yet most interviewers, no matter how experienced they are, make instinctive judgments based on cognitive and other biases. And then afterwards, when reporting on the qualities (or lack their of) of the candidate, they back-fill logical reasons to support the intuitive responses that they’re in denial about. It’s no surprise that most job interviews are deeply flawed and unfair experiences.
Contributing factors include:
- Few are self-aware enough to sort out their biases. Few people posses the self awareness to realize why they instinctively like or do not like someone they’ve just met. And even fewer, especially in business and engineering circles, feel comfortable with their feelings. It’s considered unacceptable to say ‘the guy did well but I didn’t like him for reasons I can’t explain’. It’s much easier to hide that feeling inside unfair judgments, using whatever flavor of corporate jargon can be found in the official hiring criteria (Lacked intellectual horsepower, couldn’t deal with ambiguity, didn’t know the secret handshake, etc.)
- Talking about doing is not doing. Most interviewers focus on trying to extract a prediction about someone’s ability through having them talk about their ability. This is ridiculous. Could you evaluate an NFL running back by asking them questions about how they run? (e.g. “I run really really fast”, “Great, you’re hired.”). Better interviewers work hard to put candidates in problems and situations like the real ones they’ll face, and watch. They collaborate on real problems during the interview, as that’s what much of work is. Over time they’re able to calibrate what it means for a candidate to do well, given real problems, in an hour. But this requires skill and patience few interviewers have. And even when they do, the candidates are in an awkward and artificially stressful environment that does not approximate real work well, unless the interviewer is diligent on compensating for these issues (tip: hiring candidates for a trial project is often a better use of everyone’s time. Get a sample of them actually doing the job).
- Interviews work better as a filter. The job interview loop is more effective at eliminating bad candidates than identifying good ones. The bet is by the end only good candidates remain, but that’s not true. Like bacteria responding to antibiotics, strains of bad candidates that are immune to your process survive as well, and are hard to distinguish from good ones. The process can be prone to false negatives too (people who get rejected but would have thrived).
- Recommendations are underestimated. Since interviews are mostly bullshit, it makes sense to put more weight on a recommendation from a trusted person (not necessarily the names on the candidates resume) who has worked with the candidate somewhere else. They have first hand experience on the millions of things that can only be witnessed outside of the interview room. If you trust them, and they trust the candidate, that may have more predictive ability than 60 awkward minutes in your office.
- No one else saw what happened. Interviewers are free to lie and distort, intentionally or not. All interviewers are free to invent pet theories on which questions work best, or how good they are an extracting the value of a candidate. They are the only record of what they asked, how they asked it, and how the candidate performed. If they have bad habits that bias the candidate, no one will ever know, as the candidate has almost no ability to report on the interviewer. Every interview is a cat and mouse trapped in the room, and the mouse is motivated to do whatever it can to survive the cat, no matter how cruel or unfair the cat is.
- We never go back a year later and evaluate. The hiring loop at nearly all companies is broken, as there is no feedback loop. No one forces you to go back 6 months or 2 years later and see: how many of the hire decisions you made worked out well, and how many of the people you rejected kicked ass at other companies with similar cultures and needs. With no data, the value of any interview process is guesswork, not rigor.
Despite my affinity for this theory, I believe groups that take interviewing seriously, and leaders who reward interviewers for putting more time and careful thought into interviews, end up with better teams. The choice to hire someone is the most important decision you make that month, or year and the wise know this. At a start-up it can make or break the company. And the more seriously people take the process, even if it’s flawed, the higher the odds they’ll recognize the natural shortcomings above and invest in minimizing them.
While I don’t think all interviews are a crapshoot, I agree with the Winchester theory – in most interviews, most of the time, it’s mostly bullshit as a tool in truly evaluating how well a person would perform in the job, even if the people doing the interviews don’t intend it to be.
The real story is in who you recruit to interview in the first place. Better candidates in, better candidates out: see my essay how to interview and hire people. What is your take on interviewing? How do you work against the Winchester theory?
[This post was inspired by Royal Winchester who, when asked about interviews, offered: they’re mostly bullshit]
Good points. I’ll reread and six months to evaluate.
I’ve had that theory for years now (including four or five of the contributing hypotheses noted). I didn’t realize I should have been letting you know! :)
As a department head, I tried to get back with the individual interviewers six months to a year later on the occasional less-than-stellar hire to figure out what we missed. However, I was less than rigorous about this, which was a mistakes, as you point out. I was probably most rigorous with myself, since the less-than-good hire was affecting my success personally through my team.
There were a handful of misfires, but I feel good about most of the people I hired over the years. I was able to get recommendations on most of them, and I had a couple of people on my team who had really good sensitivity to something “off” about a candidate, which helped.
These days it’s very hard to get recommendations from former co-workers outside a company. In this litigious society, many companies will not allow employees to speak about former employees except to confirm dates of service.
For what it’s worth, 20 years ago, I gave a good recommendation to another company about someone I’d worked with, and the person promptly turkeyed-out on the new job, an incident I’ve never forgotten… though I’ve never been sure quite what lesson to draw from it. One thing is that I was always up front with someone from another department asking about an employee on my team; I never tried to “shop a problem” by concealing something about an employee I was unhappy about.
I think most people have no clear idea of how to interview someone effectively. Even those using a point system and psychometric testing have little idea of what kind of person they actually need or want.
In my area of project management this is even more of a problem since you can’t give someone a technical test, and qualifications are not widespread. So many of my peers are asked questions about the project management basics or project manager duties none of which tells you much.
Luckily I’ve reached a level where I’m usually only asked one question, “can you turnaround and deliver this failing project?”. But it begs the question why so many projects are in failure mode when apparentely the project manager passed the interview with flying colours!
Regards
Susan de Sousa
Site Editor http://www.my-project-management-expert.com
Steven: I’m convinced there are too many variables to ever isolate why one person, in one job, does great or fails.
A personal life crisis can arise for a hundred reasons that have nothing to do with their abilities, but destroy their performance.
Or a a personality conflict can demoralize someone sufficiently that they appear to be a turkey, when they’re really an eagle, or a hawk, or whatever the opposite of a turkey is.
But the fact that you can’t isolate the causes, good or bad, doesn’t suggest they should be ignored. Even though there is no answer, the teams that are rewarded for thinking about it, and investing energy in it, will, over time do better.
I’d like to see more companies fix bad hiring quicker. Get rid of the lemons! Help them leave, find them another job they’d be better at, performance manage them out; whatever works, for goodness’ sake make it happen!
And when you strike it lucky with someone good, do what you can to hang onto them.
This combined should increase the quality of people you have over time, and hopefully these better people will find more better people more often.
Someone mentioned in email it’s good to filter out people based on their chemistry with the team, or with you, and I agree.
The problem is it’s hard for most people to discern between positive reasons they like someone, and negative (e.g. they share your bias or bad habits or something). We just mostly sense we like them, or they agree with us.
I once had an interview where I figured out very quickly that the interviewer loved to talk about himself.
I answered every question he asked with “Yeah, I’ve had experience with XYZ. I’ve done this with it and, as I’m sure you know yourself, found X to be very interesting…” or
This was enough to trigger another 30 minute diatribe on why he was so awesome at X. I spent the whole time smiling and nodding and laughing at the appropriate moments.
At the end of the interview, he said “I think you’re going to fit in perfectly here” and I got the job.
This is actually very encouraging for me, because I freak out from the pressure on technical questions in interviews, but I think I’m pretty good at making a good first impression. So, right on!
It is frustrating that with so many professional psychometric tests on the marketplace companies are still using subjective interview processes.
The formula is simple:
Step 1 – Define Your Target – What will high performance in the position you are filling look like?
Step 2 – Define Your Potential – Which candidate fits the high performance requirements?
Step 3 – Teach the manager that the candidate will report directly to how to leverage this information, removing all obstacles to high performance, and watch the individual excel.
Using a psychometric test that is accompanied by exert interpretation will help you to follow this 3 step process that leads to success.
I don’t think a lot of people work against the Winchester theory (this is the first time I hear about it anyway).
A few months ago we setup interviews to hire 2 programmers. We were 2 persons doing the interview and if any of us feels (note the word “feel”) that the candidate will not fit then we’ll thank him/her for the time, and that’s it. What’s interesting is that we ended hiring up someone, but that person elected, at the last minute, to work somewhere else, so we tried with the second best, whom I thought would not fit into the company’s culture and dismissed earlier. What’s interesting is that against all my expectations/hunches/expectations, that person thrived in the company.
Sometimes giving someone a chance, even if against your bias, might work. This is definitely an exception to the rule IMO, but again, I might be wrong.
PS: I did publish a collection of 300 PM interview question in case someone is conducting a PM interview.
Quick note, I just read Scott’s comment about the chemistry between the candidate, the manager, and the team. Well in the real life example I gave above, that chemistry did not exist during the interview (in fact I thought that the candidate was a bit rude, and I was wrong). It is important to note that during interviews a lot of candidates are not really themselves, they might lie in ways that might help them get the job (and later hurt them while in the job) or vice versa.
This is a good article, thanks.
I really enjoyed this article, which made me recall an incredibly exhilarating interview process I went through once. I am an engineer, and went to interview for a new position. I spent the whole day in a conference room with various engineers from the company coming in and asking me how I would solve real world problems they were facing or had faced in the past. As soon as it appeared that I already knew the answer to any specific problem, they changed to something new. They were genuinely interested in how well could I think and thence express my ideas. I have to disclose that it was for a consulting position, so they needed to make sure that they would not be embarrassed by something dumb I might say in front of the clients, so that needs to be factored in. I got the job and had a great experience working closely with many of the people who interviewed me that first day.
This theory would be more useful if it were accompanied by a better idea for sorting through people to determine who to hire.
In lieu of a better system, even the gut-feeling judgments most interviews result in beat the alternative – hiring someone blindly.
Very interesting post. I do a lot of interviews of potential post-doctoral science trainees to join my lab.
First I interview the candidate one-on-one for about an hour to see if they are capable of thinking on their feet while discussing science. Then they deliver a formal “job talk” seminar to the entire lab in which they describe their PhD research. I and the other trainees in the lab ask lots of tough questions during the seminar to try to knock the candidate off message and see how they respond.
Finally, the rest of the lab takes the candidate out to lunch without me. This has a number of purposes. (1) The other lab members can figure out if the candidate is going to be able to be a good addition to the “team”. (2) The other lab members can tell the candidate the real truth about how our lab works and about my leadership style. This is important, because different trainees respond better or worse to different mentoring styles, and while I do have some dynamic range available to me, I do have my own style. I wouldn’t want someone to join my lab thinking they were going to be mentored with a style that I am incapable or unwilling to deliver. (3) The other lab members can tell me if they think the candidate is intellectually sound.
This method of combining an interview with the “manager”–me–and a less formal interview absent the manager with the other members of the team works really well.
This is the best line I’ve read for a while:
…Like bacteria responding to antibiotics, strains of bad candidates that are immune to your process survive as well, and are hard to distinguish from good ones.
and explains how some people managed to get hired by some great firms.
I remember a public speaker talking last year about employers not hiring the right people, simply because they didn’t like them. “There’s a person who could turn your company around, but you let him or her go,” he said. (I cannot remember the speaker’s name, unfortunately.)
But we want to work with people we like. It is very difficult to overcome that preference. Many people, I think, are also reluctant to hire those they think are smarter than they are. They’re okay with maintaining the quality status quo. Few people want to be shaken up in their thought or work processes. Also, if you make a bad hire of a super-intelligent person, your life will be hell. It’s a risk.
How do I work against the Winchester theory? I’ve stopped applying for jobs I don’t want. I evaluate potential bosses and co-workers on the basis of whether I would be inspired, motivated, and happy to work with them.
Over the years I’ve been the candidate a few times and been the interviewer at least a couple hundred times. I’ve made great hires, good hires and a couple awful ones (mostly engineers and various technical types). My thoughts about them paraphrase Churchill on democracy–terrible way to do business but the best humans have invented so far.
To protect a little against the weaknesses, I always do team interviews and include as wide a range of people as I can while keeping the group(s) reasonable. I’ve learned to listen very, very carefully to the feedback I get from the others and to pay attention to that “gut feel”.
And yes, there have been times when I knew 5 minutes into an interview we were or weren’t going to hire the person. Sometimes I consciously knew why and sometimes I didn’t–chemistry? fit? magic? got me but it was real.
It’s hard though because some folks interview well and some don’t and that’s not a very reliable predictor of success in the job.
Still, I think a well done interview can tell me a lot about the person’s energy and enthusiasm. It should also tell them a lot about whether the environment and the work will generate that energy and enthusiasm–where those line up, there’s a reasonable chance of success.
Re: some folks not interviewing well and also the idea that as well as false positives there are false negatives, I am one of these false negatives. I don’t interview well because I am not ‘confident’. It’s a purely physiological thing: if there is so much on the line and I’m being scrutinised in a hostile manner by a stranger who only cares about whether I stare into his eyes and spout bullshit, I stutter and look abominable. However, academically I’ve only ever had top marks, and whenever I’ve managed to get a job without the hurdle of The Interview (e.g. because I sang on some guy’s radio show and then he needed an office drone and had figured out that I wasn’t an idiot because I wasn’t abjectly terrified when we met), I’ve solved problems, cleared backlogs, made friends with colleagues and so on. I have skills – learning and applying the information I learn – but those do not include the skill of interviewing well, and I’ve never managed to change that. I’ve experienced a couple of situations where I knew the individuals who got jobs that I had also applied for: they were Brimming with Confidence during the interview and performed abysmally after being hired! Therefore I definitely support trial projects rather than the ghastly, nerd-repellent interviews. I’m sorry, but for some jobs, nerds are needed, not cheerleaders.
This is so true. I’ve been saying this for so long. Interviews have nothing to do with the actual job how well you are at talking about your work.
I’ve become convinced trial projects are a better use of resources and time. Hire people to do a week or two week long project, and hire based on their actual results. People who don’t perform are worth what you pay them (you learned it before you hired) and the time spent managing the project ensures those who do well, can do the actual job well.
That’s how it’s been done in Europe forever – 6 weeks trial period.
It creates extra energy for the new hire to do their best, and also lends the opportunity for both of them to figure out synergy, which in the end might be more important than job performance itself (especially if smaller company size or smaller teams)
I don’t know about biases but basically during the interview I’ll be watching for the person’s character and try to figure out what kind of a person he is by his attitude and tone as well as how much we’d get along. It’s about his ability to work under the pressure of being scrutinized by his future boss more than just telling me what I want to hear.
Hacking the (technical) interview:
– Anchoring
– Mirroring
– Syncing
List of types of lies from job Candidates:
http://www.tlnt.com/2013/07/26/how-job-candidates-lie-and-the-best-lies-they-always-seem-to-tell/
I’m so glad I found this. I’ve been looking for solace since last week after the total sham of an interview I had with a big corporate here in the UK. After prepping for a competency interview, the interview itself was completely different. A 45 minute affair in which the interviewers spoke for 25 minutes about what they do and then proceeded to ask me to present my portfolio. This I did, and to be fair I thought the whole thing went well considering. However after not bothering to get back to me and only after getting an automated response I demanded some feedback so spoke the the companies recruiter who then dished out scores for my competency interview. I’ve never heard so much rubbish in my life, snap judgements that could not be made in the interview time we had, a rather insulting ‘lacks flexibility’ in a portfolio that consisted of album sleeves, illustration, websites and mobile apps, and that my work was too corporate, work that’s been carried out for some of the biggest companies in Europe. This post says a lot, and probably explains what goes on behind the scenes when they can’t just be honest. Needless to say the interviewer hiring manager fled to a holiday afterwards.