By Scott Berkun, Sept. 2005

[Note: since I wrote this my opinion of interviews has changed. Also see Why Job Interviews Are Flawed]

Before you worry about interviewing, consider this: good interviewing does not make a good candidate out of a bad one. The higher the quality of the people coming in to your interviewing process, the higher the quality of those that will come out of it. Therefore recruiting is likely far more important than your interviewing skills.

Do not rely on HR or anyone to recruit for you. The more energy you, as a hiring manager, invest in recruiting, the better your results will be. An organization that recruits well, but has average interviewing skills, might just be better off than a company that interviews well, but recruits poorly. There are many ways to improve candidate quality: the problem is they require time. The three basic things to consider are these.

1. You and your organization’s reputation, through both the things you make, and the way you present yourself to the world, significantly impacts who will choose to apply for your positions. If you have difficulty finding good applicants, based on the reputation you and your organization has, there might be good reasons the best candidates aren’t heading your way. Deceiving people into making your organization seem different than it is rarely works: they’ll find out once they’re hired, they won’t stay long and they’ll tell everyone they know about the deception.
2. Make sure you are investing energy harvesting leads from your networks in your industry. Ask people you trust or who have connections (and perhaps owe you favors) to find people for you. One major reason for managers to attend conferences and participate in professional organizations is to help them build and maintain networks that come in handy when looking to hire.
3. There is no law that says job descriptions must pervert the English language into clinical, jargon filled, oxymoronic verbiage. Good job postings read like an actual human being wrote them. They have a voice and a point of view. They express something of the personality of the organization, or the hiring manager, which are all good things to do if you want to attract good, creative, smart candidates. Generic, formulaic, cheesy, HR-speak, job postings will tend to attract generic, formulaic, cheesy candidates.

How to use 60 minutes: 5,25,25,5

Lets assume that candidates have been selected, and an interview loop has been set up. Your task, and the task of others in your organization, is to use your 60 minutes with the candidate in the best way possible. An hour is not much time to decide a person’s fate (“Answer me these questions five, I mean three”). Have a plan for how to break out the hour so that you’ll get what you need to make a fair evaluation. I’ve used the basic framework of: 5,25,25,5:

• 5 minute warm-up
• 50 minutes for two big questions/problems
• 5 minute wrap up and answer any questions they have

Get them warmed up – It’s important to see a candidates best stuff first. If they get a bad start because you made them uncomfortable or defensive, you will never see their potential. Be nice at the beginning – you can always get more confrontational as things go. Ask them something from their resume – if they have a website, walk through it and ask them questions. Or ask them to pick one cool thing from their resume and describe it to you. Help them to dominate the conversation, get comfortable, and get their energy up. Try to spend only 2-3 minutes here, unless you hit on something interesting.

Frame it as working together – Will the job involve solving problems with other people? Ok, then. That’s an important thing to explore in the interview. A brilliant rock star programmer, who can only work in complete isolation, is probably less valuable in the long run than a good programmer who adds value to the work of everyone he gets in a room with. Being smart is rarely enough on many teams: you have to be smart and be comfortable communicating your smartness to others, exchanging ideas back and forth, not just forcing them all in one direction.

Include in the set-up for problems something about how the candidate will be working with you on some project. “Lets say you and I create a video game start-up, and we want you to build the AI architecture given these 3 constraints. Go.” or “We are reorganizing our team to solve these 2 problems, and put you in charge of the effort. How would you start?” Act as a wise, but distant guide through the problem. It’s ok to answer questions (in fact it’s a good sign if they ask good questions), but only help them to clarify assumptions. Don’t do the problem solving for them, unless they’re completely stuck and need a nudge to move on. Remember, you can make it less collaborative at any time, and in any way you choose, but if you start as an adversary, it’s difficult to go the other way.

Get ready to keep your mouth shut – The more time you spend talking, the less time they spend demonstrating their talent. So while at first you should act as a host, introducing yourself and getting them warmed up, once they’re in the middle of the problem keep track of how much talking you’re doing. As I’ll cover below, you often only need a sentence or two to nudge them along. If you’re talking in entire paragraphs during the heart of the interview, make sure you’re doing it to give them information they need to demonstrate for you the skills you’re looking for. It may be ok to be chatty and tell stories, just have a reason for it.

Keeping score

There is no sense interviewing candidates if you are not clear on what you are looking for. You can’t score or evaluate a candidate in the abstract: you need to have an ordered list of traits you are looking for. Put these in two piles: Traits for fitting in the team/organization, traits for succeeding in this specific role (It can make sense to get rough consensus on these lists within your team or with your peers, since individuals will have different ideas about what those traits are, and you don’t want to debate this higher level stuff when making a decision about a candidate). Hiring criteria will help you to sort out what the strengths and weakness of a candidate are, help people on the interview list to discuss their opinions, and evaluate if there are other opportunities in your organization that they might be better fits for. Interview loops are time expensive: if there are ways to reuse the investment your team has made in interviewing a candidate, use them.

From the outset, the candidate’s score should be zero. Anyone that gets into the interview loop should be starting from about the same score. The whole point of the interview is to get at all of the things you can’t capture on paper, mainly to observe their performance in certain situations. This is the closest you’ll ever get to seeing them work before hiring them, and the bar should be high. This not only places a burden on the candidate: it places a burden on you. You must make sure that you are doing everything in your power to seek out the talents the candidate has. You should think of yourself as a talent scout, who is trying to frame problems and questions so that if talent/skill is there, even latent or hidden talent, you will find it in the course of the hour. This is the only attitude that maximizes your chances for making a good hiring decision. You do not need to be confrontational or terse to give a tough interview. On the contrary, it should be the quality of the challenges you give the candidate, and not your demeanor, that makes the interview tough.

Good interview questions for problem solving

In this general primer, I’m assuming that problem solving ability in whatever domain the job is, is one of the most important traits you’re hiring for. This can be for managers, programmers, designers, whatever. As long as the ability to solve problems is what you’re looking for, this advice applies regardless of the specific job. Of course, other important traits are communication skills, conviction, intellect, passion, sense of humor, integrity, working style, maturity, and experience. So while some of the advice on questioning for problem solving applies to them as well, it only goes so far. (If I get enough requests, I’ll follow up on interviewing specifically for these other traits). To explore problem solving skill, use questions that have:

• multiple approaches to good solutions
• multiple answers that you would consider “correct”
• many aspects that force the candidate to communicate, describe and evaluate different ideas (preferably with some whiteboard work), and ask questions
• clear objectives, but no obvious paths to correct answers
• have different levels of complexity that you can bring in, or remove, at different points
• are fun/interesting/challenging for the candidate

Use real situations over abstract brainteasers. Many programmers and tech-sector folks have a habit of abusing puzzles as a tool in the interviewing process. Puzzles can be great tools for exploring different kinds of intelligence, but they have to be handled carefully – many of the popular puzzle questions are easy to misuse. If you use problems that are very abstract and heavily logic based, you run the risk of measuring a candidate’s ability to make logical abstractions, instead of their ability to solve problems and get work done. It’s much better to pick an algorithmic, design or management problem (as appropriate to the position they’re interviewing for) from your current work, and ask them to solve it (Change the names or details to protect the innocent).

Unlike brainteasers, if the problem comes from a real situation, you’re unlikely to get lost in abstractions. If you insist on using brain teaser like questions, or trite HR type interview questions (“What are your 3 weaknesses?”) be clear on what skills or information you are trying to uncover. Remember that the focus should be on their thought process, and their ability to communicate their thinking as they go along, not their ability to achieve a correct answer. This is not a spelling bee. Problems involving 5 people on a boat, three light switches, Monty Hall, scales with different weights, or trains at different speeds are good examples of brainteaser problems that won’t reveal much when handled poorly.

Give them a sense for what you want to see. Always be clear with candidates about two things: the process and the objective. When you explain your question or problem, explain the process you expect them to use. “I’m looking to see both how creative you are, and how well you evaluate your own ideas. So it’s ok to come up with lots of ideas, and throw some away if necessary. And it’s ok to disagree with me – I may just ask you to explain your ideas in more detail. Does this make sense?” Set the ground rules and make sure they understand. This makes it impossible to miss good candidates for the stupid reason that they didn’t know what you were looking for. If you want to see if they will naturally disagree with you, or naturally pursue a certain style of problem solving, that’s fine.

You can hold off on informing them, and see what happens. However if 5 minutes go by and they’re clearly not doing what you want, let them know. Remember: they are relying on you to inform them about what the rules are, and how to score points. Interviews should not be guessing games. Be up front with them about what you need, just like you will be when they’re hired an on the job. The objective you want them to go for should be absolutely clear, yet without giving hints about specific paths to a solution. Something like “Your task is to design an easy to use toaster oven for three legged arthritic German shepherds living in the Antarctic. Go!” or “Write out an algorithm for re-ordering a linked list in place, while standing on one foot, blindfolded, with your dominant hand tied behind your back”. Even if extremely challenging or ridiculous, the victory condition for problem solving questions should be understood.

Actually have them do real things. If you are hiring a designer, give them real design problems. Programmers get programming problems, and managers get management problems. Don’t let candidates get lost in abstractions “Well, first I’d do research, and then I’d talk to everyone, and then I’d sit down and come up with 3 ideas…” That’s talking about problem solving, not actually doing it. Give them real specific tangible situations, and ask them to give real specific tangible solutions (as tangible as possible given the problem). Make it clear that they can skip the details. Pseduo-code or object level design is fine. Rough & ugly whiteboard sketches are fine. Let them know you’re after the heart of the thinking of solving the problem, not the detail level skills (If you want to assess detail level skills, do it separately.). Pick your problems with this in mind.

Let them make mistakes. If they go down a bad path, give them the chance to discover it for themselves and recover. This is hugely valuable information about a candidate: do they evaluate their own work well, and self-correct? Don’t smirk, smile or give away your opinion – let them work on their own. Sometimes you’ll hear the most interesting and creative thinking from candidates when they are headed in the wrong direction. However, don’t let them spend half the interview in a rat-hole. When they’re stuck, note it, and then give them the smallest possible nudge in the right direction. This can often be simply asking a question. “Why did you do X, but not Y or Z?” But be careful. The more direct your question, the more they’ll assume you’re giving them an answer. Any intelligent candidate will look into your questions to see if they can find hints for answers. So prod cautiously. “I’m not sure if it’s good or bad, but I’m curious why you’re going with Y. What about Z? or X?” If they still don’t see what they’re missing, continue your probing as a conversation. “Well often I’d prefer Z over Y in a situation like this. What do you think?” Try to gradually increase how much information you give, so you can learn the true point at which they were stuck at. Only abandon a question or problem after you’ve poked around it for awhile.

Look for good process and whiteboard skills. Keep an eye out for how candidates start the problem. I give points for candidates that start out by listing assumptions, or asking questions to help narrow the challenge – especially if they are management, engineering or design candidates. It shows that they are thinking broadly and logically to help frame the problem, and seek out boundaries to limit scope. Good problem solvers tend to look around a little bit before unleashing their horsepower so that they know what their aiming for. However, some candidates get lost in boundary seeking – More than a couple of minutes is excessive, and hints at someone that’s a little scared to jump in. Nudge them forward if necessary.

Some candidates seem scared of the whiteboard. Encourage folks to use it, especially if your team culture uses whiteboards often. The whiteboard is one of the most effective communication tools teams of people have, so on my teams its always been an important skill for people to demonstrate in interviews. All I mean by whiteboard skill is the ability to take a complex abstract idea, go to a whiteboard, and use some kind of visual representation to help communicate their thoughts to others (Funky squiggles and wacky diagrams count, if combined with what they say it gets me to understand the idea). A good interview trick to get a shy person to use the whiteboard during an interview is to draw something horrible on the whiteboard to represent what they said, and say “you mean like this?”, and then offer them the marker. If you drew something sufficiently bad, they’re guaranteed to grab the marker.

Use restatements and prodding to help examine their thinking. Use questions to force the candidate to articulate their ideas. “So wait a second – let me see if I understand. Why did you use that algorithm there?” You can also use restatements for similar effect “Let me see if I understand. So you would cut that item from the spec to help the team move faster?” – Don’t lead them, but force them to step back and communicate what it was they were trying to achieve. Challenge the candidate. Probe. Push. “That’s not bad, but is there a better way to do that?” That’s what you’re there for. To keep them honest, use erroneous restatements at least once. Ask a question or lead in a direction that you know is wrong and see if they follow you. If they do, call them on it by correcting yourself (“Oh wait a second, I’m not sure that’s right. What do you think?”). See how they respond. You don’t have to be mean about it, but you do want to probe for conviction and honesty (note: if you gave them a sense at the beginning that it was ok to disagree with you, you should be concerned if they don’t when you suggest completely stupid and insane ideas). See if they’ll stand behind an idea that they clearly think is a good one. This doesn’t have to be slimy or manipulative: if you’re creative you can come up with other ways to get at the same qualities in a candidate.

Use time as a relative guide. Watch the clock for each problem you give. One of the reasons for 5,25,25,5 is you get landmarks of time (timemarks?). If they’ve been communicating well, and they’re smart, they may finish a “25 minute” problem in 15 minutes, If they do well on both problems, I’ll have a smaller problems/questions ready to go to make sure I see as much of them as possible in the hour. If 10-15 minutes into problem #1 they’ve gotten nowhere, I’ll move on. I want to make sure that their failure isn’t tied to something odd about the specific question I asked, so I’ll try to move on and give them other questions/situations to try. If they don’t shine in the second and third problem, it’s pretty much over. Do your best to keep going with other situations to see if you’ve missed something.

Use familiar questions as a measuring stick. Using the same problems on different candidates can help you develop relative measurements. If you use the same interview question with different candidates, you’ll get a feel for common solutions, rough spots, and good timing and content for prodding and questioning. Eventually you’ll develop a stable of questions you’re comfortable with for different kinds of jobs, situations and candidates. However, don’t give up on seeking out new questions. If you’ve been using the same questions for a few years, sit down and think up some new ones. At Microsoft, where waves of campus candidates came annually, interview questions were often shared with friends when they got back to the dorm (and some websites catalog them, such as these guides to Interviewing at Microsoft, Interviewing at Google and getting hired there. A good percentage of the advice on these sites applies well to the entire tech-sector). To prevent the situation where candiates have already heard the questions before, I’d come up with 3 or 4 new problems every year (When using an old problem, I’d often ask if they’ve heard it before. I’ve found people tend to be honest). It’s a fun exercise in solving the problem of evaluating problem solving skills. For fun, you can test run interview questions on peers or people on your team to help you calibrate their difficulty.

Finishing the Interview

Make a good handoff. You are not done with the interview until you’ve spoken with the next interviewer, and they know where the candidate is. There is nothing worse than an abandoned candidate sitting for hours in a lobby. Even if they get an offer, they won’t feel good about how they were treated. To avoid this, make handoffs yourself. Take them to the lobby, or inform the receptionist/whoever might be able to watch them. Give them specific instructions about what is happening next. Give them your business card, and write down the name and phone number of the HR person (if there is one) involved, and the name of the next person they’re supposed to talk to. Reduce to zero the odds of them getting abandoned or forgotten. As soon as you get back to your office, call the next interviewer. Give them a 30 second summary: “hire, strong problem solver. A little awkward communication style, but good personality and good fit. She has a very broad perspective for a new graduate and seemed energetic and motivated”. If they are not in their office, call their cell. If still unavailable leave them voice mail.

Alternatively, if the next interviewer is nearby, forget the phone call and just drop by their office to give your summary. But be warned: never ever, bring the candidate straight to the next interviewers office: they may not be ready for it, you miss your chance to pass on any feedback, and it can be awkward for everyone involved. Pass on your feedback immediately. Always block off 5 or 10 minutes after the interview to write up feedback. Each interviewer benefits from all of the feedback from proceeding interviews, if for nothing else, to avoid collisions with interview questions (Smart teams often chat before the interview loop, and plan out who’s going to go after what attributes). In your feedback you do not need to provide a play by play. Instead, behave as if you were the only decision maker. Should we hire this person or not? What are the 3 or 4 key things you observed in the interview that support this decision? What are the intangible things you felt that were important? What would you have done if you had another 30 minutes with this person (aka what should other people spend their time on?). The 5 minutes it takes you to write up a short summary will improve the quality of the remaining interviews on the loop.

Things not covered

This topic is much larger than a single essay. Here’s some topics that made it into my notes, but not any further. I list them as fodder for the forums or future essays. If you know of good sources that hit on the topics in in this essay, or in this list, let me know and I’ll add them.

• Making good hiring decisions (What to do when you’re on the fence and the hiring manager perspective)
• Evaluating other attributes (communication skills, conviction, intellect, passion, sense of humor, integrity, working style, maturity, and experience)
• Interviewing for management/leadership roles, not line level roles.
• Teaching people how to interview others
• Deciding who should be on a given interview loop
• How to deal with interview burnout for your team
• More on recruiting candidates (industry and college)
• How to make offers and sell candidates
• The evil and bad habits many experienced interviewers have
• Interviewing horror stories (both sides, interviewer and interviewee)