Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

The Joel Test question #11 is: "Do new candidates write code during their interview?". What are arguments for and against to ask new candidates to write code during the interview and to make a decision upon it?

share|improve this question

closed as primarily opinion-based by Jim G., MichaelT, GlenH7, Robert Harvey, gnat Jul 7 at 22:09

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise.If this question can be reworded to fit the rules in the help center, please edit the question.

    
Write code as in physically write it with a pencil and your hand or write as in type code out on a machine? –  Chris Nov 24 '10 at 18:10
    
The main con is asking stupid or useless questions and thinking you did it right. Such as the someone comment about being asked to write a linked_list. –  acidzombie24 Sep 9 '11 at 9:51
add comment

14 Answers 14

up vote 13 down vote accepted

I don't see the cons. An interview has many parts, and a candidate should be endorsed 'up the chain' a few times.

I interview ~10 people weekly. I really, really, REALLY appreciate the fact that HR has done all of the background work and presented me with plenty of notes. By the time they get to me, it's time for me to score their tests.

The tests depend entirely on the position. Generally, I try to probe:

  • General programming skill. Can you use operators effectively? Can you conceive a numerical system that has a radix which is not 10?

  • Do you know how to do what we're hiring you to do?

  • Evaluation of your contributions to any open source projects that you listed

I try to keep it short, and fun. When I go into the office, I grab the answers, look them over and then conduct a secondary interview. To get hired, you typically have to get through three interviews.

I also gauge how well you will blend in with the team that will be receiving you. That team counts on me to do that effectively.

It is one thing to answer questions in meta form, it is another to actually produce code. If I'm going to hire you, I really need to see you produce code.

share|improve this answer
    
Hmmm... I consider myself as a qualified developer. During my last three interviews, when I was asked about something like "what does a particular method do", or a similar thing, I decided to give it up. Since I never search a job, where I would be intended to do something I know very well. It is not interesting. As my ex-boss said: "Reusable? Programmers should be reusable, programs - maybe". –  duros Nov 24 '10 at 18:06
    
@duros I'm sorry to hear that. A good interviewer should be able to make the process fun .. and should realize that other programmers hate tests. –  Tim Post Nov 24 '10 at 18:13
    
it is not that I dont feel comfortable when being tested. I just realize which opportunities to learn and try something new I would have, if they are going to hire me as a coder... During the last one I sent the interviewer to read the javadocs... :-/ Maybe, I'm wrong... –  duros Nov 24 '10 at 18:58
10  
I was once asked in an interview to write accessors for a linked list in Perl. In the 8 years that I have been working with Perl I have not encountered a single production program that uses linked lists. Of course, I muddled through and produced,on paper, insert() and remove() methods and a hash based object which I thought would do the job. The first thing the interviewer said when he looked at the paper was "You are missing a few semicolons" –  leed25d Nov 24 '10 at 20:34
1  
it is another to actually produce code -- this is so, so, true. I have repeatedly been surprised by people who describe a plausible algorithm but can't even start reducing it to code. That, or they have skated over some non-algorithmic step that can't be avoided when you write code. –  poolie Sep 9 '11 at 6:41
show 2 more comments

With apologies to Scott Whitlock:

Cons:

  • none

Pros:

  • Saves a LOT of time and heartache down the road if you prevent hiring someone who can't program
  • Requires you to have a technical person in the interview
share|improve this answer
add comment

If you were going to hire a juggler, you'd be insane not to have him juggle for you. Or a musician you'd have audition. Otherwise you get stuff like the awesome yo-yo "master" K-strass.

Walking through something on a whiteboard is the programmers equivalent of a quick demo juggle.

share|improve this answer
4  
+1 for the K-Stass reference –  Conrad Frix Nov 24 '10 at 18:54
    
+1 for the link. I still think that juggling is not remembering stuff like method signatures, or similar, but just being able to think and solve problems you have never met before... –  duros Nov 24 '10 at 19:03
3  
Except that juggling is an immediate skill with only a few variants, often performed in front of an audience. Programming is a ponderous, deep task, rarely performed as such. It's also much less productive on paper or whiteboard. That said, pseudo code tests (or ignoring trivial medium bugs) can be very useful. –  Bruce Alderson Nov 24 '10 at 23:44
add comment

I think it's super useful, and I always do it, but since the benefits have been covered so well I'm going to discuss only the (apparent) negatives.

I think code tests are fairly unlikely to give you false positives: the odds are low that someone who actually can't write code will manage to fake it in an interview, at least if you have a scale of questions of increasing difficulty. (Perhaps the most likely scenario is they're cheating by asking a friend, if it's not a face-to-face interview.)

The problems are more on the false-negative side: will code tests lead you to reject the person who's actually the best candidate?

Stage fright

You may have someone who's actually a really good developer, but who is very nervous about this interview, and they get essentially stage fright. Performing under pressure is important to some extent, but dealing with stage fright is not such a key part of being a programmer (compared to other professions), and it would be unfortunate to reject someone who suffers badly from it. This can compound: if the person can't answer a question they know they ought to answer, they may get more up tight. Or, as in this question, they feel they can't talk and code at the same time.

mitigation: Start off with some getting-to-know-you questions about their background, goals, etc, before you jump into technical questions. Perhaps start with some easier technical questions so they get some momentum. Don't be a dick during the interview (quibbling about semicolons etc).

It's a noisy measure

Interesting code questions might have more than one correct answer. If one person writes a correct answer, and another writes a correct and more efficient answer, how much weight should you put on that?

To some extent this is like the problem with some "puzzle" questions: the person either has the insight or not and you get a nearly binary result. Intelligence probably affects the probability of having that insight, but sampling only a few times gives you a crude measure.

This bothers me about code questions (though I still use them.) The best mitigation I can think of is to as much as possible have a ramp of possible solutions: the person can at least write a crude brute force answer, or an answer for a part of the problem. Realizing that's better than nothing is a good sign. Then if they discover more, they can make it more efficient or more elegant code. As much as possible, avoid asking questions that get binary responses.

It's not really representative

Programming isn't a job of solving ten-minute algorithmic problems one after the other: there's lots more work about understanding and designing larger systems and concentrating for extended periods of time, to say nothing of the interpersonal skills. Code questions don't really test this.

But, code questions aren't the only questions you're going to ask: you can look at their background, their references, their open source work (if any), to find evidence of sustained effort, creativity, interpersonal skills.

Knowing how to solve small algorithmic problems and how to reduce them to code is a necessary but not sufficient condition: if you can't solve small problems and you can't write nontrivial code then all the big-picture thinking in the world is not going to make you a productive developer.

Anybody could solve that

No, apparently not. As the famous FizzBuzz post points out, problems you might think are trivial trap not only new graduates but people with years of industry experience. I don't know why. Either the code test is a poor measure (which is possible, though I think it's unlikely), or they weren't contributing very much to the projects on their resumé, or they were getting a lot done by copying-and-pasting non-algorithmic code (which is possible.)

It's worth acknowledging that you really can get a lot done without writing any algorithmic code. People make a lot of money on apps whose value is in the graphics or business logic not in what you might call "programming", and that's fine. But, if you actually need programmers, it's not a good fit.

It may not be well calibrated

If you come up with a question, the answer may very well seem easy to you. However, if you're asked any otherwise comparable question out of the blue, or a question that's not skewed towards your own particular interests and background, it may be much harder.

mitigation: Run the tests over some developers you already know, and see how they do. Perhaps someone already on your team, who you know is very smart, will have trouble with one of them and you can consider adjusting it. Perhaps they will think of a better or different answer.

It's too much like trivia

I think code questions certainly can get in to trivia, if you insist people know obscure APIs by heart, or get the syntax perfect, or remember the exact definition of a nontrivial algorithm. Those are all reasonable to rely on docs, web searches, or compiler errors to pick up, and have little correlation with real expertise. Not even knowing where the API's likely to be is perhaps a clue the person hasn't used it recently, but that's not necessarily a problem as long as they're not lying on their resumé.

So the answer to this is pretty simple: don't ask trivial questions and don't get hung up on trivial mistakes. Remind the candidate what the API is called or let them look it up; fix up syntax errors; don't test for people memorizing data structure definitions.

How do you compare?

If you have two candidates and both answer the questions well, how do you pick between them? You could choose the one who finished fastest, but perhaps there you start to pick hares over tortoises. You could do another round and ask much harder questions but I'm not sure about that either. Perhaps you just give them both an A+ and try to choose between them on other criteria (or try to find the money to hire both.)

share|improve this answer
add comment

Cons: None. Any time you spend setting up a PC, designing a code test, and reviewing it will save untold headache in the future.

Pros: "Trust, but verify" - Ronald Regan. So many times I've seen and heard of people finally let go from a position, where in the interview you'd think you were getting a rock star. The proof is in the pudding; I want to see what they can do. It will represent what happens once you invest time and money hiring someone and stick a new project in front of them.

share|improve this answer
    
+1 for the Renaldus Magnus quote. –  Captain Claptrap Nov 25 '10 at 4:54
add comment

One con I can think of is that it is hard to "code out loud" in front of other people. I find it hard even to type with someone looking over my shoulder, and I'm not alone. I notice when someone calls me over to their workstation to help with something, they suddenly start making typos, selecting the wrong code-completion, even making downright mistakes — none of which they would have made had I not been sitting right there. Hell, I've seen people start using the menu for cut & paste operations, just because they were under observation. This is not normal behavior, and the coders I'm talking about are excellent programmers and quite smart besides.

I recently had an interview in which the interviewer asked me how I would code a particular operation and he said, "Just show me the math." Well, I had to think about the problem first before getting to the math of it, so that had me hemming and hawing. What I put down on the white board at first was embarrassing, and I felt I was losing at that point. I eventually got the A-ha moment and found the answer (actually when it finally occurred to me what he was really asking), but the "mess" I made before I got there made me feel very uncomfortable. Nevertheless, I got the job, but if the interviewer had been less patient I might not have.

I think if you give interviewees a coding task, give them some time alone to work it out on a computer, perhaps even in an IDE they're familiar with. Let them write the code for you and then talk about it. Ask them why they did things a certain way, and whether another way might not be better. You will find out more from that kind of process than by asking them to (figuratively speaking) pee into a cup right in front of you.

share|improve this answer
4  
That's ok, though. The goal of a coding interview question isn't to test typing ability or even perfect knowledge of the API. Typos and whatnot can be ignored and instead you focus on the candidate's thought process and familiarity with foundations of programming. For example, I was once part of an interview that showed the candidate's utter inability to think a few steps ahead (they'd fix one small problem, realize that created another issue, etc.). –  Anna Lear Nov 24 '10 at 18:05
2  
It's "hard to code in front of other people" ? Fine. I only hire people who can do hard things. One of them is being able to discuss code (i.e. program) with (in front of) other people. –  kevin cline Sep 9 '11 at 13:25
1  
@kevin cline: You miss my point. Do you care how people get results, or do you only want them to get results according to your whims? Plenty of people can code just fine in a team environment, but need "alone time" to produce at peak efficiency. You sound like the kind of guy who wouldn't have hired Mark Zuckerberg because he needed to be "wired in" for maximum productivity. –  Robusto Sep 9 '11 at 14:08
1  
@Robusto: I am not asking deep questions in an interview. I just need to see someone solve a simple problem in a few minutes. And I need people who can work in a team. This means an ability and willingness to talk about code. Sure, I may miss a great programmer who just can't handle the pressure of an interview. That's ok. But a bad hire is disastrous. –  kevin cline Sep 10 '11 at 3:05
add comment

Cons:

  • Requires you to have a technical person in the interview

Pros:

  • Saves a LOT of time and heartache down the road if you prevent hiring someone who can't program
share|improve this answer
25  
If you're interviewing developers without having technical people participating, you're doomed anyway. –  David Thornley Nov 24 '10 at 17:28
    
@David: I agree, I think the pros here far outweigh the cons, but it was the only "con" I could think of. –  Scott Whitlock Nov 25 '10 at 1:23
add comment

When I interviewed for my current job, I was given a list of questions to write code for by the recruiter. I was very impressed because the questions were obviously written by someone who had a deep knowledge of SQL, so it works both ways.

share|improve this answer
add comment

You really want to have the person write code in the interview - even better, get them to pair program with a member in your team for X amount of time (whatever you can comfortably afford in time/manpower).

It's pretty much one of the only ways you can tell if that person can code or not.

I slightly prefer the pair programming as it'll show their team work, gives them a real IDE to work with and lets them work on a 'real' problem (the other person in the pair can guide them past any environmental specifics that the interviewee could never reasonably know about).

We've started using this hiring policy and we're really happy with the results.

share|improve this answer
    
+1 for pair testing: proves both ability to work with a team mate /and/ the ability to code. –  Bruce Alderson Nov 24 '10 at 23:39
add comment

You judge a bird by its feathers and a programmer by his/her code.

When I started with the current company I'm working for they asked me to write some C code that generates or checks the parity bit of some binary input (depending on whether you are encoding or decoding). This was an interview question exactly because these kinds of problems are solved during work. Of course I'm thinking of not parity checking but rather working on a low-level.

share|improve this answer
add comment

All the answers so far (which I read) didn't address the fact that the Joel Test is NOT (just) a best practice list for entrepreneurs but a check list to ease your evaluation of a prospect employer.

The thing is ... if they thoroughly test their candiates then they probably hire people who know their stuff ... that means for you

  1. less headache and also
  2. the increased chance of learning something from your co-workors

instead of bug-fixing after them ...

share|improve this answer
add comment

I would say:

Pros

  • Demonstrates the candidate has at least a passable knowledge of programming, since resumes can be fabricated/embellished
  • If the interviewer discusses the code with the candidate, as opposed to it being more like a written test, could be a good indicator of how you "mesh" socially and if the candidate is a good fit for the company/company and team are good fits for the candidate

Cons

  • Could be insulting to candidates if the questions are rote/trivial nonsense that would never come up during the job (e.g. "Write a bubble sort" when all modern frameworks that one would be using on the job have sorting built in, "Reverse a string" when there's a built-in Reverse method or similar, or for written tests things like "What are the arguments to the Foo method of the Bar class", when any idiot could Google that or use the documentation) as opposed to architectural/design questions that show the candidate can get things done and solve business problems.
share|improve this answer
1  
I think "Reverse a string" is an excellent starting question in a coding interview. If they don't know where to begin (and an amazing number of people don't), you probably don't want to hire them. If they use the built-in method, that's good -- it tells you that they won't try to build their own wheel if one is provided -- but then present a hypothetical situation where the built-in method has a bug so they need to write their own. You can then use their code as a starting point for asking other questions like how to fix any bugs, memory usage, running time, etc. –  Gabe Sep 10 '11 at 5:49
add comment

One pro is that it shows that somebody does have basic knowledge of programming or whatever (the last time I encountered that, I was surprised how basic the SQL question was). It also can serve as a basis for a technical discussion, asking why the candidate did such and such, and how it could be improved.

It does take time in the interview, which could be used for other things. Moreover, writing code on a whiteboard is not a natural environment, and some people will have more and less serious problems with it. It could cause you to miss a developer who just gets nervous without normal tools or references.

share|improve this answer
3  
What would surprise you even more was how many more people could not answer that basic question than who could. –  HLGEM Nov 24 '10 at 17:49
add comment

Programming is a highly technical skill with a bunch of clear "deliverables." Either a candidate can or cannot deliver them. So there are no "cons" to asking technical questions. It's entirely in order to say, "show me some code for this application, or "show me the code for an application you have ALREADY written."

NOT doing so could lead to a result like the following: A rich man interviewed a tutor to teach his children to play chess (as a mind-expanding exercise). The tutor opened up a checkered board and started talking about the 64 squares but didn't touch a chess piece. Pressed for time, the father hired the tutor anyway. And the tutor taught the children to play CHECKERS.

share|improve this answer
    
That just shows an example of an incompetent interviewer, not the necessity to actually play chess in an interview. The rich man could have just asked 'explain Castling' to me, or something similar, and immediately have gotten an idea of what the tutor knew. –  GrandmasterB Sep 9 '11 at 20:26
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.