Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

After our phone screen, but before a face to face interview, I ask our candidates to complete a short programming assignment. It involves writing a small command line program in C, which should run to around 100-200 LoC and take 3-4 hours to complete. We allow the candidate 72 hours to complete the task and get it back to us.

It involves a little simple file and string handling and a simple data structure (which the candidate has to select appropriately). We ask for a solution that is "reasonably efficient, concise and easy to understand" to hint that we don't expect it to be super clever or micro-optimized at the expense of readability.

The task is not completely specified, so we make it clear that we'd like to see any assumptions document.

So my question is: how should I rate the solution?

I have seen a couple of solutions so far (it's early days). But I'm a bit disappointed that there have been bugs that I've found within a few minutes of hand testing or looking at the code. But what sort of standard should I expect?

If anyone has any links to good general advice on the topic, it would be very much appreciated.

share|improve this question
    
If it's a clear specification I don't see why you shouldn't be able to compile and run the code in a VM. In that case you could have an automated test suite that has a number of inputs (including some edge cases) and judge based on number of correct outputs. You might want to tell candidates that they should expect you to run the code (and the compiler you are using) and just fail them if it doesn't compile. –  Victor May 2 '13 at 11:00
5  
Is it possible to have the candidates talk their way through their code? I'm thinking that you could get the candidates sat at a laptop with you, you both read through the code and the candidate discusses why they chose certain algorithms/structures. That might give you a better insight into the candidates than just blind testing their code. –  Jamie Taylor May 2 '13 at 11:17
    
@Victor sure, I definitely expect to be able to compile it, and that's pretty much the first thing I do. I ask for standards conforming, portable C without external dependencies, so if I can't compile it, there's a problem. I intentionally don't spec the problem well enough to make a test suite feasible. I expect (and ask) the candidate to take the rough spec and make some sensible, documented assumptions. I check out their assumptions and code and do some simple manual testing. So far it has been very easy to find bugs and undocumented assumptions –  Ned May 2 '13 at 12:46

6 Answers 6

up vote 2 down vote accepted

There's obviously no one-size-fits-all answer to this, but here are some thoughts :

  • Don't hand the candidates "not completely specified tasks" on purpose. There are already so many reasons for them to get the exercise wrong, you don't want an otherwise brilliant applicant to fail miserably because she misunderstood the specs or interpreted them in a way that's going to make her work really hard to assess. Plus, in real life you most often get (or at least should get) a chance to talk to your business analyst/domain expert/product owner instead of remaining stuck with your questions before a piece of paper. One exception could be that you precisely want to test their ability to get in touch with you to clarify the task though.

  • Obvious bugs are not acceptable. I mean, if the program really takes 3-4 hours to write and you give them 72 hours, there's no reason it should break within 3 seconds of execution - except if there's a platform or hardware related problem. Any crash of the application in a main scenario case should be a big minus for the candidate. Best written software in the world isn't worth anything if it isn't working software.

  • Rate each desired quality in the code. You said you were asking for efficiency, conciseness and readability. Well there you have your assessment criteria.

  • Give reward for unit tests. Because they're actually proof that the candidate 1. understood the requirements (or at least it tells you which way they understood them) 2. paid attention to quality and 3. shipped executable documentation along with their code.

  • Have candidates explain their work during the face to face interview. A program is only a snapshot of what a candidate is able to do at a particular point in time, it doesn't tell you anything about the applicant's ability to introspect and improve herself. A face to face conversation can. Would you rather hire someone that got the program exactly right but is impervious to change, constructive criticism and improvement or someone who may not have completed the task perfectly but from the interview seems to have a tremendous potential for progress and a forward-looking mentality ?

share|improve this answer
    
72 hours is a pretty reasonable limit considering that good candidates may need that long to squeeze the time it takes to solve the problem into their schedules. –  Blrfl May 2 '13 at 15:31
    
We ask the candidates when they'd like to attempt the problem and send them the problem description at some convenient time just before that. The clock only starts ticking when we send the problem. –  Ned May 16 '13 at 8:02
    
I'm slightly conflicted about the not-completely-specified thing. The answers here make some good points and of course a complete spec would make "marking" easier. But I think that spotting and dealing with incomplete specifications is easily as important as being able to code something that has been well specified. What I'm looking for is not that they make the "right" assumptions, but that they've spotted the holes in the spec, thought about what the requirements should be, preferably documented and justified the decision and definitely stuck to the decision throughout their code. –  Ned May 16 '13 at 8:46
    
I tend to think a developer can't "deal with incomplete specifications" on his own - better to ask a domain expert, at least for non-trivial business problems. If you intend to make them act as domain experts for the sake of the exercise, maybe you should state that clearly in the instructions. –  guillaume31 May 23 '13 at 9:50

We require both a pre-screen design exercise as well as several white-board exercises during the interviews. We'll also request a code sample if I can't find anything of theirs online.

I have found a strong correlation between the design exercise and how well they'll do within the white-board exercises. It also appears to correlate with their on-the-job performance as well.

Generally, I'll forgive a mistake or two within any of the exercises. We don't require compiled code for the design, but bonus points are definitely awarded if it's a complete, working program. Likewise, the white-boarding exercises give me insight into the candidate's mindset since I'm watching them while they sketch out their answer.

In your particular case, I would say listen to what the code is telling you about the candidate. By the time you see their solution, you've had a number of touch points with them: their resume; notes from HR on the phone screen; your notes from the phone interview; etc... If you think they're sloppy and the code they submitted is equally sloppy, consider that a red flag. Ask if your team can handle that level of quality coming on-board. Determine if that's something you can train out of the candidate.

Likewise, if the candidate has been otherwise stellar but you notice a "rookie" mistake or a corner-case not being covered, then consider proceeding with the interview process. If everything the candidate did was already perfect then they would be demanding a much higher salary. There needs to be opportunity within your team for the candidate to grow and learn to become a better programmer.

These types of exercises are just a data point in the overall process. As they require more effort, they can be more telling as a data point. But they aren't always a make or break type decision point. It's kind of like requirements gathering. The initial rounds are usually a bit murky and you have to keep imploring to get to the actual needs. Ultimately, the interview process is about identifying a match between the candidate's skills and your projects needs. Likewise there needs to be a match between the growth and learning opportunities your team presents and the candidate's interests.

In short, use their code project as part of your overall assessment of their skill level. Interviewing is like fuzzy logic - it's generally not a binary up | down vote at this point, but the code exercise will sway the analog signal into the well defined regions of 0 or +1.

share|improve this answer

Now that you have already received solutions, the answer should be: don't.

It's simply unfair/unsuitable to setup this scenario, then evaluate it using criteria, that the authors did not know prior to the evaluation. Like this you get a complete reality shift.

Whatever your criteria are, you cannot come to the conclusion, that a candidate is unable to provide them. If you tell the candidate in a transparent/fair way how your evaluation works, then they can optimize for these criteria. If you don't tell them, you will never know if they would have been able to do that.

Of course, one might reason the other way round, as in business, we often get assignments, which are later evaluated on a different basis. While this is fact, one must however also see that whenever this happens, you have a valid point of objection toward management. For an employee it is totally fine to tell the manager something like "of course it's slow in these extreme cases. We haven't received any requirements that state it should be otherwise."

You should allow at least the same level of objectionism to your applicants. Hence, I second the opinion of others here in that you should talk to the applicants about whatever bothers you about their code.

In addition, the errors you found in that code will similarly occur in the work environment later on (unless you truly want to tell me you have developers that write error-free code all the time). Hence, it gives you even more information: you can find out how the applicant deals with errors when you confront him with those.

share|improve this answer
    
Thanks Frank. I take the point that it would be beneficial to be more explicit about the decision criteria in the problem statement ("correct," "reasonably efficient," "understandable"). However, I don't want to go too far with this. For example, I expect the code to handle error conditions, such as missing arguments and files, or internal buffer sizes being exceeded - but I'd like the candidate to do that without prompting. –  Ned May 2 '13 at 12:28
1  
If you're expecting the candidate to write a piece of production code, and they think you're asking them to write a one-off script, then you will get suboptimal results. If you're expecting the candidate to write unit tests, and they know how to program in a TDD environment but don't go out of their way to write tests, you're going to get suboptimal results. Give them as much information as they need to give you the information you need about their skills as a programmer. –  Jonathan Rich May 2 '13 at 16:56
    
Frank - While I agree about not using new criteria to judge submissions under the old set, I think the OP is looking forward towards new candidates, so he can modify what he tells them. –  Bobson May 2 '13 at 17:10
    
@Ned - Personally, if you want it to be able to handle bad data, you should either add "robust" to your criteria, or something like "It should be able to handle basic user errors, such as a missing file". That indicates what you want, without giving away all the things you're going to look for. –  Bobson May 2 '13 at 17:12
    
@Ned - personally I think you've pitched it about right. I think it's more valuable to see if they can fill in the gaps without prompting than to see if they can type the correct code from a complete spec. We do a similar thing (a spec with room for interpretation) and as such, and as Frank says, we can't judge it on any absolute criteria - there's no right or wrong, no "scores", just a taster of how the candidate approaches things. Are they a careful, detailed oriented coder? Or are they sloppy? –  HappyCat May 2 '13 at 17:19

In a previous company that I worked for, we had a written C++ or C# programming test that would be given to candidates to do during the interview. Interestingly, one day, some of the programmers in the team started discussing some of the exercises and each had different ideas on how some of the questions could be interpreted, showing that what one person thought of a wrong answer could actually be argued to being right.

Perhaps, if possible, it would be an idea to get other programmers in your company who haven't seen the programming assignment to take it and see what they come up with. That would then put the candidate answers into perspective and be able to give you an expectation of what answers you should be getting.

share|improve this answer
    
Some of my colleagues seem a little shy about attempting it - I think perhaps they fear embarrassment if they "get it wrong". –  Ned May 16 '13 at 8:06
    
Make it anonymous. –  Merlin069 May 16 '13 at 8:36
    
That is a good plan, Merlin. Pretty obvious in hindsight! –  Ned May 16 '13 at 8:56

I wouldn't evaluate the solution without the candidate being involved. I think it's unfair to make assumptions based on the code you see without having a way to check those assumptions.

Instead of just looking at it for bugs, design flaws, etc. why not bring them in for a couple of hours to pair program on the solution they provided?

This will give you an indication of how well they know the code the wrote, if they understood your requirements, how they work with somebody else, and if you can work with them. There are tons of other things you can learn about a candidate as well during the pairing session.

Ultimately, the goal of the interview is to find people you can work with right?

share|improve this answer
    
"Why not bring them in" - I think that's the gold standard, but we originally introduced this to get more information to avoid unnecessarily bringing in candidates from distant places, which for some candidates is going to be a waste of (our) money and a waste of their time and ours. Or looked at another way, it allows us to let more candidates through the CV review and phone screen because we know that we're not going to waste up to a day of everyone's time as a result. –  Ned May 16 '13 at 8:53

If you're asking for some data processing (that's what your description looks like), you should be able to have some test cases that allows you to place it in either "seems to be correct" and "definitely not correct" with some simple automated testing. This is good, because it allows you to automate part of the grading.

For the "reasonably efficient, concise and easy to understand" bit, you're starting to verge into subjective territories. The efficiency bit may be objectively gauged by having problem sets of varying sizes and see how the execution time depends on the problem set.

Conciseness may lend itself to a simple code size check ("is the source code less than X KB when fed through a pretty-printer"?).

Easy to understand is definitely subjective.

share|improve this answer
    
I think that to be fair to the candidate, I'm nearly always going to end up reading the code in some detail, so automated tests are of limited value. And since I'm doing this after the phone screen, it's not a bottle-neck in the process. –  Ned May 16 '13 at 8:31

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.