Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We are currently recruiting for a Junior Developer position working mainly in C on Linux.

As part of the process, we require candidates to complete a code test at their leisure in C.

So far we have rejected two candidates on the basis that their code, although readable and in one case rather idiomatic, suffered from buffer overflow errors due to unbounded buffer writes.

Are buffer overflows acceptable from a graduate developer? Are we setting the bar too high? What is the expected capability of graduate/Junior engineers?

[Edit]:

  • We explicitly ask for error-checked, production quality code.
  • We provide a test & build framework for the candidates

[Update]:

As a result of this thread, and conversations we have had with other developers in person, we are changing the way we carry out code tests and who we target with our recruiting.

We decided that a candidate being unable to fix or understand a buffer overflow means that he would be unsuitable for the work we carry out, in particular he would take more mentoring than we are comfortable with. We will therefore still reject candidates that cannot eventually submit a robust code sample.

However, we have put in place some measures to make the recruitment process more productive for both us and the candidates.

In particular:

  • We make our expectation more explicit, with clear explanation of what we mean by production quality, and a warning that the code is expected to be robust with respect to input and errors.
  • We now link candidates to resources on defensive programming and the C standard library in the description of the code test.
  • We changed our target audience from Junior developers and graduates to target people with some relevant experience.
  • In case the submitted code fails in some way but would otherwise be accepted, we now provide a minimum test case that causes the error condition and give the candidates a chance to correct their mistakes (unless the code is rejected for some other reason). We'll also point out problematic lines/functions if appropriate.
  • The goal of the tests itself has now slightly changed from a front end filter to a chance to build a better picture of the candidate, in particular it will inform our phone discussion. That said, we are still willing to reject based solely on code.
share|improve this question
28  
Maybe...? Considering that it seems a lot of people in school now have much more experience in Java than in C. If the candidates are fresh out of school and 2/3 of their coding exposure is Java, their C might not be strong enough to pass your test. BUT... if they're open and willing to learn, then what would you say? After all, this is a Junior position... –  FrustratedWithFormsDesigner Aug 2 '13 at 14:13
4  
It's also code written in an interview, perhaps forgivable given time constraints. I'd at least be sure to mention to them that their code needs to not suffer from buffer overflow. –  Mansfield Aug 2 '13 at 14:20
3  
Interview code is different, this is a code test to complete in their own time. –  brice Aug 2 '13 at 14:23
3  
I second @FrustratedWithFormsDesigner. I'm a junior developer, and the only time I had to deal with C was for a couple of assignments in a class on CPU architecture. Meanwhile, I consider myself pretty OK at C#, Java, and Ruby. –  Kevin Aug 2 '13 at 15:16
44  
I can't say you're doing anything wrong , but after interviewing dozens of major college CS graduates over the last few years I've found that most have no concept of what is meant by 'production quality code'. I don't think most CS instructors care about writing code for use by others. –  Jim In Texas Aug 2 '13 at 16:31
show 10 more comments

7 Answers 7

up vote 108 down vote accepted

I don't think you've set the bar too high, I think you might need a different bar.

I think code tests are useful for determining the competency of a candidate, but they shouldn't be pass/fail. You should use the results of the code test to start a dialog with the candidate.

If you see mistakes that they've made (especially if they're junior developers) point them out and ask them what they'd do differently or if they understand why there is a problem.

share|improve this answer
1  
+1 that's true, and it's a good suggestion. We did in fact ask the candidates to review their code for errors, giving them plenty of time to do so, but they came back with the wrong kind of corrections, in some cases making the code worse, without fixing the errors that caused the overflows in the first place. –  brice Aug 2 '13 at 14:25
15  
@brice Pretty much a solid demonstration that they are in fact junior developers. Avoiding seemingly obvious mistakes comes with practice and experience. –  Tombatron Aug 2 '13 at 14:51
34  
@brice: he was saying to discuss the specific errors with them; don't say there was errors and ask them to get back to you. Discuss the errors in real time -- give them a hint (line number or maybe a description and function and ask them to give you the line number), then ask how it could have been prevented. The goal not being error free test code, but a good understanding of the applicants strengths and weaknesses. –  jmoreno Aug 2 '13 at 15:11
6  
If the junior developer was able to see the issue, they would have fixed the issue. If it were me, I'd test their ability to work with you. Tell them what the issue is and where it can be found, give them the same treatment you'd give your actual colleagues and see how they react. If working with them is a chore, they're not right for the position. If working with them is a pleasure, then hire them. –  zzzzBov Aug 3 '13 at 2:22
add comment

I think the junior qualifier is what makes all the difference here. Juniors shouldn't be tested for competence, they should be tested for learning ability, curiosity, passion, ethic, and definitely humility. The assumption with a junior should be that they are not competent, it's your job as the senior to make them so.

Obviously they should be able to write basic code like fizzbuzz and have a general knowledge of concepts; if you pointed it out to them and they didn't even know what a buffer overflow was, then I would say it's a no go, but I don't expect a junior to write more than 5 lines of code without a bug.

The day you trust your junior's competence is the day yours should be questioned, juniors should be treated with a lot of mentoring and a healthy dose of "trust but verify". I was a junior once, and to all those who were there at the time: I'm sorry. Remember the terrible things you did as a junior? (Please don't tell me it was just me; you'll give me a complex..)

share|improve this answer
1  
I'm officially still a 'Junior Software Engineer' with a whole 2 years of experience! My background is in neither CS or SW Engineering (It's in Physics). I would not willingly hand in code that segfaults on any input, now, or when I was recruited. –  brice Aug 2 '13 at 15:09
29  
correction, you would not knowingly hand in code that segfaults on any input. If you just claimed you never write bugs, than sorry to bother you John Carmack. –  Jimmy Hoffa Aug 2 '13 at 15:11
    
Harg! backpedal! That certainly isn't the claim I make. I've released buggy code, some worse than others. But none that looked like the first example of what not to do when you google for "buffer overflow" –  brice Aug 2 '13 at 15:13
    
We even provide a test framework for the code, including tests that triggered the overflow! –  brice Aug 2 '13 at 15:21
    
@brice When you say you provided test framework, did you provide a tool like valgrind to test for memory problems? –  BЈовић Aug 2 '13 at 16:03
show 2 more comments

I see a few issues here.

The first is assuming that an average computer science graduate knows, well, anything. They don't. Quite frankly, I'm pleasantly surprised when I see a computer science graduate that knows how to install and setup Visual Studio. Heck, I recently worked with a guy claiming to have more than five years experience on the Microsoft stack writing .NET code that couldn't figure out what TFS was or how to connect.

The second is your very limited pool. We also have candidates do a programming test. There are five separate "programs" they have to write. They do it at home and ship the code over. The tests are extremely simple (no database, no external dependencies) and can easily be done using the Express version of Visual Studio. The tests themselves are easily completed by a senior guy in about 30 minutes. Note that we generally only do these for recent graduates on up to three years of verifiable work experience.

What we've quantified is that around 70% of those given the test simply never get back to us. Roughly 15% turn in stuff that won't compile, usually due to blatant syntax errors (for example, missing ;). Another 10% compiles but fails to perform the required actions.

This leaves a whopping 5%. At this point, we aren't even considering conditions such as entering an alpha character as input when a numeric one is required. It's purely on given a very limited set of X as input the application does the appropriate output. Also, these numbers come from about 500 candidates over the past four years: we kept statistics because we wanted to know.

If we were to look more at code structure and defensive coding techniques such as properly disposing unmanaged resources or usage of try .. catch statements then almost no one would pass.

The question, of course, is why?

Why can't a kid with a degree in this field from a four year university accomplish what are simple programming tasks? The answer is that colleges are completely out of touch with business needs and lag many years behind what we consider state of the art. 10 years ago coding standards were such that security was something you did after the fact; and unit tests weren't even in vogue yet. Whereas today security better be at the forefront of your mind with every feature or enhancement. Just remember: most professors have either never actually worked in this field OR haven't worked in it for a long time. Once you know that, then you start to understand why they are so far behind. Worse, some of those professors spend too much time on a particular technology (Java, PHP, whatever) and fail to discuss serious foundational issues such as code structure or acceptable approaches (and WHY!).

Just a side example. A recent graduate was telling me about some of his experience writing a mobile OS for one of his classes, but he couldn't explain, even in basic terms, how a web server worked. He simply didn't know. 15 or 20 years ago was probably the right time for understanding how to make an OS. Today... not so much. Yet this was a required class when a class on defensive programming would be so much more useful to them AND the outside world.

So, what do we do?

Out of those 5%, we will interview a bit further to get an idea of their personality and fit. Then we pick the "best" ones in full knowledge that we are going to spend around six months "reprogramming" them to get rid of the crap their professors filled them with.

share|improve this answer
2  
Totally agree here, I have learned more in my 2 1/2 years in the industry then I EVER learned in college. I learned this painfully after I got my first internship as a web-developer while still in school. –  ryan Aug 2 '13 at 22:28
5  
Now I want to try your programming test.. –  Akash Aug 3 '13 at 4:52
1  
Really? Experience installing specific pieces of software and an ability to give an oversimplified version of what a webserver does is what you're looking for? If somebody can deal with writing a mobile OS, they can learn how webservers work. –  Michael Shaw Aug 3 '13 at 14:06
    
@MichaelShaw: If someone is writing an OS, but has yet to be taught the basic operations of the most common server type then it shows that his school was skipping large (and highly relevant) areas of his education. The question then becomes what else was skipped? –  Chris Lively Aug 5 '13 at 14:40
2  
@ChrisLively: I don't see how either of those things are a big problem. It's not like we have a tiny field that moves slowly. Learning on the job is going to happen. I think you may have an essentially good point here, but the examples are not convincing. If there is a problem with the CS curriculum, adding "How to install Visual Studio 101" and "Basics of Webservers 101" wouldn't fix it. (I do like the idea of a "Defensive Programming" class.) –  Michael Shaw Aug 5 '13 at 21:23
add comment

I think you are looking at the problem the wrong way. What you should ask yourself is this: What do we require from a candidate for them to be able to do the job? You should properly evaluate the position and what it entails. Below are some suggestions on when to hire a junior developer and when not to.

When to hire a junior developer: - If there is an overflow of easy work to be done, which would be a waste of time for a more senior developer. - If you are willing to mentor and train this person over the next few years. - If you are trying to grow the company and want someone who will stay for a long time. A junior developer who stays for only a year would be a waste of company resources, they would do little to produce anything, and most of the results would be in their own personal growth. - When you feel like spending money on someone's growth. Since again, most of the benefits will be what they learn.

When not to hire a junior developer. - When the work is too complicated. In this case it is just out of their league. - When you want to save money. A senior developer should complete the same tasks in a fraction of the time with better quality results, and thus should always be cheaper. - When the work is outsource-able or isn't enough to keep an employee busy. In such cases it would be better to offload some of the work to a private contractor.

One last important point. Don't hire a junior developer because "that is all we can afford", or "that is all we are willing to spend" when they are not suited to the job. In the end all you will end up doing is flushing money down the toilet. Besides, once they acquire those skills they will request the big bucks anyway.

About me:

  • Degree in physics with almost no formal training.
  • Two years of work experience. So I know what the learning process is all about.
  • Start software developer. I have done very demanding work and have seen all different skill levels from various individuals. Many of them can't handle a lot of what I do.
share|improve this answer
add comment

A buffer overflow is an absolute no-go. You could have a corresponding code question. In a what all is wrong (can go wrong) with this piece of code the candidate should be able to pinpoint the problem. The question is, whether the issue is irrelevant, as you are running clint anyway.

However in an artificial free-form code test I would be mild on a violation like sprintf. Too little time (assumed), mind hyperactive, too large an urge to present something working.

share|improve this answer
10  
I almost wrote the exact same answer until I noticed he's referring to a "junior", keep that in mind.. I've seen some sharp juniors, but even still they do stupid things without realizing, much of software engineering can only be taught by experience –  Jimmy Hoffa Aug 2 '13 at 14:45
    
@JimmyHoffa yes just read the first line of yours, corrected my "absolute no-go". Your point is worth considering. Till now I could use and estimate every programmer, but one psychic case and one "liar". –  Joop Eggen Aug 2 '13 at 14:50
    
@JoopEggen: I'm pretty sure "psychic" wasn't the word you were looking for. Otherwise they should have been able to read your mind... ;) –  Chris Lively Aug 5 '13 at 14:41
add comment

As others mentioned, junior positions may have little experience with C. I personally only briefly was taught about buffer overflows in C and even if I could watch out for them, it is likely I would still introduce some (especially if given an assignment that lends itself to creating buffer overflows). Likely, many junior developers will be a similar situation where they may know of buffer overflows, but they have not been prepared to identify and handle them in any extensive manner.

Given this, I would think the appropriate response is to have the issue brought up in the next possible interaction and ask them what they know about buffer overflows to test their overall knowledge. After that, tell them that you found one in their supposed production ready code. This would give you a good window to judge how they would react to correction and instruction.

Isn't the common thought that a junior developer who knows less but is willing and able to learn and improve is more valuable than a junior developer who knows more but who cannot or will not improve?

That being said, in one of your comments you mentioned you handed them tests that would have pointed out the buffer overflow in their code if they had used them. So perhaps the bigger question is why didn't they run the tests (of if they did, why did they turn in buggy code)?

share|improve this answer
add comment

I think you're asking the wrong question. There is no objective bar to be a professional programmer. If there were, there'd be a standard programming exam, but there isn't. Your individual bar just has to be set based on how picky you can afford to be, how much you can afford to pay, how much blunder your software can afford to accept, and how much time you can afford to spend teaching.

In this case, I'm guessing that a buffer overrun probably means that this code, which the candidate presented as a sample of exemplary work, crashes with a segmentation fault instead of doing what you asked. So, should you accept a coder who writes code that crashes with a segmentation fault instead of doing what you asked for? Ask:

  • Is your organization capable of attracting anyone who can write working code?

  • Is your development process robust enough such that someone who can almost write code could write working code with the help of peer review and testing support?

  • Are you capable to teaching sort-of-programmers how to be a programmers, and are you willing to expend that effort and wait maybe a few years and hope that the candidate's inner talent will come to fruition?

share|improve this answer
add comment

protected by World Engineer Aug 2 '13 at 19:47

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.