How do I assess whether my expectations to the interviewees are adequate?

Question

Our company is hiring C++ developers and we have to estimate applicants skills based on resumes and (usually) code samples.

I find it problematic to decide whether I'm realistic in my assessments. For example, there was a guy who stated two software development jobs, 5 years of experience total. He claimed that he worked in a project for porting some operating system to new hardware and one of his duties was "fixing bugs in the operating system".

Then I open the sample and see the following:

 char buffer[someSize];
 strncpy( buffer, someString, someSize );

obviously this code will not null-terminate the destination once the string is of length equal to or greater than someSize and the program might run into undefined behavior.

Now my thought goes like this: "that guy claims he fixed bugs in the operating system code and now he can't properly use a simplest C runtime function... WTF???" and this motivates me to write a feedback to our HR that he can only qualify for a position right above the entry level.

What bothers me I can't get rid of a feeling that I'm being too harsh and demanding.

How can I decide if my expectations are realistic or not in such situations?

Answer 1

You are not being too harsh.

The sample code you posted shows a lack of attention-to-detail. Not only does this introduce an error into your product, it's a potential security issue too. If you have applicants that cannot demonstrate an understanding of 'normal' C code, they clearly do not have the skill level you are looking for.

It's one thing to have not-quite-optimal choice of algorithm, odd coding style or some other non-serious trait, but not demonstrating a command of the language is pretty bad.

If the person described themselves as relatively new to the C language, then perhaps give them some other tests - for example, ask them to write a strdup() function and see if they (a) check incoming pointer, (b) check for malloc failure and (c) remember null terminator.

There are many, many 'faker' candidates out there who'll apply for a job, get pre-processed by HR teams and some non-C-coding interview panel and may make it through a number of interview stages before having to actually demonstrate that they can actually write code. I've rejected people who emailed me excerpts of impressive-looking code as a sample of their work yet, when interviewed, were forced to admit that they didn't write that code after crashing-and-burning in a strdup()/toupper()/isalpha() whiteboard test.

To address this problem, we created a coding test, with 5 different questions, each in a different language (C, C++, JavaScript, C#, ASP.NET) and asked candidates to answer any 3. This eliminated the majority of fakers as couldn't web search for the answer to our very-specific questions, and we also didn't give the answers to agencies/HR so no chance of any help.

I commend you for setting your quality bar appropriately - when you do hire someone, you'll feel assured that they're not going to have painful 'need to learn to code in C first' issues.

Answer 2

What bothers me I can't get rid of a feeling that I'm being too harsh and demanding. How can I decide if my expectations are realistic or not in such situations?

If too many applicants fail (say: more than 20 in a row, many of them with good references), you are propably too harsh. But don't give up too early, because for every open job, a lot of people apply who are not really qualified, but hope to land a job anyway, with some luck somewhere in a not-to-well managed organisation where their results are unimportant as long as they look busy.

What I did when I was looking for new developers is that I made a test, and let my coworkers do it, as a calibration. I knew how well they performed, so after that I could say something like: Excellent people score 90%-100%, good enough people 60%-80%.

Answer 3

I find that running your tests past your existing developers is a good measure of fairness - these are people you have the measure of and you can see how they do with the questions. If people you know to be good developers aren't getting good scores then your test is probably too hard.

Obviously you have to make it clear to them that they won't be judged on the results as it's about developing the test, not assessing them.

Generally though you're better off rejecting many good candidates than accepting a single bad one so if in doubt say no.

(Specifically in this instance I think you're being fair but I thought I'd try and tackle the wider point.)

Answer 4

Choosing who to hire, is one of the most important decisions you can make, as it will have a vast impact on how well your team performs.

So, I think it's better to be too harsh then not harsh enough.

However, if you are too harsh, you need to recognise that, probably after rejecting too many candidates. If this is occurring you often need to re-evaluate the advertised job spec, and your criteria for reviewing candidates.

Answer 5

Just to play devil's advocate, you state:

... he worked in a project for porting some operating system to new hardware and one of his duties was "fixing bugs in the operating system".

and then supply an example of bad code.

However, what proof do you have that he wrote that code? If his job was to port the code there's a good chance that it was written by someone else. This might be a perfect example to test his bug fixing ability.

Highlight the method and use it to ask a series of questions. Which way you go depends on his answers, but start by asking him if he wrote it. If he says "yes" then perhaps he's not as good as he claims he is ;). If however, he says "no" then you can ask if he had to fix any problems in it. Again "yes" indicates he missed something obvious while "no" leads on to the next question - "can you see anything wrong with it now". Ask him to talk you though what he's doing and what he's looking for.

It's difficult to ask these without alerting him to the fact that there is something wrong, but then a failure to spot it then becomes a bit more serious.

This also gives you a chance to assess his approach to solving problems.

Answer 6

When you are preparing a code sample to give to a potential employer, you should take care to make sure it is correct and will not cause them to question your skill level. That the candidate did not do that is in indicator that he doesn't care or is not competent. Either indicates he is a bad choice for a hire.

Many people make their experience sound better than it is. There are many people with years of experience out there who are still working at the entry level. Just because he was assigned to a duty, doesn't mean he did it well.

My personal opinion is you were too nice on the guy. If he hasn't learned better in five years, he is probably not ever going to be out of the entry level category and thus is not someone I would want HR to consider as qualified at any level.

Answer 7

Allowing a bad hire is the worst thing you can do to your company ... the worst! It's very difficult to undo a bad hire. Because of that, I calibrate myself to pass on some good candidates if it means keeping bad hires out.

Often times, I feel like I am the subject of an Asch conformity experiment where everyone else says "hire" and I say "NO!" But that's the nature of the beast, it's much easier to say "yes" and pass the buck than to say "no" and take responsibility. Let me give you a test, which one sounds better when you say it ...

"I gave him the simplest memory leak I could think of, he didn't find it, but I hired him anyway."

... or ...

"I gave him the simplest memory leak I could think of, he didn't find it, so I passed."

Which would you rather say to your manager?

My advice is to set your bar high and keep it high ... remember, you might be dealing with this guy's leaky crap-ass code if you allow him to be hired into your company!

Answer 8

My recommendations:

1. You need to realize that everyone you see in an interview is different kind of person. They have different skillset and they think different things are important.
2. For C++ programmers, there are still plenty of ways to approach it and still able to make it work.
3. If you're testing people, you should make it possible for people with different backgrounds to pass it. To not just concentrate on the area that you already know best -- that kind of person anyway isn't best for the company, unless you're thinking employees are robots who all work the same way
4. But in the end it's programmers job to make code work. If experienced programmer cannot do it, it's a fail. Just need to give them opportunity to do it their own way, and not impose some restrictive framework on them which prevents it from happening. Everyone has some quality issues, even though people tend to try to fix them eventually, just need to find out if those are critical to what you need to get done.
5. Previous experience is very critical in this. Some people have been in projects where errors cannot just be tolerated at all, and these people can spend countless hours fixing things that are not really problems. On the other side of the scale are people who cannot see a bug when reading the code -- they have never needed to fix what they have written. Good plan is to find something from the middle, people who can do it quickly, but with small number of problems in it. But that would be wishful thinking - you're not going to find one.