Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

What is the best way to evaluate the best candidates to get a new job (talking merely in terms of programming skills)? In my company we have had a lot of bad experiences with people who have good grades but do not have real programming skills. Their skills are merely like code monkeys, without the ability to analyze the problems and find solutions.

More things that I have to note:

  • The education system in my country sucks--really sucks. The people that are good in this kind of job are good because they have talent for it or really try to learn on their own.

  • The university / graduate /post-grad degree doesn't mean necessarily that you know exactly how to do the things.

  • Certifications also mean nothing here because the people in charge of the certification course also don't have skills (or are in low paying jobs).

We need really to get the good candidates that are flexible and don't have mechanical thinking (because this type of people by experience have a low performance).

We are in a government institution and the people that are candidates don't necessarily come from outside, but we have the possibility to accept or not any candidates until we find the correct one.

I hope I'm not sounding too aggressive in my question; and BTW I'm a programmer myself.

edit: I figured out that asked something really complex here. I will un-toggle "the correct answer" only to let the discussion going fluent, without any bias.

share|improve this question
    
I'd value creativity, as you say regarding code monkeying. I don't like the brute force approach, if generations of past programmers have used a given approach, it might mean it's great, or just that it's been perpetuated for a long time. Also education is not supposed to focus on trade skills, and i'd say it's super important but put the actual grades acquired beyond a basic competent level as not so important. I'd love to see more of a Khan Academy style system of many small pass/fail modules with other module pass dependencies and a cool-down period before being allowed to retake a module. –  alan2here Nov 24 '12 at 23:36
add comment

closed as primarily opinion-based by MichaelT, gnat, Bart van Ingen Schenau, GlenH7, Michael Kohne Oct 31 '13 at 16:59

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise.If this question can be reworded to fit the rules in the help center, please edit the question.

14 Answers

up vote 45 down vote accepted

Regarding candidate selection, I usually go with a three-strike plan :

  • Regular test with FizzBuzz-like coding questions and many knowledge questions where they have to give coded examples. Depending on the position, it can be OO principles, SQL design principles, etc. I increment the difficulties of questions across the test to see how far they can go. The idea is not really to have all the questions answered (if they do, the better), but also to see if they can acknowledge when they don't know something. Trust is essential, and I don't want to have someone lying to me in my team.

  • Return on the test with the candidate, and discussion around the answers. Possible extension of the questions to reach the candidate's limits. This can be extensive, and the more extensive it is, the better.

  • Last part but not the least, The Code Review. I ask the candidate to bring a piece of code (I generally space the previous test/discussion and this review by a few days, to let them write and polish one piece of code). Then we do a regular code review of it with two people : one person that will directly work with the candidate and the person that reviewed the test with the candidate previously. Regarding the code review you can read this article from JohnFX.

At the end of all this, you should be able to decide if you want this candidate to be part of your team or not.

share|improve this answer
4  
I agree on coding questions and knowledge question, I never agreed to ask candidates to bring a piece of code. In my opinion it's not easy to find a substantial piece of code that: shows something non trivial, does not require too much knowledge of a bigger system and that you are allowed to show to others. –  Andrea Zilio Jan 3 '12 at 21:12
    
I would say code review. FizzBuzz is so overused. And you might scare off people. I've been a programmer in health care and financial services, and stuff like fizzbuzz is useless. You gotta be able to understand much more complicated interaction. Ask them for code samples, even if it's a FPS in pygame. If they have no code samples, they're not coders. –  Christopher Mahan Jan 4 '12 at 20:56
1  
@ChristopherMahan The value of FizzBuzz (and simiarly trivial coding) is as a simple gatekeeper. If they cannot implement FizzBuzz, in their language of choice, can they write ANY code? –  Vatine Mar 21 '12 at 16:27
    
@Vatine FizzBuzz tests for logical reasoning ability, not programming skill. Granted, one ought to be able to do it. A simple programming question would be: display on screen a list of items sorted in an interesting way. They would have to code, but they would be able to draw on experience to come up with something to write, and not have to also try to figure out the brain teaser. I once failed an interview because they wanted me to use regex and I replied that for that problem regex is an overkill and python has built-in capabilities to do this. I didn't want to work there I guess. –  Christopher Mahan Mar 21 '12 at 22:13
    
Code Review is a terrible idea, and JohnFX's rebuttal is insufficient. I know plenty of devs who are great but do not work on anything outside of their employer's proprietary code bases. These are people with families and things to do outside of work. However, when they work, they are very productive. –  MrFox Oct 31 '13 at 15:47
add comment

Start with giving them FizzBuzz to solve. That should weed out the worst of them.

Then something a bit harder - for example, how to reverse a string without built in library functions. Ask them to talk while solving in order to see what their thought process is.

You can keep giving harder problems if they find these very easy, until you are convinced they can walk the walk and not just talk the talk.

share|improve this answer
1  
I guess it may depend on the level of programmer you're interviewing. While it might be OK to access the ability of a junior level hire, I'd consider such questions in an interview a clear indicator that this wasn't a company I wanted to work at. –  jfrankcarr Dec 15 '11 at 15:22
3  
I disagree with the talking part. Do you often talk to yourself or other people while solving problems? How about giving a person some space and letting him think a little bit? So unless it is completely trivial, I think it is not a good practice. –  Andrey Dec 15 '11 at 15:49
1  
@Andrey - The talking part is to gain insight into their thought process. You want to see how they think about a problem and approach solving it. Do you have a better option? –  Oded Dec 15 '11 at 15:52
4  
@Oded, yes, in fact I do. Let them think a while, alone, as they would do in real life, especially if it is a hard problem, and then ask them as much as you want. Like in any oral exam. –  Andrey Dec 15 '11 at 15:58
3  
Even as an attendant at a school that has several recently recognized CS professionals I find that trivial fizzbuzz is a fine filter. Most of the people I graduated with probably couldn't solve it in a reasonable amount of time. Those people struggled to find employment or didn't. I think following joelonsoftware.com/articles/GuerrillaInterviewing3.html is good. –  Rig Dec 15 '11 at 18:26
show 4 more comments

Just look for passion about the job.

To quote Joel, look for people who are "Smart, and get things done."

The rest doesn't matter

share|improve this answer
7  
Problem is that you cannot tell if they are smart. –  user1249 Dec 15 '11 at 15:13
4  
@Chad: a passion for learning doesn't "get things done". To find if someone can get things done, you have to ask them to do something. –  kevin cline Dec 15 '11 at 15:55
2  
@kevin cline, tweaked my post. They need to have passion about the job, usually that's shown by wanting to learn. Yes, it takes some questions, the conversation should flow, it shouldn't be just a series of questions. But to find out if they get things done, ask for examples where they had an obstacle, and how they overcame it. Everyone hits road blocks in their job, be it from technology, people, processes, but a smart person, who gets things done, will find a way to overcome it, and be able to explain it, in detail, enough to sell you on their ability. –  CaffGeek Dec 15 '11 at 16:22
3  
@mouviciel not in my experience. Some of the smartest programmers I know are very extrovert. –  user1249 Jan 4 '12 at 10:19
4  
If you can't sit in a room with a developer candidate and figure out if they are smart or not, find someone who can. –  JeffO Jan 5 '12 at 14:20
show 5 more comments

Evaluating programming intelligence is a form of Turing Test. Thus there are (currently) no closed form evaluation procedures that are guaranteed to work. It takes intelligent programmers to recognize other intelligent programmers, but only with some reasonable probability.

Your chances will be better if you have interviewers on your team who can smell snow jobs, and instinctively dislike working with stupid people (even the ones who are good looking, have impressive looking resumes, and can spout all the usual canned solutions from memory).

(One possibility methodology that would help the quality of stackoverflow as a side-effect is to dig up old stackoverflow questions, related in some way to your job requirements but that in your opinion have inferior answers; ask the interviewee how they would answer, and have them post it if it's a good answer. Similar to a recapcha for crowd-sourced OCR.)

share|improve this answer
add comment

Based on my 25 years of programming (which, admittedly includes only a 5 or 6 instances of hiring other programmers) :

Positive indicators:

  • Passionate about technology

  • Programs as a hobby

  • Will talk your ear off on a technical subject if encouraged

  • Significant (and often numerous) personal side-projects over the years

  • Learns new technologies on his/her own

  • Opinionated about which technologies are better for various usages

  • Very uncomfortable about the idea of working with a technology he doesn’t believe to be “right”

  • Clearly smart, can have great conversations on a variety of topics

  • Started programming long before university/work

  • Has some hidden “icebergs”, large personal projects under the CV radar

  • Knowledge of a large variety of unrelated technologies (may not be on CV)

Negative indicators:

  • Programming is a day job

  • Don’t really want to “talk shop”, even when encouraged to

  • Learns new technologies in company-sponsored courses

  • Happy to work with whatever technology you’ve picked, “all technologies are good”

  • Doesn’t seem too smart

  • Started programming at university

  • All programming experience is on the CV

  • Focused mainly on one or two technology stacks (e.g. everything to do with developing a java application), with no experience outside of it

In addition, I'd suggest:

  • The FizzBuzz test (or something like it to test basic ability to write an algorithm.
  • Harder version of the FizzBuzz test (to get them to the failure or near-failure point.)
  • Discuss their code and see if they are willing to be self critical and look for improvements (which they probably didn't have time do do in a short on the spot test) such as:
    • good variable names (I've had very experienced skilled coders use variables in production like "flag" (WTF??)
    • modularization.
    • Anticipating problems and doing "defensive coding"
  • A willingness to see "flaws" as opportunities for improvement. I think the best coders always look unflinchingly for flaws in their previous code. They are not so egocentric as to think that finding a flaw their is a personal affront. They see it as an opportunity to do better. (Those that can't look at flaws unflinchingly either are overwhelmed by seeing a flaw (and become super underconfident or, to avoid just that, they ignore the flaws.
  • Can they debug?
  • Can they Unit Test? (I've talked to way too many programmers who say "QC does that". I'm not talking about Testing, I'm talking about testing: you write a function, does it work? Does it make reasonable efforts to deal with likely problems (NULL input, etc.) ? If you can't do that, how do you know when you're done?
  • Do they have good communication skills? (at a minimum: good comprehension and self knowledge about when they do and do not understand and a willingness to say "I don't understand, please explain it again".

Much of the summary above is from How to spot a good programmer, which is a great article, focused a bit more on longer range indicators. It definitely confirms my intuitions and experience. It's also a lot of things (like "passion") that aren't normally mentioned in a checklist of "what's a good programmer".

share|improve this answer
add comment

Give them a problem, preferably one associated with the problem domain they'll be working on, and ask them to discuss how they would approach it. You can have them just discuss, pseudo-code or write bits of actual code depending on how confident you are in their skill level

For example, if your organization did conferences, ask them to outline how they would code a secure online registration system. They should be able to cover some of the basics and ask good questions about exactly what needs to be implemented. As you interact, you should be able to determine if they'll be a good fit for your organization and the role you need them to fill.

I'm not a big fan of programming trivia tests and brain teasers. While they can be fun for some people, they can also annoy and/or stress out other people, including people who might just be the best fit for your team. Plus, info on many such tests are readily available online and will encourage cramming for the tests and other tactics that would blunt their viability to gauge programmer ability.

share|improve this answer
    
I agree with you about the downsides of the trivia/brain test for candidate selection. the problem is that reviewing/discussing code with every candidate will be too much time-consuming. And maybe the result would be more subjetive. not exactly what i'm looking for, i would prefer something that needs less personal supervision. and later when i have more suitable candidates then talk/discuss/interview them –  Rafael Dec 15 '11 at 15:36
3  
Too time-consuming? Someone is going to have to talk to the candidates. No written test will work. The contents of the test will quickly become public knowledge, and candidates will arrive with memorized answers. –  kevin cline Dec 15 '11 at 15:57
9  
@kevincline: Exactly, you have to talk to them. I was interviewing at Xerox (back in the 70's) and I was asked how I would handle collisions in a hashing algorithm. I didn't have much formal schooling in programming, but I had been doing it for about 5 years at the time, so I said I didn't know what a hash was. My interviewer explained it to me, then re-asked the question. We went on for over an hour as I discovered & solved several types of collision problems. He told me that if I could do that in a hour then I could handle anything they threw at me. I got the job. Because he talked to me. –  Peter Rowell Dec 15 '11 at 18:23
    
@PeterRowell That is how things need to be. +1 –  Chiron Jan 19 '12 at 13:44
add comment

Reading this question and some of the answers it has received prompted me to write an article which I feel might be of interest:

Odd recruitment practices when hiring software developers

Ok, so the article title is rubbish, but the article gets to the heart of the problem. It's not the candidate's problem that you've chosen to interview them no matter how inappropriate they may be for the role you have in mind. If you haven't managed to define a well-factored hiring procedure to allow you to find the gems in the rough, then you're going to just have to live with the consequences, and yes, this means getting a few candidates that could never meet with your expectations. Filtering your candidates based on their letters and resumes requires you to first, ask your applicants to write a letter about themselves and what they want from the role, and then look at how the resume is written. If you've only got one or two potential candidates to interview, then you've probably done the pre-screening properly. If you can't decide between your candidates at this stage and you still have a hundred applications, then you've probably either set your expectations too low, or you haven't been aggressive enough in your filtering process.

When you eventually do find the 1 or 2 candidates that you consider actually worth your time, don't simply ask a handful of inane tester questions, but instead invest the time to get to know these people, and to engage in open discussions about software engineering in general. You'll learn more from a casual approach about the candidate than you ever will in the traditional (and somewhat adversarial) interview situation. Also, don't simply settle for a single interview, but instead treat your key candidates to several meetings where open discussion is used, and where the candidate can meet with their prospective colleagues. The time is never wasted, as inappropriate candidates will not thrive very well in a highly technical discussion, and will show their flaws very quickly as they drop their guard. If you spend the time and still don't have a hire, you've had an opportunity to learn more about what your needs are, and can continue to improve your interviewing process based on what you learned from the 'failed' interviews.

share|improve this answer
    
Good points. However, I'd be careful about too many interviews. Both the candidate's and your time is valuable (particularly if the candidate is currently employed elsewhere). In my experience more and more interviews have diminishing returns, so I'd limit to one or two interviews. An (additional) phone interview might also be a compromise. –  sleske Jan 5 '12 at 8:43
1  
@sleske, I agree in principal, particularly if the same people attend all of the interviews. It's better therefore to share the burden to find the best fit for both company and team, and gives you a chance to learn from the observations of others. Bad interviews won't go further, but the more stakeholders with an interest in the candidate the more interviews you may need, so it's not unusual to have 3 or even 4 interviews in very broad teams. Too many more though would give an impression of being terribly disorganized. It also pays to tell the candidate about the number of interviews up front. –  S.Robins Jan 5 '12 at 10:14
    
@s-robins interesting opinions, only want to put some ligth about some aspects of my question. Due to a reason out of our control we cannot select our candidates based on a normal Recruiting process, instead the candidates only comes and we need to say if him/her has the correct skills/knowlegde to take the job. Maybe in a normal recruiting processes these things don't happen too often. but in our position we need to deal with this situation. –  Rafael Jan 5 '12 at 14:33
    
@Rafael, If I understand you comments correctly, you're saying that you get fed candidates from "somewhere else" to evaluate and that your difficulty is in making an objective assessment of a candidate without prior knowledge about that candidate. This sounds more like a systemic problem within the organization where you work. I would suggest meeting with the people who send candidates your way, and work with them to devise a system for filtering out the obviously inappropriate candidates before you interview them. Perhaps even request that a more formal application process be implemented. –  S.Robins Jan 6 '12 at 5:14
    
@s-robins you understood well ... –  Rafael Jan 6 '12 at 13:54
add comment

You haven't said for which language, but it is fairly easy to test someone's knowledge. It also depends on the level you are looking, but there is a fairly large pool of questions regarding the interview questions.

However you decide to hold your interview, don't asks those "lateral-thinking puzzle" interview questions.

share|improve this answer
2  
propously i haven't specified the language that we are using to develop, because we believe that a good programmer (with his/her respestive capacitation course) can learn to program in any language independent of it's sintax. –  Rafael Dec 15 '11 at 17:01
2  
@Rafael norvig.com/21-days.html . As I said, it depends if you are looking for a junior programmer, or a senior one. –  BЈовић Dec 15 '11 at 17:05
    
Because i'm talking about most of the candidates are newly grads. I'm referring to junior programmers, but my question goes in more broad context that my specific personal recruit process –  Rafael Dec 15 '11 at 17:18
    
@Rafael In that case you are expecting too much from a junior. Read the article I posted in the comment above, where it tells how long it takes to master a programming language. –  BЈовић Dec 15 '11 at 17:21
    
i'm not talking about master an specific programming language, i'm talking about to get the best person with the best set of generic programming skills, (that why i don't specify the language), i can't expect that everyone that comes as candidate master the language that we are programming, and that's why we are in the position to bring a capacitation course if people doesn't know the language. –  Rafael Dec 15 '11 at 17:25
show 6 more comments

I suggest you go with a FizzBuzz question and hire the first one that passes. Further tests tend to be flawed as not every good programmer will approach a problem like you, or handle an interview without stuttering, or know the languages you want or care about or silliness like exchanging integers without a third variable (who needs that anyway? I mean, since RAM exceeded 128 bytes?).

Think about it. If the FizzBuzz question eliminates 199 out of 200, then it just eliminated hundreds of interviews. Were you really going to interview hundreds of prospects?

Just seems like diminishing returns after FizzBuzz. That is assuming that 199/200 is even approximately close. And I presume YOUR time is valuable too...

share|improve this answer
2  
Scary how FizzBuzz is the standard test for evaluating a programmer's competence. However, this is a tried and true test - I can't tell you how many programmers with CS degrees can't do it (in their 'language of choice') –  Steven Ellliott Jr Jan 5 '12 at 14:03
add comment

The candidates must be given a real world problem to solve with the freedom to use any technology.

If she comes out in flying colors, she's In !

share|improve this answer
add comment

I'm not sure if this is a comment or answer but basically what Matthieu said. You want stupid easy questions that take a minute or two (but no longer than 5) minutes to do and they should be about different areas.

Such examples of stupid easy question is a question about recursion such as you have a list and you must print it reverse order without using a loop. A simple regex question if regex is normally done in your development. A question about bits and bytes if using C++ (write a template that accepts char to long and prints out the binary representation. Specialization isnt required, just use sizeof() to figure out the bit length)

It should take you about <=3minutes per question

share|improve this answer
add comment

Ask them about the most interesting programming challenge that they have ever tried to solve but could not, what approach did they use while solving it, why they could not solve and what other approach they think can solve it.

This is enough for me to judge a programmers abilities as a programmer.

share|improve this answer
add comment
  1. Can they defend what they claim they know? They put it on the resume/CV as a skill or something they did on another project. See how in-depth they can go on the subject.
  2. Can they learn something new? Talk about a high-level aspect of the technology you're using or something specific to the business domian you work in and see if they can grasp the subject. Do they ask intelligent questions? Can they come up with an analogy? Is it similar to something they've done in another industry or technology?

  3. Would they rather be programming? It doesn't have to be number one on their list, but they have to show a preference to writing code. And I mean actually writing code and making something, not sitting around and talking about it or drawing on the board all day long. Not to minimize planning or to promote cowboy coding, but you gotta have code eventually. Avoid those who avoid the keyboard. This isn't a managment position.

You can do some scoring on a scale of one to ten thing or just rely on being able to smell your own kind.

share|improve this answer
add comment

If it makes you feel better bad programmers exist in pretty much every country. How to weed them out is the problem.

First weeding is the resume. One thing I look for is a lot of claimed language experience and nothing to describe what they did in that language. I've seen resumes that pretty much claim they know every language ever invented and yet their experience shows they've only actually worked with Access and Visual Basic. Those go right in the trash. 10 page resumes go right in the trash (especially ten page resumes from people with less than 2 years of experience which I have gotten). From recent college grads with little experiece, you have to be really picky about how they present themselves. The best candiates are careful with their resumes, they don't have errors. Are you really looking for someone who cares so little he didn't bother to proofread his resume?

Professionally prepared resumes go in the trash too. Once you've read hundreds of resumes, you can pick them out as they use the exact same phrasing. You can't trust the content in a professionally-prepared resume and you know the person didn't do his own prep. This is the kind of person who will rely on others to solve his problems for him, do you really want that in a programming position?

Look for things that make the person stand out for the ones you pick. That's harder of course with the ones just out of school, but look for accomplishments, contributions to open source, etc.

The next weed out is the phone interview. Ask about basic concepts that relate to the actual job you have. If people don't have basic knowledge of concepts you need them to have, they aren't worth bothering to bring in to a personal interview. The young often think this is unfair becasue they can look up everything on the Internet, but the truth is I have never met a good programmer who had to look up everything on the Internet. You should have some knowledge of your profession that you don't have to look up each time.

After the phone interview you should pick the best 4-5 candidates and interview. Of course if you only have 1-2 good candidates, don't bother interviewing people you already eliminated. Now you are going to ask the hard questions and get a feel for how they approach problems. I would never use the fizzbuzz test because it is too well known so the answers tell you nothing. Instead make up some problems from your own code base. I might give them a requirement and a piece of code and ask them if the code meets the requirement and if not why not and what they might to do make it meet the requirement. I would ask them to describe the most difficult programming problem they have had to solve and what steps they took to find the answer. I would ask some more in-depth technical questions. Remember you are trying to get a feel for their technical competence, their problems solving and debugging ability and their ability to fit with your existing team. I also ask questions that they probaly don't know the answer to to judge how well they handle stress, it's a stressful job, I don't want someone who folds in the interview because the stress of the job is greater than the interview stress. I look for strengths in areas we are currently weak in and ability to work in teams and to present themselves to the customers (our devs deal extensively with users), your list may be different.

share|improve this answer
add comment

protected by MichaelT Oct 30 '13 at 21:16

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.