Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I realise this is a fairly odd question to ask but in recent rounds of hiring we have noticed a rather odd trend in new developers that spectacularly fall short of our expectations. So much so that they to all intents and purposes appear to be an entirely different person to the one interviewed and tested.

We pre-screen applicants during their initial interview with a fairly nasty PHP/MySQL written test which tests both their knowledge of the PHP/MySQL as well as some fundamental aspects of application design. You cannot score highly on this test by simply memorizing the PHP and MySQL manuals (We've made that mistake in the past). I'm sure It is by no means fool-proof, but is intended to be sufficient to weed out those who have clearly lied on their resumés.

Applicants that pass this stage are offered a second interview which is followed by a 2 hour practical test where they are required to develop a small application to a given specification. They are not necessarily expected to complete the task but are judged on their approach to the given task.

Three times now he have hired the "best" applicant based on interview and test results, and each time the person that has showed up for work on the first Monday morning has been completely devoid of the skills and experience they have previously demonstrated. Each of these applicants scored over 90% on the written test and exceeded our expectations when given a practical task to complete.

Hence my coining of the phrase "Developer Twin Syndrome"

Has anyone else experienced a similar trend in hiring developers?

share|improve this question

migrated from stackoverflow.com Dec 13 '11 at 14:12

This question came from our site for professional and enthusiast programmers.

11  
I know this is off topic but: Are you finding that candidates are willing to subject themselves to this kind of testing? I would never agree to a 2 hour test. What is the candidates reaction when told you need to take another 2 hour test after passing the initial test? –  Morons Dec 13 '11 at 14:23
2  
Are these tests proctored? Have you tried an internet search on key parts of the test to see if Google is being too helpful for the candidates? Some candidates will go too far to get a job for which they really aren't qualified. –  Jamie F Dec 13 '11 at 15:12
6  
The practical test sounds an awful lot like work to me, and I dislike doing work during an interview that I'd normally bill for. Moreover, this is against some employment agreements. You may be chasing off the best candidates. –  David Thornley Dec 13 '11 at 15:28
4  
Hmm 2 hours test after and initial test would be a no go for me, if you won't trust me I would not work for you, but maybe it different here in Holland (or even europe?) then in other countries –  Ivo Dec 13 '11 at 15:43
5  
What exactly happened on Monday morning to make you think they're like a "different person"? It is most likely that they do know the stuff you're testing for (I don't buy the "leaked test" theory). Perhaps they're just coming to terms with your codebase and it makes them look like don't know what they're doing? Did these three devs eventually work out? –  Angelo Dec 13 '11 at 17:33
show 5 more comments

8 Answers

up vote 13 down vote accepted

Maybe your tests have been leaked. A candidate might get a friend to interview before them and prep them on the problems they encountered.

If so, the best solution is to develop a suite of basic programming challenges that can be solved quickly on a whiteboard by a candidate with genuine skills. Randomly selecting a subset of the tests will make it very difficult to game the interview process. If you have ten tests, and give each candidate two of them, you can dish out 45 different combinations before repeating yourself. Even a determined cheat with two friends would have less than a one-in-three chance of knowing both questions.

Also, do you put candidates in front of real programmers? It isn't clear from your question, but I get the impression that the only technical engagement they have is via written tests. A technical interview with a dev team member usually weeds out idiots.

share|improve this answer
1  
I guess it is possible that the test could have been leaked. Candidates aren't screened by a real programmer as early as I'd like in the process, but they do undertake the practical programming test alongside us in our office, and we try to get a feel for where they're at. –  WibblePoop Nov 10 '10 at 12:32
1  
This could be more likely than it sounds. Are the candidates coming from the same contracting agency (if they come from one)? I know that after an interview the recruiter almost always grills me about the type of questions that were asked. I have even occasionally been given a few tips before an interview by recruiters. –  Dunk Dec 13 '11 at 22:05
add comment

Maybe you're testing the wrong things? Instead of testing for "mysql", it should be about core database design concepts. You don't want to miss out on someone good just to avoid the week or two it takes to learn a DBMS. If it was a DBA, then I would be more specific about the platform. The same for PHP. Anyone experienced with web applications in any language won't have an issue picking up PHP quickly.

share|improve this answer
1  
I agree completely, and we've made this mistake in the past. Our current written test focuses much more on application/database design concepts rather than pointless questions about what functions do what. What we've found is a massive drop in test scores since we implemented this new approach with applicants clearly exposing their knowledge gaps. Which is what we wanted. Maybe the testing process needs more work though. It's looking more likely that it is our (my) fault. –  WibblePoop Nov 10 '10 at 12:37
    
Agreed. I have given technical tests to candidates before, but have allowed them to write their responses in any language they choose. I've even allowed psuedo code, since I am really looking for design/problem solving skills. –  Neil N Dec 13 '11 at 16:48
add comment

I've seen smart, smart people get dropped into a new situation and suddenly forget how to tie their shoes. Their first Monday morning is no time to be assessing their credentials.

They're madly trying to figure out how to check their new email account and get to the code repository and understand Jimmy-in-the-next-cubicle's code. Doesn't seem like a real fair time to judge them, especially after putting them through such a wringer in their interview process.

And then there's this: when you hire someone, you're committing to them just as much as they're committing to you. You're saying: Look! Here's this person. I found them and they found me, and I'm going to support them with whatever they need to be the developer I need them to be. If that's NOT the attitude you have with new hires, let me know your company name so I can avoid even bothering with your interview gauntlet.

share|improve this answer
add comment

The only explanation I can think of is that you are assuming that your tests are complex and throughout, but in reality they're not.

You could also ask the candidate to work with the team for a few days, doing regular work, before signing a contract. So you put the "first Monday" problem as part of the hiring process, not the firing process.

share|improve this answer
    
I know, and I really have agonised over this myself. The comments here have reinforced my doubts and I think I'll go over our whole testing process again before we attempt to hire anyone else. I'm not sure how well the working before contract signing would go down with HR, but i've certainly been thinking along the same lines lately. Even considering giving up and offering internships instead. –  WibblePoop Nov 10 '10 at 12:39
add comment

I think you need to take a long hard look at your hiring methods and revise them heavily, you have some major problems if your "best" has been a dud 3 times. You should test your interviewers and see if they are improperly ranking interviewees. You also need to check your tests to make sure they haven't leaked, are up to date and truly do test what you think they test, it would be worth it to involve your current developers in this process, get a senior devs comments/suggestions and have a couple junior devs that have been there a while take the test. Also you need to decide if your tests are even worth it, if you are getting unqualified candidates passing the test you are wasting a ton of time. Also I'm willing to bet that your company has a bad reputation with your local pool of developers and more qualified candidates aren't applying. good programming jobs are plentiful enough that good developers can be picky which means employers can't be dicks about hiring, or offer lower pay/less benefits and expect to attract desirable candidates. Personally I would feel insulted to be asked to take a written exam and then come back another day and take a second test, I would really recommend combining these into a single day and keeping the total test time at ~2 hours so with an initial interview that's still 3-4 hours per candidate for a first interview which is a pretty long time for what seems to be an entry level position.

share|improve this answer
add comment

A common problem. I've seen a strong correlation between those that do really well on very specialiazed questions/tasks and a lack of productivity. While those that tend to answer in generalities (often to the point of seeming like they don't know much about it) tend to do quite well in the real world. Perhaps the specialist group only know those exact items and the generalist answerers can't answer specifically as they know too much in general. No matter how well we try to simulate the work conditions for applicant testing, it just doesn't happen. I don't believe there is a way to build a suitably scientific method at this time. Subjective interviews/measures still seem to hold the upper hand on getting the best employees, despite all the attempts to objectify the process.

share|improve this answer
add comment

It's likely that your tests have leaked or they don't test the knowledge of the programmer. You can teach people how to use a calculator, doesn't mean that they can do math.

You might find this a usefull read:

http://www.joelonsoftware.com/articles/GuerrillaInterviewing3.html

share|improve this answer
    
The problem I've always had with that is that the typical developer is an absolutely horrible interviewer. You can script them, but then they lose all usefulness as they just follow the script... :-) That's why most places have HR/Management do the interviewing despite not knowing the tech bits. At least they can do an interview without completely botching it. –  Brian Knoblauch Dec 13 '11 at 17:13
add comment

Does your work environment have anything like the level of structure you have for your hiring process? It could be that you're not testing for the skills that employees will need in order to find a useful role in the company.

If your company's culture gives employees a good deal of autonomy, you might want to try to test more for candidates with the soft skills to deal with that rather than testing on pure technical merit.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.