Interviewing by providing an application to work on?

Question

While thinking about better ways to design technical interviews (meaning not based on social skills but purely technical ones) I started to think about this possible way of doing (without having the possibility to test it):

1. Determine the kind of problems, high or low level of abstraction, from architecture to bitwise manipulations if that's your shop's business, that you are solving every day in the applications you work for. For this example I'll suppose we are in a PC game development company that produce a game based on extensions via modules.
2. Write a simple application that solve those problems. Lets say a simple game that have not the same scale than the one we're working on but that would be
3. Add bugs. Architecture bugs and more language-knowledge related bugs if it's critical (for example it's important to know about virtual destructors in C++). Make it obviously buggy, maybe make it crash first and buggy after that. Make sure it works with the tools that most people use in the domain (like cross-platform C++ for a simple game, providing the libraries code with the rest).

4. Provide the full source code of the application to the candidate and tell him to a) fix the code, b) add a functionality (related to the position if necessary). The candidate would have a limited time to send something back. For example, one week or two. That time would be several times the time needed to do the work in full time, it have to be fair.

   4'. Maybe a different way to do that would be to provide the application source code publicly online and start to meet only the candidates who sent you the working application with the additional feature...but that would work very differently so I'm not sure if it's really interesting too.

For 4.b., I'm assuming that the functionality to add is simple but requires basic understanding of the overall organisation of the code. That would not be adding content.

5) Make the candidate explain what he did.

All that process would be a kind of interview screening, there would be a meeting after that if the candidate did provide something. It feels like parallel screening : you screen several people at the same time without spending time sequentially on each. A meeting with discussions about the code is obviously really necessary.

The questions :

a) Do you think that it would be pertinent? What do you think? (I'm not very experienced in interviewing so I'm relying on your experiences)

b) Would it be considered as paid work? Even if it's not work done on your product but just a interview-specific application?

c) I'm assuming that the candidate will use whatever he have available to solve the problems, even friends helping because I think that reflect better an real development environmental. Is there something wrong with assuming that? (Yes I'm questioning my own way of thinking, they say that's sane. Are they wrong?)

d) Is there a variant that would be more interesting?

Answer 1

I would hope you planning on compensating these applicants way more than your competitors, that is a pretty high barrier to entry, and only the most desperate candidates will take the time to apply for a job with you, is that really who you want to limit your candidate pool to?

Ask yourself, would how many hours would you spending working on a coding exercise for FREE in order to get a chance to get a second interview where you work now?

My guess is not many, unless you work some where very prestigious, and then you probably would not be asking this question, they would have already figured out how to interview.

For everything you are planning on them showing, you can devise very simple questions to show the same level of knowledge and comprehension with much less effort on both your part and the candidates part.

My favorite types of screening questions are more NLP oriented and much more effective than detail language lawyer oriented.

1. What in your opinion, module/library/class do you find most useful in [insert language]?
2. What in your opinion, is the best designed module/library/class in [insert language]?
3. What in your opinion, the most poorly designed module/library/class in [insert language]?
4. What in your opinion the biggest opportunities for creating subtle errors in [insert language]?
5. What in your opinion is the top 3 must know frameworks for [insert language]?
6. Given [language specific error] how would you go about diagnosing the problem?

You get the idea, anyone with indepth knowledge of [insert language] will have opinions, that you can then ask more direct follow up questions which will show how the candidate thinks, what they know has already been established.

Asking for personal opinions frees the candidate from trying to overtly worry about the correct answers. Having an opinion is a correct answer, not having an opinion is an incorrect answer.

For example, if someone says they are a Java developer and they don't name the java.util.Date and java.util.Calendar classes as one of the most obvious hideous parts of Java, they haven't really written any real Java.

Regardless of an answer, barring an I don't know, it gives you an opportunity to ask follow up questions like Why do you think Class X is the worst design?, How would you have done it differently? and How have you worked around that in the past? type questions will give you real insight into the candidates thinking and mind set and personality.

If you ask someone about tracking down memory leaks in a C program on Linux and they don't even mention Valgrind then you have an opportunity to question them about why they didn't mention it if nothing else.

You can apply this to every other language in use, they all have warts. Strong opinions are formed through experience, if you don't have strong opinions, you haven't enough experience for them to form.

Personal experience: In 20 years I have only supplied code for an example application to 1 job, and it was during an extreme slump in the job market, I was desperate and applying to everything that I was capable of, I wasted a entire 8 hour day writing up a solution to the problem given, in the end I didn't get the job because my solution was more advanced than what they were looking for. Never again, and my advice to other candidates if the same, it will be a waste of your time.

Answer 2

I think in general this idea sounds really good, the main problem you might run into is that although it might take some knowledge of the language in general and the code will have some bugs, reading code and figuring it out is a whole lot easier than writing something from scratch. I have fixed bugs and added small bits of functionality to applications that were in a language I had never touched before.

I think that on top of doing something like this you would have to require interviewees to still do white board coding. I think the one on one code discussion is a much better way to feel out potential candidates. This is especially true if you purposefully use a question that is hard. You can learn a lot about how someone works when you give them something they struggle with.

Answer 3

I have had a similar interview to that for a company (and was offered the job, but didn't end up taking it). In fact in addition to the regular interview, they gave me a specification for a software product and basically told me to get on with it. A few days later I sent back the code, with a description of how much time I spent on it and the things I would've additionally done had I had more time to spend on it. I didn't consider it a burdonsome part of the interview because I decided (unilaterally; there was no guidance on how long to spend from the interviewers) how much I wanted to invest into the challenge.

I think that if you're planning to have some long project like that, then it needs to be open-ended and your evaluation should allow for people who don't have much time to spend on your challenge, but are able nonetheless to show that they understand the problem and could make progress given more time. Remember that many people applying for jobs already have jobs, education courses or other commitments on their time and aren't going to want to give up a lot of free time for speculative work - which is essentially what this sort of problem is.

Answer 4

First off, I disagree that you have avoided analyzing social skills - step 5 is a test of social skills, since there are plenty of brilliant-working-alone people out there who can't explain how they are brilliant or how they did what they did. Also, I'd quibble that "no social skills, but brilliant technically" is NOT the demographic I'd want to hire for my team. Obviously as engineers we don't need to win Miss Congeniality - but I think we all have to speak and act well enough to solve problems together and work through tense situations when tempers are running high and I don't see anything in your process that gives me the assurance that when I disagree with this person or when the two of use are slammed with a project this is someone I can work with and trust.

a) Do you think that it would be pertinent? What do you think? (I'm not very experienced in interviewing so I'm relying on your experiences)

I think that:

• asking for a sample of work
• walking through a fix and/or implementation
• having a conversation about both

are all great ways to judge competence that are used today in many companies.

I am doubtful that by constructing an artificial environment that you will create enough complexity to be worth the added effort of doing it. You have a couple of key issues here:

1. Don't give people you haven't signed your NDA your biggest asset - the code for your product. The lawyers will have a nutty and you don't really want to give your competition that leg up.
2. The real product is the real product. A demo app will be as much of a fake construct as a CS engineering class or as another artificial test. Any program simple enough to be debugged and added to in a reasonable time frame is likely NOT to be a good indicator of real work. Are you going to make a test system that is so complex that it takes the candidate a solid 2 weeks to get up and moving and do the first task? That's a FAST ramp-up time in real-work situations.

It seems to me that your demo app suffers from the same thing that most college project do - it's just not possible in a short span of time to have a piece of work that is complex enough to give a perfect accuracy of actual work.

b) Would it be considered as paid work? Even if it's not work done on your product but just a interview-specific application?

I'm not sure why you would want to pay a candidate before interviewing. It's certainly not the norm, so you'd be bucking expectation.

Keep in mind that hiring a new person is as much an investment as anything else the company does. For any given interview process a company may invest in: - recruiters to save the time of screening the masses - time spent by HR screening either what the recruiter brings or acting as their own recruiters. (trust me, you want HR or a single POC handling recruiters... they will distract your engineering leads to no end) - time spent by interviewers interviewing candidates

Then when someone is paid - whether it's hired or on contract - or even when you give them $ for an interview - you also need to plan the overhead of finance, legal, and IT to make sure the person is setup and going and getting the money they should get. It's quite likely that if you start giving people money for doing this interview project, that your lawyers will want to get involved and make sure your company is covered from an employment law standpoint.

c) I'm assuming that the candidate will use whatever he have available to solve the problems, even friends helping because I think that reflect better an real development environmental. Is there something wrong with assuming that? (Yes I'm questioning my own way of thinking, they say that's sane. Are they wrong?)

Nope.

Note, though, that most interview techniques don't preclude that approach. Asking someone about previous team work will yield the same information. I actually prefer technical interview formats that force collaboration, since it shows me something of what we'll be getting if the person is hired.

In terms of practicality - paid work is whatever you have to pay people because they won't do it for free. For example, Stack Overflow seems to be unpaid work, since we all seem to love it so so much we do it for free. But I have yet to find someone who's overjoyed for the opportunity to play "Clean Bethlakshmi's House", so I pay someone. I think that rule applies here - you'd end up paying someone if and only if you can't get them to do it for free. If you're Google, you can probably get them to do it for free. If you're developing internal products for a non-software company... good luck, if you get 'em to do it for free, please let me know here at the "Clean Bethlakshmi's House" open source project. :)

d) Is there a variant that would be more interesting?

My absolute favorite was being asked, on the phone, by the interviewer to step through a real-world problem that his team had had in recent months. The process was entirely on the phone - I asked questions, he answered them. It was like role-playing being an engineer.

But this let the guy see how I thought - he could see exactly what my process was and he was giving me feedback on how long things that I tried would take. So, for example, when my hands on attempts took a simulated 2 days of testing and failing, I said "well... 2 days is long enough, I'm going to call in tech support - do we have a support contract on this product?", then I got extra points for being aware that schedules are not infinite.

The whole process took about 2 hours - longest phone interview of my life, but really worthwhile.

Judgment calls are, IMO, a big difference between a great engineer and a marginally good one, so when I interview, I always want to find ways of figuring out if the engineer is capable of using good judgment.

Answer 5

I think that it is a very interesting approach that would show you how they would do in a simplified way. I feel it is better then aptitude tests or anything like that.

I still feel that a face to face interview is needed because there is no way to tell how this person will communicate with your team/you. I feel that communication is essential in working as a team and this method provides no insight into how they will perform in that situation.

Answer 6

How eager are people to work for your company? And how thoroughly are you weeding out prospective employees before this step?

With normal technical 1-on-1 technical interviews, both sides are making a pretty symmetric time commitment to each other-- they're both indicating that they're willing to invest an hour to get to know the other. If you give candidates code to work through, on the other hand, the time commitment quickly becomes asymmetric. You can very quickly give the test to a large number of applicants without investing much time yourself.

A qualified applicant is likely to need a few hours just to set up an appropriate environment to do a meaningful test for most organizations where you probably need at least a database or an application server to replicate what you do day to day. If the test is going to be meaningful, it would then require at least a few hours of work to understand what was written, debug the problems, add the functionality, prepare notes for the interview, etc.

Since it is relatively easy for you to give the test to one more applicant, the temptation will be to give the test relatively early in the process as a way of weeding out applicants. On the other hand, applicants recognize that you're potentially asking them to do a fair amount of work with no guarantee of even getting to talk with a human. That works well if you are a particularly attractive place to work either because you offer superior compensation or because you do solve problems that are particularly interesting to programmers. If you are a more run-of-the-mill shop, however, good candidates are likely to decide that it isn't worth investing that many hours of effort just for the chance at an interview and that their time is better spent applying elsewhere.

You can mitigate this to some extent by only using this test at the very end of the process. If you've already had a phone screen and perhaps one round of face-to-face interviews and you've narrowed the field down to a few candidates, you could probably use this sort of test as a final step. There is a risk that you'll alienate good candidates at this point if it feels like you're "springing" a number of hours of work on them after they feel like they've jumped through your other hoops so you'd need to make it clear early in the process that if they make it to the last stage of the process, you'd ask them to spend a few hours on a practical exam. But at least at that point, both sides have some idea whether they want to work with the other and the candidate can decide whether it's worth investing time on your practical test.