Is this a good test to check programmers' ability? Any other suggestions?

Question

For hiring new engineers at our company, we strongly believe in testing whether someone can actually write meaningful, usable, reasonable-quality code. Have been trying to devise a test for this for some time. The qualities i am looking for in this test are:

1. It should not take less than 3-4 hours of work which is the allotted time for the test.
2. It should be language and technology independent.
3. It should not be a trick question, if you can code well in your language of choice, you should be able to do it easily with a bit of googling.
4. The exact answer code should not be available from google itself.
5. It should consistently be of the same difficulty level so that we are always comparing apples to apples. At the same time, one candidate walking out of the door after attempting the test should not be able to give any hint or answers to the next candidate.

Given all these considerations, the one test we are considering is to have a very large number of URLs pointing to news stories on the web and the idea is to give each candidate a random URL from this large set and ask her to scrape the news story, given a language of her choice, and save it in a structured way (json, xml) such that we have all required fields properly extracted. For example story title, date, author, text etc. Candidates are free to use internet and use whatever help they would otherwise have available when attempting a real programming problem in a job-setting.

What does the community think of this way of testing? What biases do you think exist in this test? Also Id love to hear other people testing programmers with similar testing methods. Would also like to hear suggestions for similar/better test questions.

EDIT: I realize that the question we devised is not ideal, in fact far from it. So while, your critique is welcome, you should also suggest some questions which are language-neutral, domain-neutral, require non-trivial coding and dont have answers already available on the internet.

EDIT: Clearly people think its actually offensive for programmers to call their friends if they are stuck with a problem so i am taking that out from the process. (But, ironically asking someone on stackoverflow is ok)

Answer 1

I would not allow general googling, ask on StackOverflow, phone a friend, etc in a technical test. You should be testing the candidate's ability to solve problems by himself.

(Alternatively, get his phone calls traced and make a job offer to his friend instead ...)

If a problem requires the candidate to use a 3rd-party library, provide links to the library and its documentation ... or download the required stuff to the machine / network used for testing.

---

I also don't like the problem that you have chosen:

• Unless the candidate is already familiar with the APIs for programmatic web access, scraping HTML and so on, he / she is going to struggle with getting something done in the timeframe.

• Unless the candidate is intimately familiar with the technology web scraping (permissive HTML parsers), he / she is likely to come up with a crap solution ... like regex-based "parsing".

• What you are asking the candidate to do is (I hope) not an accurate reflection of the way your group operates (I hope!). But you risk giving him / her the impression that it is.

Answer 2

This is not the worst test I've ever seen, if I was given this to do I would not feel it was unreasonable. A couple (OK, 3) of weaknesses I can think of:

• Although it is language neutral, it is not skills neutral, so some one with a lot of HTML and Regex expereince would find it far easier than someone who has not.
• An exact solution might not be google-able, but there are some HTML screen scrappers out there.
• Asking for production quality is not great - do you really want them to implement a logging framework, localizable strings, full unit tests, documentation comments etc etc. I think you want good quality working code, but not full on production quality, it will simply take too long.

An improved approach might be to set a simlar test but in your office with access to reference material such as books or help files but no internet. And you could consider simplyfying the task and reducing the time; e.g. ask them to parse a CSV file and present some results such as the sum of the numbers in column "n". To do this you will have to noarrow down the allowed tools, you probably can't install every SDK known to man on a spare laptop.

However

My personal view is that you well get better interview results and both you and the interviewee will get more out of the interview if you do a coding test on a whiteboard. This allows you to gauge their approach, theor problem solving skills and thei attitude a lot better. It allows you to help someone who gets an attack of the nerves, and it allows you up the ante if you get a genius. I have to say of all my interview experiences the toughest (as an interviewee) were white board ones, but I think they were the best test of my skills. I have done tests similar to the one you describe on a couple of occassions too, so I know both sides. And as an interviewer I feel the white boarding has given my the best insight possible with in the interview timeframe.

Answer 3

I like to give candidates actual problems that I encountered recently and was unsure of the best answer. This way I know if the answers are clever or not, and can intelligently discuss them with the applicant when they come in.

For example, When I was writing a speed controller, I could have asked for an algorithm that would compute optimal speed at each time step, to arrive stopped at a given distance as fast as possible, with constraints on min and max accel and velocity. [But I did not interview candidates at that position.]

At my current job, we created a standard test for our applicants where we give them some sample data and ask them to write a program to find "interesting patterns" in that data. It really shows up how (and if) the candidates think. Some focus entirely on statistics, some pick something simple and write something bulletproof, some write a complicated mess of spaghetti, and many end up not ever responding :)

Answer 4

What we do for our Java interviews is to give the user a very simple class to refactor. The class contains 1 bug. We don't mention it explicitly, but they find it when they try to use the class.

When you watch someone try to refactor real-time, it's incredibly easy to see who has real experience and who has been padding their resume.

The class is so simple, if the person is not experienced in refactoring, the candidate will struggle to find things to refactor.

It is important to remember with any real-time test that the level of pressure is extremely high. Mistakes will be made that wouldn't be made normally. So, we don't worry if the person has some difficulty as long as we can tell he's thinking about the problem correctly and heading the right direction. The fact he can't remember the keystroke for "extract method" doesn't matter if he's thinking he needs to extract a method.

Answer 5

If real problem solving and analytical skills is what you are looking for, and I am assuming you don't do junk work in your office 90% of the time, what's stopping you from just making an assortment of the technical issues your team faced over the last quarter and present them as programming challenges to the candidate?

This helps you decide 3 things:

1. How good the candidate is when comparing his/her technical solution to what you already have
2. Aptitude of the candidate in your domain from the kind of responses and questions you hear from him/her
3. Time bound programming output

Answer 6

I don't think that is a good test for these reasons:

1. It's too trivial: get html, process html, write output. You don't need much skills to come up with that that.
2. The actual implementation is just as trivial if you pick the right tools (curl url | awk '...')

This really is a test to check the programmers ability to select the correct tools. Having that ability is very important of course, but it doesn't say anything about programming skills.

I would instead look for tests that encourages the programmer to show some insight and (potentially) brilliant ways of solving a problem, preferably while working together with others.