Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

For hiring new engineers at our company, we strongly believe in testing whether someone can actually write meaningful, usable, reasonable-quality code. Have been trying to devise a test for this for some time. The qualities i am looking for in this test are:

  1. It should not take less than 3-4 hours of work which is the allotted time for the test.
  2. It should be language and technology independent.
  3. It should not be a trick question, if you can code well in your language of choice, you should be able to do it easily with a bit of googling.
  4. The exact answer code should not be available from google itself.
  5. It should consistently be of the same difficulty level so that we are always comparing apples to apples. At the same time, one candidate walking out of the door after attempting the test should not be able to give any hint or answers to the next candidate.

Given all these considerations, the one test we are considering is to have a very large number of URLs pointing to news stories on the web and the idea is to give each candidate a random URL from this large set and ask her to scrape the news story, given a language of her choice, and save it in a structured way (json, xml) such that we have all required fields properly extracted. For example story title, date, author, text etc. Candidates are free to use internet and use whatever help they would otherwise have available when attempting a real programming problem in a job-setting.

What does the community think of this way of testing? What biases do you think exist in this test? Also Id love to hear other people testing programmers with similar testing methods. Would also like to hear suggestions for similar/better test questions.

EDIT: I realize that the question we devised is not ideal, in fact far from it. So while, your critique is welcome, you should also suggest some questions which are language-neutral, domain-neutral, require non-trivial coding and dont have answers already available on the internet.

EDIT: Clearly people think its actually offensive for programmers to call their friends if they are stuck with a problem so i am taking that out from the process. (But, ironically asking someone on stackoverflow is ok)


migration rejected from Feb 10 '14 at 16:08

This question came from our site for professional and enthusiast programmers. Votes, comments, and answers are locked due to the question being closed here, but it may be eligible for editing and reopening on the site where it originated.

closed as primarily opinion-based by MichaelT, gnat, Kilian Foth, Bart van Ingen Schenau, GlenH7 Feb 10 '14 at 16:08

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise.If this question can be reworded to fit the rules in the help center, please edit the question.

What does this test a) tell you about the interviewee's ability to solve problems that are relevant to your domain b) tell the interviewee about the type of problem you think is a useful measure of their technical competency? – Rein Henrichs Apr 16 '11 at 7:35
HtmlScraper a language to help assisting in the scraping of HTML. Too bad no compiler exists (yet!) ;) but it might be a good business opportunity to write one. – Sjoerd Apr 16 '11 at 9:49
Actually if you know a language where one line can scrape any page on the web and give you only the information you want, trust me you will be hired by any company at the world at any price you want :) – Yasser Apr 16 '11 at 10:06
call their friends???? Are you recruiting software engineers or intervieing candindates for "Who wants to be a millionaire"? Seriouly that is a bad idea. If someone calls his/her friend, recruit the friend instead of the candiate. – DPD Apr 16 '11 at 10:32
@Aaronaught: when you're that good, interviews are as much about you interviewing them as about them interviewing you. When I was changing jobs last time, the impression I got from the interviews about my future teammates was a major factor in making the final decision. I would NOT want to have to decide whether to sign the contract after speaking only to my future boss. In fact, if an IB offered me a job too quickly, I'd see it as a warning sign -- why are they so eager? what's the catch? why won't they let me to know them better before I have to say yes/no? – quant_dev Apr 16 '11 at 17:34

This is not the worst test I've ever seen, if I was given this to do I would not feel it was unreasonable. A couple (OK, 3) of weaknesses I can think of:

  • Although it is language neutral, it is not skills neutral, so some one with a lot of HTML and Regex expereince would find it far easier than someone who has not.
  • An exact solution might not be google-able, but there are some HTML screen scrappers out there.
  • Asking for production quality is not great - do you really want them to implement a logging framework, localizable strings, full unit tests, documentation comments etc etc. I think you want good quality working code, but not full on production quality, it will simply take too long.

An improved approach might be to set a simlar test but in your office with access to reference material such as books or help files but no internet. And you could consider simplyfying the task and reducing the time; e.g. ask them to parse a CSV file and present some results such as the sum of the numbers in column "n". To do this you will have to noarrow down the allowed tools, you probably can't install every SDK known to man on a spare laptop.


My personal view is that you well get better interview results and both you and the interviewee will get more out of the interview if you do a coding test on a whiteboard. This allows you to gauge their approach, theor problem solving skills and thei attitude a lot better. It allows you to help someone who gets an attack of the nerves, and it allows you up the ante if you get a genius. I have to say of all my interview experiences the toughest (as an interviewee) were white board ones, but I think they were the best test of my skills. I have done tests similar to the one you describe on a couple of occassions too, so I know both sides. And as an interviewer I feel the white boarding has given my the best insight possible with in the interview timeframe.

+1 for useful suggestions to improve the test. I am editing "production quality" out of my question. I like the whiteboard too but people do get really nervous sometimes, especially shy programmers. I feel its best to give them a test environment, similar to their work environment. – Yasser Apr 16 '11 at 11:06
@Yasser: You could give candidates a laptop with a beamer. That way, they can sit down and type the exercise, but you still can watch and discuss their thinking process live. – nikie Apr 16 '11 at 11:43

I would not allow general googling, ask on StackOverflow, phone a friend, etc in a technical test. You should be testing the candidate's ability to solve problems by himself.

(Alternatively, get his phone calls traced and make a job offer to his friend instead ... :-) )

If a problem requires the candidate to use a common library, provide links to the library and its documentation ... or download the required stuff to the machine / network used for testing.

I also don't like the problem that you have chosen:

  • Unless the candidate is already familiar with the APIs for programmatic web access, scraping HTML and so on, he / she is going to struggle with getting something done in the timeframe.

  • Unless the candidate is intimately familiar with the technology web scraping (permissive HTML parsers), he / she is likely to come up with a crap solution ... like regex-based "parsing".

  • What you are asking the candidate to do is (I hope!) not an accurate reflection of the way your group operates. But you risk giving him / her the impression that it is.

I agree with some of your observations but not all. Yes, web-scraping is a specific problem but then are there any generic problems? I find it hard to believe that someone with a few years of programming experience does not know structure of html documents and can not create an http connection to retrieve one. Also regexes are not ideal but we are not looking for the ideal solution in such a small time frame. Lastly, i dont know of any programmers these days who do not use google/web/ to assist them. If nothing else, it allows easy resolution to syntactic issues. Why take that away? – Yasser Apr 16 '11 at 9:43
I have to agree with Yasser on the Google restriction. What next, take away documentation and require the programmer to memorise all APIs? Also, how are you going to download a useful library without Google or another search engine? Memorise the URL? Yes, if you're lucky. – Robin Green Apr 16 '11 at 12:09
@Yasser, @Robin - of course I don't expect people to memorize APIs. See my update. – Stephen C Apr 16 '11 at 15:04
As a counterpoint to the 'no google' restriction, I will offer the story of my experience last year where I was sat down at a mac for a programming test. Having never typed on a mac keyboard before, I had no idea how to get it to produce a # symbol. If I hadn't had access to google, I wouldn't have been able to write the very first line of code - That has nothing to do with my ability to program, it is just about familiarity with a tool. I would much rather give people access to the internet and review how they used it than prevent access. – Mark Booth Apr 17 '11 at 0:00
@Mark - and you didn't think to ask "how do I get a '#' on this keyboard" ??? – Stephen C Apr 17 '11 at 7:28

What we do for our Java interviews is to give the user a very simple class to refactor. The class contains 1 bug. We don't mention it explicitly, but they find it when they try to use the class.

When you watch someone try to refactor real-time, it's incredibly easy to see who has real experience and who has been padding their resume.

The class is so simple, if the person is not experienced in refactoring, the candidate will struggle to find things to refactor.

It is important to remember with any real-time test that the level of pressure is extremely high. Mistakes will be made that wouldn't be made normally. So, we don't worry if the person has some difficulty as long as we can tell he's thinking about the problem correctly and heading the right direction. The fact he can't remember the keystroke for "extract method" doesn't matter if he's thinking he needs to extract a method.

I think this is a fantastic approach! You could also observe how the canidate felt about this assignment--if they don't absolutely enjoy it then you have another huge warning flag! – Bill K Apr 3 '13 at 15:53

I like to give candidates actual problems that I encountered recently and was unsure of the best answer. This way I know if the answers are clever or not, and can intelligently discuss them with the applicant when they come in.

For example, When I was writing a speed controller, I could have asked for an algorithm that would compute optimal speed at each time step, to arrive stopped at a given distance as fast as possible, with constraints on min and max accel and velocity. [But I did not interview candidates at that position.]

At my current job, we created a standard test for our applicants where we give them some sample data and ask them to write a program to find "interesting patterns" in that data. It really shows up how (and if) the candidates think. Some focus entirely on statistics, some pick something simple and write something bulletproof, some write a complicated mess of spaghetti, and many end up not ever responding :)

There is a difficulty bias in the first approach. You can compare two candidates only if the problems you posed to them were of the same difficulty level. Also someone with background in physics may do better on your question than someone who is a better programmer. I have given you +1 for the standard test question, i like that. – Yasser Apr 16 '11 at 9:44
It's true, but our goal is not to compare candidates, merely to hire any we can find who seem like they could contribute usefully to our day-to-day work :) – Brad Apr 16 '11 at 19:34

If real problem solving and analytical skills is what you are looking for, and I am assuming you don't do junk work in your office 90% of the time, what's stopping you from just making an assortment of the technical issues your team faced over the last quarter and present them as programming challenges to the candidate?

This helps you decide 3 things:

  1. How good the candidate is when comparing his/her technical solution to what you already have
  2. Aptitude of the candidate in your domain from the kind of responses and questions you hear from him/her
  3. Time bound programming output
This might work in some domains where you can reasonably expect candidates to have a good understanding of your domain (e.g. office applications, online shopping sites etc.). But in most domains I've worked in, it can takes weeks just to understand the problem space. There's no way a candidate could solve such a problem in an interview. – nikie Apr 16 '11 at 14:57
I agree with nikie. I want to keep the test problem domain-neutral. Also like i said earlier we sometimes process 20-25 people for jobs in a week and i dont want to constant spend time updating the test for them. – Yasser Apr 16 '11 at 20:18
I agree with the answer--part of being a good programmer is being able to comprehend a new problem. As long as you spend some time to restrict the problem to a reasonable ammount of info, a good programmer should be able to take it all in. I had one interview where they discussed specific problems they were facing and asked me how I'd approach them--in both of the cases they presented they said they would incorporate/make use of my answers in some way. – Bill K Apr 3 '13 at 15:57

I don't think that is a good test for these reasons:

  1. It's too trivial: get html, process html, write output. You don't need much skills to come up with that that.
  2. The actual implementation is just as trivial if you pick the right tools (curl url | awk '...')

This really is a test to check the programmers ability to select the correct tools. Having that ability is very important of course, but it doesn't say anything about programming skills.

I would instead look for tests that encourages the programmer to show some insight and (potentially) brilliant ways of solving a problem, preferably while working together with others.

Martin, i agree with your comments mostly but id love to hear examples of programming tests you would give which meet some, if not all of the criteria in my question. – Yasser Apr 16 '11 at 20:19

For a serious, shortlisted candidate, I would suggest sitting down together and working together on a real programming problem which you are trying to solve at the time. No, the difficulty level will not be consistent across different candidates. But you will see how well they do programming in the same domain as what your company does "for real". They will see what the design, coding style, etc. of your codebase is like, what tools you use, and how good you are at software development. (If you are really very good, the prospect of working with you will be attractive to the best developers.)

If, on the other hand, you are just trying to quickly filter out candidates who are incompetent, any simple programming challenge will do. You don't need 3-4 hours for that.


I believe tests should reflect the person you're looking for. I'm not really a fan of your test because it kind of lack a job description. Take as example, a firm that searches for a translator from spanish to english. During the interview, the person would be asked to translate a small text about bioengineering. I'm pretty sure that not everyone is capable to translate domain specific language.

For that reason the test should really reflect the job offer. 3h-4h Might be quite long for an interview.

If I had to pass some interviews, I'd split it in multiple small parts.

Ability to find errors in code

The first case is to see how one person is able to find errors in the code that aren't syntax error for example. Functions shouldn't be hard to debug (aka no spaghetti code).

Ability to find bad design choice and offer better solutions

Show some code and ask them to refactor it and make it better. I believe that it's not needed to write actual code and discussion could be enough.

Knowledge of basic datastructures

For example explaining the pros and cons of certain datastructures like LinkedList, HashTables, B-Tree etc. It is still language independent in some ways and is really important to understand. Sometimes people even reinvent the wheel with algorithms that works much more worse than if they used a proper datastructure for the right job.

Writing some code in a simple non existent language

I'm not sure with that one but this is my idea that most languages look alike. A good programmer should be able to write some piece of code right away. If the syntax of the language you're designing is simple enough, it shouldn't be harder than writing pseudo-code.

I would add, Ability to explain big-O, basic data structures and algorithms. Can't tell you how many candidates couldn't distinguish adequately between doing lookup in an unsorted array vs. sorted array vs. hash table. – John Pirie Feb 9 '14 at 20:23
@JohnPirie that's right – Loïc Faure-Lacroix Feb 9 '14 at 22:08

Not the answer you're looking for? Browse other questions tagged or ask your own question.