[I originally only commented, but the OP requested I add an answer, so I'll expand it somewhat and add as an answer - The TL;DR answer is in my original comment]
Setting any kind of development task in a job interview situation has to be approached very carefully, IMHO, however, it's absolutely correct to set such a task as it's very important to get your candidates for a programming position to actually write some code.
The difficulty of course is make the task just difficult enough that you can "test" the candidate's ability to both really think through a problem, then write code towards solving that problem, whilst keeping the task simple enough that you can expect the candidate to produce enough code for you to actually judge their ability in a very small and compressed time-frame.
In interviewing web developers, I've always devised my tests that consist of a complete application. This is somewhat akin to the real-world in that I'll expect web developers to know at least something about the many "layers" of a complete application - the database, the "business logic" code, as well as HTML/CSS and any client-side code (i.e. ASP.NET Markup/Controls etc). I can and do judge candidates on each "layer", but of course, for a web developer I don't expect them to display a "seasoned DBA" level of knowledge about databases for example.
Now, although I use a complete application, I always make it a fairly simple one. Something with a single use or function. My most recent one was a URL Shortener website. Something akin to bit.ly or tinyurl or goo.gl (without any statistics or link tracking). This is a good example, as the functionality of such an application is limited to accepting a long URL and spitting back out a short one. It may include a "preview" page (similar to tinyurl.com), but there's definitely a requirement for some algorithmic logic there in order to produce the shortened URL. (Shameless plug for my own implementation!) The benefits of this are that each single layer is just simple enough to be done relatively quickly (the database, for example, could be as simple as just one table!).
Given that candidates will only have a few hours to complete this task, and given that this is an interview situation wherein candidates are likely to be under additional pressure and display more nerves, I have to take this into account when "judging" their final application. In this respect, this is where the programming task differs quite drastically from the "real world". I believe that as long as you accept and acknowledge that certain factors or metrics differ in certain ways, and then account for that when judging the candidate, you should be ok.
Now for me, the golden rule for this is:
I explicitly stress to the candidate that I am NOT going to judge them on the "completeness" of the application, I am ONLY going to judge them on the quality of the code that they do deliver.
I also always tell the candidate that I expect them to write their very best code for the task, as though this were a real-world, production application I'd just asked them to write. Again, I explicitly state that I would much prefer to see a well-written incomplete application, than a badly-written complete one.
When the time limit for the task is over and I'm reviewing the candidate's code, I don't care if the application is functionally complete or even if it compiles. What I'm looking for is for the code that exists to appear to be good quality code (i.e. SOLID & DRY) and code that demonstrates that the candidate had a good understanding of what was required of the application, and of them.
In response to my original comment, keppla asked:
Is there so much that can be repeated
in an URL Shortener? Or a big chance
for enough classes to demonstrate the
Letters of SOLID?
I tend to agree that it's fairly difficult if not impossible for a candidate to display all of the attributes of SOLID (and DRY) code in such a simple, and probably incomplete, application however it should be easy to show the S in SOLID (Single-responsibility principle) within the code, and if I also see even just a single unit test or even rudimentary dependency injection (or any other mechanism the candidate has employed in order to reduce the dependencies between higher and lower level objects) well, that's a big plus for them. Hell, even the existence of a hand-coded interface shows a level of understanding that lots of other candidates simply won't have.
Ultimately, in judging the candidate's code, I have to ask myself a number of questions:
- Did the candidate understand the problem domain?
- Is this code good quality code?
- Given enough time, would the candidate have completed this application to a good, real-world quality standard?
- Is that time-frame (from the question above) acceptable in the real-world?
That last question is admittedly a bit "fuzzy". You have to try to extrapolate how much time the candidate would have required to complete the application from the code that the candidate did produce.
Although there are more factors involved in making a hiring decision, so long as I can answer YES to all those questions above for the programming task, the candidate will have scored very highly on that element of the interview.
All in all, what you're looking for is: SMART AND GETS THINGS DONE!