Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

As part of an interview process for a senior software development role, we are being asked to create a language-agnostic 'exam' that can be sat as part of the face-to-face interview.

The reason being that we will need to produce non-subjective empirical evidence as part of the justification of who is offered the role.

How would you structure/format it to give the most reasonable results possible?

What sort of questions would you ask?

(Note: this is not the majority of the interview process, just a small piece we must do)

share|improve this question

closed as too broad by gnat, Ixrec, MichaelT, GlenH7, ChrisF Apr 18 at 19:09

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs. If this question can be reworded to fit the rules in the help center, please edit the question.

"Language-agnostic" and "non-subjective empirical" is a pretty tough combination. The former tends to force you in the direction of design and architecture, and it's pretty hard to avoid subjectivity when judging someone's answer to those sorts of questions. –  Carson63000 Mar 31 '11 at 11:27
I agree with @Carson63000. Does your company support so many different languages that any language is allowed, or are you approaching this from a standpoint that if Candidate X is proficient enough in Language A that you automatically assume they can be equally proficient in Language B in a short period of time? –  oosterwal Mar 31 '11 at 15:01
@Carson & @oosterwal: Actually, it is because there is next to %0 chance that they will know/heard-of the language we use. –  Dan McGrath Mar 31 '11 at 19:06
Is it MUMPS...? –  Matt Mar 31 '11 at 21:16
@Matt - Hopefully not, although quite a few developers have heard of MUMPS. –  rjzii Mar 31 '11 at 21:50

5 Answers 5

up vote 2 down vote accepted

Aha! The situation is clearer after your "..there is next to %0 chance that they will know/heard-of the language we use.." comment! My first two programming jobs were with relatively obscure 4GLs under VMS, where the employer basically didn't have the option of hiring someone with experience.

How about something along the lines of problems involving reading and understanding pseudocode? For example, give the applicant pseudocode for an algorithm which has a bug that will cause it to fail under certain input conditions. Ask them to determine what condition would cause it to fail, and to add whatever checking/handling would be needed to handle that situation. Or ever just give them pseudocode and input and ask them to figure out what the output will be.

By focusing more on reading and debugging pseudocode rather than writing it, you would get questions that would have non-subjective right or wrong answers, and would test their general aptitude for logic and control flow. Obviously you could tailor the algorithms to an area that would have some relevance to your particular work, whatever that might be. String processing, heavy maths, whatever.

share|improve this answer
+1. Thanks @Carson. Yes, reading/debugging on-topic pseudo-code is definitely a good direction to head. –  Dan McGrath Mar 31 '11 at 20:58

never works. Either the "exam" will become the sole part of the "interview" that counts, or its results are so useless they should be discarded (or in practice, more likely than not both will happen, and you end up with employees who do well cramming for the useless exam rather than those who're really suited for the job).

So either 'fess up that you're just going to put supposedly senior candidates through the type of experience that's degrading even for juniors, where they're judged worthy to work for you or not based on a list of idiotic questions and some "aptitude test", or judge the candidates in a back and forth discussion with them where the content of the discussion evolves during the interview as long as all the basics are touched (like whether the candidate indeed meets the requirements, which is likely a formality when interviewing people for a senior position).

share|improve this answer
If you don't want people to cram for the exam don't tell them in advance. That's what happened in the interview for the job I currently hold. –  Gilles Mar 31 '11 at 12:05
you'd still tend to get people who specialise in answering exam questions rather than having applicable knowledge. Those who're good at an actual job but poor at answering rote questions with stock answers fail the exam and get rejected as unsuitable when in reality they aren't. Seen that too many times. –  jwenting Mar 31 '11 at 12:16
+1 Couldn't agree more if I really put my mind to it. –  pdr Mar 31 '11 at 12:28
I recently sat through part of an interview exam in which I was asked to identify the correct esoteric compiler message for a given part of broken code. Honestly, I have no idea - that's what I have a compiler for. Believe it or not, I'm usually too busy working to remember the exact phrasing of compiler messages when I have a perfectly good and very expensive computer to do that for me. The interview told me considerably more about the company that it did them about me. I didn't bother finishing the test. –  Ant Mar 31 '11 at 12:32
@jwenting, I disagree. I think the exam would be very useful in that it'd give the candidates empirical reasons not to work at such a pace. –  GrandmasterB Mar 31 '11 at 20:49

First off, I agree with jwenting in that an exam might give you mixed results if you are testing candidates, more so if it language-agnostic than if it is not as you are going to have to decide what to test people on since it's not strictly about how well you know a given language. Furthermore, if you are interviewing senior level candidates then you are better off looking at their CV to ensure they have a track record of deploying applications. Concerns about language learning abilities could be checked for by how varied their background is: the more languages they have worked with, the more likely they can learn new languages with a fair degree of ease.

In terms of a non-subjective, empirically based test, this is really hard to do as it limits what the test can be as most tests that would involve someone writing pseudo-code would subjective as developers will usually arrive at different solutions to the same problem if it is complex enough. Ensuring a test is non-subjective and empirically based is a difficult problem to cover come and is part of the reason that most standardized tests are multiple choice since there is only one correct answer.

However, if you must have some sort of test to give the candidates, your best bet might be to have a short list of pseudo-coding problems or general design that would focus more on high level understanding of the algorithms or design patterns to be used. To try and minimize the degree of subjectivity in regards to which solutions are ideal you would need to give those problems to a reference group (i.e. senior developers at your company) so that you have a set of acceptable solutions that can be used when reviewing the solutions that are given by the candidates.

share|improve this answer
Thanks Rob Z, we have already planned to have several developers at our company sit/review the short 'exam' as you also suggest. I agree it isn't the best solution, but due to our awesome company politics, it is something we must do (as I've explained in some of the other comments). –  Dan McGrath Mar 31 '11 at 20:40

Can you explain some more. If it is used to justify the offer that would suggest someone else won't get the offer because their test score was lower but they could have performed better in the rest of the interview? You said it's only a small part of the interview, but forms the butt of the justification for offer which is a contradiction in my eyes.

Does it have to be language agnostic? If the role isn't language agnostic, surely the test should not be either? Otherwise you're criteria for selection are wrong.

What are empirical measures for team fit? You could actually ask HR that (I assume this is driven by company HR policy). The goal isn't to show how wrong their approach is, they may actually have an answer, there is a lot of research on this stuff.

If it's only going to be a small part of the interview, that means it will have to be short, which would suggest not many questions. There is no language agnostic empirical measure that I could even begin to describe that would be useful with a small question set.

My suggestion:

There are a number of third party companies who provide technical tests online, tailored to specific roles. I do them occasionally, companies use them to filter down the candidates. If you adopted one of those, you have introduced a layer of impartiality, you have set a technical bar to be reached already and the test providers do all the hard work. There are two companies I've sat tests through; IKM and BrainBench. You do them at home, it takes an hour or so and gives a pretty fair indication of ability.

share|improve this answer
I've taken a couple of BrainBench tests and found them to be nearly useless in evaluating developers -- they were focused on details of language syntax and libraries. This is fine for someone who is currently working in that language, but useless for candidates who have most recently been working on something else. –  kevin cline Mar 31 '11 at 16:12
It isn't like we are going to choose the highest scorer regardless. It is because we need to show that whoever we picked is at least in the top x% of candidates. Our biggest issue is that we will be having people apply for whom will not know the basics most people would take for granted in a senor dev role. If we screen them out, we need a non-subjective metric to justify it so 'certain' people cannot kick up a fuss. Some times real-life company's suck, I know. :( –  Dan McGrath Mar 31 '11 at 20:33

Exams and standard questions in the long run won't be too helpful. Most recruiters ask candidates who have interviewed at specific companies what questions they were asked. And most candidates tell them because they want to be considered for any future openings that come up. Anyways, the recruiters build up a database of questions and ensure that any future candidates of the company are provided the questions. So some candidates will go into the exam knowing the questions and hopefully answers while others will be at a disadvantage because they don't know the questions ahead of time. So you won't be comparing candidates equally.

share|improve this answer
We don't hire through recruiters, so that part is a non-issue. –  Dan McGrath Mar 31 '11 at 20:37

Not the answer you're looking for? Browse other questions tagged or ask your own question.