Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am lucky in the sense that I have been given the opportunity to be a 'Technical Troubleshooter' for our offshore development team. While I am confident and capable of dealing with most issues, I have come across something that I am not. Based on initial discussions with various team members both on and offshore, a requirement for a 'repeatable, consistent' skills assessment has been identified.

In my opinion, the best way to achieve this would be a combination of objective and subjective tests. The former normally being an initial online skills assessment on various subjects, for example General C#, WCF and MVC. The latter being a technical test where the candidate would need to solve various problems and (hopefully) explain the thought processes involved with the solution whilst doing so.

Obviously, the first method is consistent, repeatable and extremely accurate. The second is always going to be subjective and based on the approach, the solution (or possibly not) and other factors. The 'scoring' of this is also going to be down to the experience and skills of the assessor and this is where my problem lies;

  • The person that is expected to be the assessor initially (me) has no experience.
  • The people that will ultimately continue this process for other people will never remain the same due to project constraints and internal reasons, this changes the baseline for comparison.
  • I am not aware of any suitable system that can be classed as consistent and repeatable for subjective tests with the 2 factors above, let alone if those did not exist.

So anyway, I have to present a plan that will ultimately generate a skills/gap analysis and it is unlikely that I will be able to use an objective method (budget constraints most likely reason). The only option left is the subjective methods and the issues above.

Does anyone have any suggestions for an approach that may tick all the boxes?

share|improve this question

closed as primarily opinion-based by durron597, GlenH7, gnat, MichaelT, Snowman Jul 7 '15 at 2:19

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise.If this question can be reworded to fit the rules in the help center, please edit the question.

good practice! I want that kind of assessment as well. – Hoàng Long Jul 20 '12 at 8:27

A "skills assessment" to me sounds like the classic problem of mixing requirements with solution.

I would backtrack and document the business problem and expected benefit. I don't know the specifics in your case, but an example might be:

Problem: "40% of the candidates we hire do not meet our needs, which causes our team to lose 200 hours of productivity each month."

Expected benefit: "Improve candidate selection to reduce our costs to no more than 40 hours per month."

Estimate: "One FTE month of effort."

Month 1: Result = -140 hours invested Month 2: Result = 20 hours saved = Monthly savings (200 - 40) - 140 implementation hours Month 3: Result = 160 hours saved = Monthly savings (160) + 20 hours accumulated 1 Year: Result = ... 2 years: ... 3 years: ...

Getting to the heart of the problem opens up alternatives that may not be considered otherwise.

The problem with "consistent tests" is that the questions you ask will eventually leak to your candidates, and will no longer be consistent.

You could try solving this by creating many questions and only providing a subset to each candidate, however you would not achieve consistency until a sufficiently large population took your tests.

I suggest you read up on ways to predict IT performance. You are likely to find some surprises, and tests are not required.

share|improve this answer

With regard to assessing talent, let me point you at Joel Spolsky's book, "Smart and Gets Things Done". It is a good, quick read and among other things, stresses that the range of ability between applicants may vary by a factor of 10, 20, or even more.

Having many times interviewed candidates for software development positions, I find candidates are pretty good at differentiating themselves in both subjective and objective tests. Unless you are hiring a great many positions, you might benefit by weighing the cost of an automated test vs. a manually administered test.

I would like to suggest an approach borrowed from software testing that is called operational profile based testing (OPT). In OPT, we identify and operational profile (OP) by observation what features are used most often. To refer to the Spolsky book in which he describes a Yale programming class that had developers write some pretty mundane programs with names like CMDLINE99, MAKE01, or TAR00 to show they could, and measure how fast (i.e. productive) they could be, but the instructor considered his programs to be typical of what professionals had written previously and what his students would write later.

To make an operational profile for the questions or programming problems for your candidates, pair program or just spend an hour or two watching a good performer in a similar role. Sample the activities and code that they do. Discuss problems they are solving, and jot down vocabulary that might differentiate between novice and expert.

If the test is well designed, I suspect success on this kind of test could correlate strongly to fluency in key skills, low need for supervisory assistance, and the ability to complete most portions of the task in a timely fashion.

Hope this helps, thanks for reading this answer.

share|improve this answer

The best interview question I ever had to do was to do a code review. They gave me the IDE with a project in it and said "we've written this but imagine its a junior dev who's just done this and its your job to review it - show us the bugs, the things you'd do differently". And so I got to show them I knew about code organisation and best practice as well as the usual trivial bits of programming.

Try that, considering there are no explicit "answers" you can come up with something that is gauging their thought processes as well as a few objective tests (ie a bug or obvious error) that they should find too.

share|improve this answer

For the second type of test (subjective), you may be able to follow a process such as this. Try and break up each of the topics for which skills are important to you, into groups. So C# can have groups such as:

  • String handling
  • OO programming
  • IO
  • Threads

You could do this for other topics as well.

Then for each of these groups try and write some simple statements. For String handling you could write statements like:

  • A developer should know how to concatanate strings
  • A developer should be able to determine if a String contains a specified substring
  • A developer should be able to split a String based on a regex
  • A developer should be able to determine if a String matches a regex

Then for each of these you can create some test cases which run against empty functions. Once completed these functions will get the test cases to pass.

This could be your subjective tests which can be run automatically.

Though keep in mind that creating your initial collection of tests will probably take a good deal of time.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.