Are there coding tests to recruit a QA engineer?

Question

I'm a software engineer in a small company and we are currently recruiting our first QA engineer. The guy will have to write as many as possible automated tests on our web-based tools suite.

As I've participated in the recruitment of almost all the developers of our engineering team, the QA manager has asked me to interview the candidates.

Most of them sound really smart. When they explain their past experiences, they give the feeling that they know what they were talking about, even when they give a lot of details on the framework or tools they are familiar with (like selenium or ohter stuff).

However at the end of the interview I like to challenge them with a small coding test. I know we aren't recruiting a developer but as he'll have to code tests, I believe that he has to be a bit skilled in that domain.

I systematically ask the same question:

"In the language of your choice, could you write a the body of the function that takes in input a integer and return its string representation.
For instance in C#, the prototype of the method could be something like public string itoa(int n)"

And unfortunately, until now, 100% of the candidates have failed...

So, is it because the question is somehow out of scope for a QA engineer? and in this case could you propose a better exercise?
Or is it just unacceptable and we still have to look for someone that would be at least able to pass the test?

Thank you for you answers.

Answer 1

You should ask the candidates to write code, since their work is actually... to write code. The problem is, you are asking is wrong.

• First, in real life, they would never have to write code without an IDE, internet, documentation, etc. That's why it is important to make them write code, instead of asking them questions.
• Second, your question is somehow strange. First, as a C# developer, why would I care? There is already a ToString method for this. It's also not so easy to answer, especially in a stressful environment like during an interview. In my case, I even thought you were asking the inverse: parsing an integer from a string.
• Third, a candidate who will give your own answer has actually no chances to be hired in my company, since the prototype you gave is false (what is the reason for not making this method static?) and does not match C# style convention.

So instead of asking theoretical questions which does not help you to really evaluate the skill level of your candidates, try to give them a larger, real, concrete problem to solve, behind a PC with IDE and internet, without time limitation and without stress being watched by somebody. In your case, I would suggest to ask writing some automated tests for a small hello-world-style application, since you are looking for people who will write automated tests.

---

There are cases where, instead of seeing how do they code, you want to see how do the candidates think in order to solve a problem. In this case, you may want to change the formulation of your question. It's not "How do you [...] in the language of your choice", but more "How do you approach the problem of converting an integer number to text representation". With the first formulation, you're inviting a candidate to:

• Choose a language,
• Code, in her head, a solution to a problem,
• Report you the actual code.

In the second one, on the other hand, you're inviting a candidate to explain the logical language-agnostic approach.

For example, one of my favorite questions like this is:

Given a valid XHTML page, how do you remove meaningless whitespace to minify the source code?

If you ask somebody to actually code this during an interview in a language of her choice, chances are most candidates will never be able to do that, even after a few hours. On the other hand, when you just ask them to explain how would they approach this problem, some may actually give the right answers¹.

---

^{¹ The answer I expect from the candidates is: "parsing XHTML code with an XML parser, then output it, removing meaningless whitespace, while being careful with <pre/> tags, CDATA, etc.". Another answer I generally accept is to use regular expressions, even if this approach is not very correct. Also, a third answer I got once and appreciated a lot was: "Why would you remove whitespace? Isn't it just easier to GZip the page before sending it?".}

Answer 2

It sounds to me like you aren't screening candidates heavily enough before bringing them in for the interview. If this question is a common failure, you should not be asking it last. You should ask some simple coding questions during the phone screen, and eliminate non-coders right away before wasting your time.

I'd also be concerned about why your ads aren't bringing in candidates with the desired level of coding skill. What does your ad look like? Do you mention that you require coding skills right away? Can you adjust the job title? You might want a title like "Software Engineer in Test" or "SDET" instead of "QA Engineer" for a coding-heavy position. I have also seen "QA Automation Engineer", for additional clarity about job responsibilities. What is the pay scale? If you want a tester who can code, you will need to pay them at least as much as a software engineer with the same level of coding ability, and probably more in addition for their testing expertise.

It is not uncommon for QA Engineers to have lots of skills using tools and great skills at finding important bugs, but less skill at coding. Testing and coding truly are two very different skill sets, and good automation engineers need to have both - making them difficult to find. You might try asking your question at http://sqa.stackexchange.com or reading some of the questions asked and answered there, to get an idea of the broad range of programming skills amongst QA professionals, even highly skilled QA professionals.

I'm answering this question as an SDET (Software Development Engineer in Test) with a BS in CSE and 6 years of experience.

Answer 3

Why don't you ask them to code what they said they've done before? Ask them what kind of scripts they've written, and ask them to write a small segment that performs a similar functionality.

Answer 4

When recruiting a QA engineer and if you want them to have a coding test, the coding test should be to write test cases against the method, not actually implementing the method. A developer would implement the method. In your example:

public string itoa(int n)

They should be able to do this:

//Test 1
int test1 = 1;
string expected = "1";
string actual = itoa(1);
Assert.AreEqual(expected, actual);

Or you could give them a implemented method that has bugs and see if they can write the test cases to find the bugs.

I think that is fair as long as they know up front there will be a little quiz.

Answer 5

In my experience the best "write some code to..." questions to ask candidates with are real problems or mistakes that you have previously encountered on your team in your line of work, simplified if necessary. These questions tend to seem straightforward and relevant to the candidate, helps them to understand the types of problems they will be expected to solve, and tend to be a good indicator of ability to solve future problems along the same domain.

Unless the candidate is an expert programmer, the lack of resources like IDEs, reference materials, compilers, time to think through and test their solution, and so forth causes the interview scenario to reflect the real world ability to program poorly, and may make a competent programmer look less than their best, as they may not spend their time rote memorizing things the IDE can do for them easily.

Asking the candidate to walk you through their thought process as they write pseudo-code to solve a particular programming problem would be much more useful. I suppose this is part of why asking candidates to write out an algorithm or two is so popular, it shows they can structure loops and write conditionals and so forth. The more interesting part, of course, is not whether they can make the right answer quickly in an interview with few resources, but whether they think like a programmer, know what resources to turn to when they don't immediately know the answer to something, what testing would they want to do under ideal conditions, how they would decide which test cases are most important if there is not time enough to test everything that would be ideal.

Also, it can be useful to present code samples and ask the candidate to identify logic or syntactical mistakes, in a language actually used on the job. A competent programmer should be able to look at code and say "hey wait a minute, that doesn't look right" more easily and accurately in an interview than I would expect them to be able to produce error-free code on paper from scratch.

For a QA position, I would provide a function or block of code with a written description of the expected outputs that has some obvious mistakes in it. Ask the candidate to identify as many sytnax/logic errors as they can, as well as risky areas in the code they would want to see test case coverage for. Ask them to write or verbalize a list of test cases that would provide adequate coverage of the code to ensure the function works as described and handles error cases adequately, and why they would pick the test cases they are picking. Then I might ask them to try to write out on paper one of those cases, attempting to use actual code as much as possible, using pseudo-code if necessary.