Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Sometimes my QA team reports bugs, but neither I or them have any idea on how to reproduce them. This leads to very long and frustrating debugging sessions which sometimes do not even yield results.

My software is tied heavily with proprietary hardware so bugs can come from many directions at once.

Should I expect more from them than "your software crashed when I pressed a button" or should I figure myself what happened?

EDIT:

One of my coworker pointed out that we are probably all developers here so the results might suffer a little bias

share|improve this question
1  
This isn't really an answer but it's worth pointing out that by putting (more) logging inside your application you can reduce this issue somewhat. Perhaps your test build could be set to a verbose logging mode which would give you vague reproduction steps to assist you in debugging sessions? –  ChrisFletcher Feb 21 '12 at 23:30
    
Not really, Testing should be doing that. QA should be doing QA. –  mattnz Feb 22 '12 at 4:47
1  
Consider adding logic to your product that helps you retrace the steps taken by the user, and have a QA-button that allows the bug reporter to easily extract said information from your product and add it to the bug report. –  user1249 Feb 22 '12 at 13:14
    
At least a screenshot of the actual situation would help most of the time to reproduce the error. (see the logging comment above). Usersnap is a tool for better bug reporting and saving communication time. –  Gregor Aug 21 '13 at 13:10
add comment

12 Answers 12

up vote 30 down vote accepted

QA should always try and make the bugs as easy for you to reproduce as possible and the bug description should contain the steps taken.

However, if they can't easily reproduce the bugs, they should still get entered into the bug database with suitable title/headings and a full description of what they did to cause the bug. The bug description should clearly state that they can't reproduce the bug (perhaps with some comment along the lines of "tried it five times, it happened once"). This way, if someone else sees the same bug, they can add to the bug database with their findings and also you get as much information as possible which further down the line could be vital in saving you time tracking down the problem.

Also, you get to filter the information - there might be a lot of bugs in different systems that you know are all linked to (eg) one area of the code - if QA don't report anything (as they can't reproduce them) then this information doesn't get to you.

share|improve this answer
    
... a full description of what they did to cause the bug. How is that different from a repo? –  Steve Evers Feb 21 '12 at 20:24
13  
@SnOrfus: Repos are, by definition, reproducable. If you find a bug but can't reproduce it later, it's still helpful to know what you were doing at the time, to help track down what caused it. –  BlueRaja - Danny Pflughoeft Feb 21 '12 at 20:28
3  
@SnOrfus: The software crashed vs The software crashed editing foowidgets vs The software crashes when viewing a foowidget and toggling the frobulator 10 times followed by pressing this key sequence (stack trace attached). The last detail may not be obvious, but having the second description instead of the first is certainly helpful. –  Daenyth Feb 21 '12 at 21:23
    
@Daenyth: Fair enough. Maybe I'm getting too far into the semantics of a full description. –  Steve Evers Feb 21 '12 at 21:26
    
Make sure that "Closed Did/Could not reproduce" is available (there and acceptable) to use in your defect tracker. –  mattnz Feb 22 '12 at 4:41
show 2 more comments

It looks like your QA department is doing too much exploratory testing (ie. They don't have a good test plan).

Exploratory testing is good, and identifies problem areas, but from there they should be defining reproducible test cases (ie. a test plan) to perform that will reveal specific bugs.

There's a number of reasons why a correct repro is necessary (not just good, but necessary):

  1. You have to repro so that you can debug/track down the cause.
  2. QA will need to be able to verify the fix once you're done.
  3. Further test passes will need to do regressions on previous bugs.
  4. Known bugs can be discarded if the exposure is too small or repro is too unlikely.

So, as SteveCzetty notes: Close it as no repro and get back to work.

share|improve this answer
3  
The problem with steps to reproduce is that typically with Crash bugs they are not anticipated or looked for and they typically happen in the middle of a test plan. They should try to reproduce it but many times this is imperfect. For manual testing a good desktop screen recording software during test cases can capture every step and timestamp that led up to the crash as well as capture any simple mistakes or seemingly inconsequential details that the QA person might have missed in their steps to reproduce. With this and the test logs a developer should have a more clear picture. –  maple_shaft Feb 21 '12 at 19:02
3  
@maple_shaft This is indeed true - I don't expect every bug from QA to be 100% reproducible. Screen recording is an interesting option and definitely bears more consideration, because it allows more flexibility without giving up the opportunity to watch over the tester's shoulder. –  Michael K Feb 21 '12 at 19:09
2  
@maple_shaft: I agree, and MTM has that built-in. In this case though, there's a significant difference between what Eric is getting ("There was a crash") and what the best case scenario is (Event Logs + Application Logs + Video + Action Recording + Intellitrace + Itemized Repro). IMO/E QA's job doesn't end when the blue screen happens - and a repro is the first thing they should be trying to produce, even if it's not always feasible. –  Steve Evers Feb 21 '12 at 19:40
    
@SnOrfus I could only dream of working with a QA team that is so professional. In most organizations I have worked they were the dregs who were too incompetent or lazy to cut it as software developers. Suprisingly the best QA team I have worked with was completely offshored, probably because they actually want to be doing QA testing. –  maple_shaft Feb 21 '12 at 20:06
    
@maple_shaft: Where I work, before I moved from QA into Dev, we were doing most of that already (video takes up craploads of HDD space when you have 400000 test cases). –  Steve Evers Feb 21 '12 at 20:45
add comment

If the bug can't be reproduced consistently, how will QA ever know whether it was fixed?

share|improve this answer
    
Yes this is another problem I didn't mentionned but run into a lot. I get a bug report, make changes then close the bug. QA then approves the close. A few week later, the issue gets reopened as not fixed. I also have a lot of issues as "the software crashed", which becomes a large melting pot of every possible bug –  Eric Feb 21 '12 at 21:54
add comment

Yes, you should expect more from them. They should be able to say:

1. Started new instance of program
2. I pressed button A
3. Entered "test text" into the TEST NAME field on Form #1
4. Pressed button B
5. Observed that the program crashed with this message (see attached screenshot).

If all they tell is "it crashed", they're not very good. Even if the above steps are not 100% reproducible, a large enough sample of these crashes might help narrow down the cause, once a pattern is detected.

share|improve this answer
add comment

My advice is to read those bugs and give them a good old think. If you can't figure out a potential cause, forget about them for now.

QA should document every issue they find, even if they have no idea how it happened. It's QA's job to try and reproduce issues, but realistically this won't always be possible. Sometimes it doesn't have anything to do with what they did in the last 10 minutes. Something got into an invalid state a day ago, and it just became apparent because of one of the last 10 steps.

With these "1 in 1000" bugs, QA should try to reproduce them for a bit. If they don't have success, they should document the bug, then try a little more.

The reason why they should get the bug entered fairly early on is that the programmer knows the code a lot better than QA, and might immediately know the problem. It could be the code they refactored. It could be that function they half implemented then forgot about. They may have no idea, but there's no sense in the tester wasting a few hours trying to reproduce a problem that's obvious to the person who coded it. The tester can always add more details to the bug later.

The bug should include as much info as possible. Whatever the tester can remember about the lead-up to the issue should be written down in painful detail. Any Crash logs, database snapshots, or relevant screenshots should be attached as well.

If the bug is never reproduced, great! It doesn't hurt having it in the database. If the program is released and a user reports a similar bug later, you can compare their experience to what's in the report and look for any similarities.

In my experience, the juiciest bugs aren't found from following test plans. Sometimes you have to let things stew for a few weeks in order to have the moon and stars align that cause a nasty bug. If QA can do some detective work and find some possible causes, give them a pat on the back. But sometimes, who knows what happened?

share|improve this answer
    
+1 for "It doesn't hurt having it in the database" –  PhillC Feb 23 '12 at 0:25
add comment

A lot of bugs aren't reproducible until you know how to fix them. That doesn't mean they don't need to be fixed. I fixed a bug last year that I still don't know how to reproduce. It requires some bizarre combination of a precisely-timed transmission error together with very specific garbage data in a certain memory location on the stack. Unfortunately, one of our customers is "lucky" enough to get into that condition all the time.

So, by all means encourage QA to include reproducability steps where possible, but don't fault them if they can't. Sometimes it will help you know where to add logging. Sometimes all it does is tell you what doesn't cause the bug, but a bug report is always useful.

share|improve this answer
add comment

If you mean should QA include the steps to reproduce the bug if they can, then the answer is yes. If you mean should they only record bugs they are able to reproduce, then absolutely not. Bugs are bugs, whether they only happen at midnight on a full moon, or every time you look at it. Some bugs are non-deterministic (classic example is uninitialized variable, where the value picked up is semi-random), that doesn't mean they shouldn't be recorded, investigated, and if possible fixed.

Non reproducible bugs should generally have a low priority, but they should definitely be recorded.

share|improve this answer
add comment

IMO, you should. QA aren't doing their job if they can't give you any reproduction steps. Don't waste your time trying to reproduce something that you can't, just close it as "Cannot reproduce" and move on.

Your time is much more valuable than that.

share|improve this answer
add comment

A bug report should contain:

  • Steps to reproduce
  • Actual behaviour
  • Expected behaviour

E.g.:

  • I deleted all suppliers from the database (using DELETE * FROM tSuppliers), opened the supplier dialog, and clicked on the Supplier drop-down list.
  • The list contained the following message: GUPOS ERROR #0000000: SOMETHING WENT WRONG!. When I clicked on the message, the app disappeared from the screen, and the Task Manager.
  • I expected to see either an empty list or (preferably) a message such as "No suppliers found". Clicking on the empty list or the message should have no effect. The app obviously shouldn't disappear without warning under any circumstances.

So, yes - it should contains the steps to reproduce. The fact that they don't feel the need to include this would seem to indicate that they think their job is to "break the software", rather than to identify faults.

share|improve this answer
add comment

QA should be able to reproduce the bugs based on the steps entered. If they tried hard , still could not reproduce, they should still enter the bugs with as much as details they have with the timestamp so that the developers can take a look at the application and debug logs for more details.

share|improve this answer
add comment

Money is at stake here. Why should any team member be able to create a poorly-defined, low-chances-of-success task that burdens a (hopefully) highly-paid developer?

This isn't about pecking order, hierarchy, arrogance, us vs. them, or anything like that. This is just about investing in activities that add value to the product.

When a problem can be demonstrated to negatively and measurably affect the value of the product, then it should be investigated, reproduced, and fixed. Fix your product defect pipeline to filter out the noise.

share|improve this answer
add comment

your QA team sucks

Go to them and tell them to read a document that any professional tester has to have printed right inside their brains (I was a tester once and I still have it right there, in my brain): How to Report Bugs Effectively.

Particularly, point them to the section "Show me how to show myself":

This is the era of the Internet. This is the era of worldwide communication. This is the era in which I can send my software to somebody in Russia at the touch of a button, and he can send me comments about it just as easily. But if he has a problem with my program, he can't have me standing in front of it while it fails. "Show me" is good when you can, but often you can't.

If you have to report a bug to a programmer who can't be present in person, the aim of the exercise is to enable them to reproduce the problem. You want the programmer to run their own copy of the program, do the same things to it, and make it fail in the same way. When they can see the problem happening in front of their eyes, then they can deal with it...


If they start yelling at you complaining that "bugs can come from many directions at once", tell them they suck even more than you thought before. Tell them that Testability is a feature they should evaluate among other project features and they should invest efforts into getting this feature right.

  • Testability improvements one could get when there is a professional tester focused on it could be much like magic. I learned that myself few months ago. QA engineer working with our team gave me a handful of testability requests to develop for some components in our application. Things he asked about made very little sense to me but I just gave it to him assuming that he knows better as a professional. Soon after I finished, he came up with a tool that reduced testing efforts by order of magnitude. He said most of it was based on these cryptic requests I implemented, go figure.
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.