Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

When correcting bugs, it is encouraged where I work to first write a test that fails with the given bug, and then to fix the code until the test passes. This follows TDD practices, and is supposed to be good practice, but I noticed it tends to produce cryptic tests that come really close to the implementation.

For instance, we had a problem when a job was sent, reached a certain state, was aborted, and retried. To reproduce this bug, a massive test was written with thread synchronization in it, lots of mocking and stuff... It did the job, but now that I am refactoring the code, I find it very tempting to just remove this mammoth, since it would really require a lot of work (again) to fit the new design. And it's just testing one small feature in a single specific case.

Hence my question : how do you test for bugs that are tricky to reproduce ? How do you avoid creating things that test the implementation, and hurt refactoring and readability ?

share|improve this question
    
Is that error case relevant to the new design? To me part of a new design would be to increase reliability of the design, so you might be able use that to justify removing this test case if this type of bug was anything to do with a mistake in the original design. –  Thymine May 10 '12 at 17:03
    
the refactor is about something else, and it is still possible in the new design to slightly modify the code and reintroduce the bug, which is what the test is watching. I guess an alternative to the test would be a comment in the code saying "don't fuck with this", but it sounds like the wrong thing to do :p –  Zonko May 10 '12 at 17:24
    
if it's too complex for for a unittest make it part of the integration test –  ratchet freak May 10 '12 at 18:59

4 Answers 4

up vote 18 down vote accepted

Yes, in general you should. As with all guidelines, you'll need to use your best judgement when they run against other guidelines. For this scenario, the severity of the bug needs to be weighed against the work needed to implement the test and the quality of that test in being targeted to the business problem and catching regression of the bug's state.

I would tend to favor writing tests over not, since interruptions to run down bugs tend to have more overhead than simply developing and maintaining a unit test.

share|improve this answer
    
I would add more emphasis to this and state that in an ideal world it would only be considered a bug if a failing unit test exists, but +1 for the italics and noting that business needs should prevail. –  Joshua Drake May 10 '12 at 17:23
2  
well, indeed, in the end it's all about a balance between the time needed to maintain the test vs the time it would take for a noob to detect and correct the bug if it happened again. –  Zonko May 10 '12 at 17:31
    
I would also favor generalizing the test so that it's not just testing aborting and retrying the job when it reaches one specific state, but rather tests aborting and retrying at every state a job could be in. –  Old Pro May 10 '12 at 18:57
    
+1 for "since interruptions to run down bugs tend to have more overhead than simply developing and maintaining a unit test." –  Peter K. May 10 '12 at 19:11

I think the best practice - one I'm embarrassed to admit I don't often follow - is to create a system or integration test that demonstrates the problem observed in production, then research to find the unit(s) responsible for the problem, and then write unit tests for those units that demonstrate the problem at the unit level. Once you have the unit tests, fix the units, and run all the tests. At this point, it may be prudent to discard the original test - because it may be fragile and/or slow - but keep the unit tests in your automated suite for the sake of regression and coverage.

share|improve this answer

The practice of writing a test to identify the defect is a good idea, since it allows you to identify exactly what steps are needed to reproduce the defect and verify that it has been fixed. In addition, these tests can be run as part of smoke testing or regression testing to ensure that later changes have not re-introduced an old defect into the system.

The first thing to consider is the level of the test needed. Perhaps the test to verify the fix would be more appropriate at a systems level, or maybe even an acceptance test that's done manually. I think it's more important to have a test that's documented and managed, regardless of how it's specifically implemented.

As far as how refactoring affects tests, it depends on the specific characteristics. In some cases, refactoring (or any type of work, such as new features) could make the tests no longer necessary. The problem, as it originally occurred, might no longer be possible. In that case, it might be wise to remove the test from the possible tests to make your test process (automated or manual) more lean. In other cases, there are multiple methods of performing the test and verifying the feature at a different level might be more appropriate. If the feature is minor, perhaps the test is no longer necessary at all.

You can also consider not only relying on testing, but logging as well. For example, capturing information at run-time (with varying levels of verbosity depending on the environment - more verbose during testing, less verbose during deployment), profiling the application, capturing dumps of the current state of the system. If you can find common triggers to the problem, you can use that to guide testing at all levels.

share|improve this answer

Yes, you should.

Write unit tests for existing code base. When fixing a bug, you need to make sure that your unit test fails - this will give you confidence that you are indeed working on a bug. You then need to re-factor and make the test pass by fixing the bug.

This is not a TDD practice though. In TDD tests drive your design, but in your case the design has already been decided upon.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.