I'm working in the development of an old project written in Java. We have more than 10 million LOC and, even worse, more than 4000 functional tests.
The tests, scheduled by Hudson, are failing like mad with every bigger code change. Verification of the test failure - if it's a problem in the product or in the test, takes months. We cannot remove the old tests because we don't know what they are testing!
What we can do? How to proceed with such amount of legacy tests?
Abandon them.
I know it's hard to let go of something that was clearly a lot of effort to produce, but the tests aren't working for you, they're working against you. A test suite is supposed to give you confidence that the system does what it's supposed to do. If it doesn't do that, it's a liability instead of an asset. It doesn't matter whether the system or the tests are at fault - as long as running the test suite signals huge amounts of errors, it cannot fulfill its purpose.
What you need now is a new suite of tests that runs with no errors. That means it will initially have little coverage, in fact almost no coverage. Every time you fix or take the time to thoroughly understand something about your system, you reord that knowledge in a test. Over time, this will produce a new safety net that you can build on in the future. Trying to patch an old, ill-understood safety net is a time sink that is almost never worthwhile.
I would even advocate against transferring tests from the old suite to the new suite. Sure, some of them may succeed now, but is that because they are testing exactly what they are supposed to test, or just because some random shots always hit the target? Obviously you have to be pragmatic about what can and can't be done with the effort you have avaiable to spend, but you can't compromise on the principle that a test suite must run cleanly in order to do its job.
Go and fix the tests.
Your biggest mistake is that you allowed tests to fail, and you obviously ignored it for a while. What you have is not "legacy tests" - you are working on a legacy code. And I consider every code written without tests to be legacy.
Verification of the test failure - if it's problem in product or in test, takes months. We cannot remove the old tests because we didn't know what they are testing!
It looks like there is a even bigger problem in your organization, since you are not working with clear requirements. I can not understand that you (or someone else) can't confirm the right behaviour.
Tests are valuable. At the very least, they record that somebody considered that they should spend time writing them, so presumably they had some value to somebody once. With luck, they will contain a complete record of all the features and bugs that the team has ever worked on – though they may also just have been a way to hit some arbitrary test coverage number without being carefully thought through. Until you look at them, you won't know which is the case here.
If most of your tests pass most of the time, then just bite the bullet and invest the time in figuring out what the few tests that fail were trying to do, and either fixing or improving them so that the job will be easier next time. In that case, skip forward to the Determine the intent for each test section, for some advice on what to do with a small number of failing tests.
On the other hand, you may be faced with a Red build now, and hundreds or even thousands of tests which haven't passed for a while, and Jenkins hasn't been Green for a long time. At this point, the Jenkins build status has become useless, and a key indicator of problems with your check-in is no longer working. You need to fix this, but can't afford to stop all forward progress while you tidy up the mess in your living room.
In order to keep your sanity while performing the required archaeology to determine what value can be recovered from the tests which have been failing, I'd recommend the following steps:
There are several ways you could do this, depending on your environment, which you don't clearly describe so I can't really recommend any particular one.
Some frameworks support the notion of expected failures. If yours does, then this is great, as you'll see a countdown of how many tests are left in this category, and you'll even be informed if some of them start passing unexpectedly.
Some frameworks support test groups, and allow you to tell Hudson only to run some of the tests, or to skip a group of the tests. This means you can occasionally run the test group manually to see if any are now passing.
Some frameworks allow you to annotate or otherwise mark single tests to be Ignored. It is harder to run them as a group in this case, but it stops them from distracting you.
You might move the tests to a source tree that isn't normally included in the build.
In extremis, you could delete the code from the HEAD of the version control system, but this will then make it harder to recognise when the third phase has been completed.
The goal is to get Jenkins to go Green as soon as possible, so you can start moving in the right direction as soon as possible.
Resolve to add new tests as you add or modify code, and commit to keeping all the passing tests passing.
Tests may fail for a variety of reasons, including the fact that they weren't well-written tests to start with. But once you get Jenkins green, keeping it that way is really important.
Get used to writing good tests, and make it a big deal if tests start failing.
Go through the disabled tests one by one. Start with the ones that affect the modules that you change most frequently. Determine the intent of the test, and the reason for failure.
Does it test a feature that was removed from the code base on purpose? Then you can probably delete it.
Is it catching a bug that nobody has noticed yet? Reinstate the test and fix the bug.
Is it failing because it was making unwarranted assumptions (e.g. assuming button text would always be in English, but now you have localised your application for multiple languages)? Then figure out how to make the test focus on a single thing, and isolate it from unrelated changes as best you can.
Does the test sprawl through the whole application, and represent a System test? Then remove it from your main Jenkins test suite and add it to the Regression suite that runs less frequently.
Has the architecture of the app changed beyond recognition, so the test no longer does anything useful? Delete it.
Was the test added to artificially increase code coverage statistics, but actually does nothing more than confirm that the code compiles correctly and doesn't go into an infinite loop? Or else, the test simply confirms that your selected mocking framework returns the results you just told it to? Delete it.
As a result of this, some tests will stand, some be modified, some split into multiple independent, bite-size chunks, and some removed. As long as you are still making progress with new requirements, setting aside a little time to deal with technical debt like this is the responsible thing to do.
4000 tests is an intractable problem. 40 tests is more tractable. Randomly select a manageable number of tests to run and analyze. Classify the results as:
If a lot of the tests fall in the first category, it may be time to throw out your current test suite and put together a useful one for the current code.
If a lot of the tests are failing in ways that tell you about a problem in your code, you need to work through the failing tests fixing things. You may find that fixing one or two bugs makes a large number of tests run.
If this statement is true,
The tests ... are failing like mad with every bigger code change.
then that implies that if you rollback to the code just before a "bigger code change", then many of the tests will pass again. After doing that, grab a smaller chunk of changes and see which tests are newly failing. This will help you better isolate which code changes are causing which tests to fail. For each test, once you have isolated the problem, you should then be able to determine if the new code was flawed, or the test was. If it is a problem with the new code, make sure to compare it with the latest version just in case that particular bug has already been fixed.
Repeat until you have the latest code base.
This may seem like an overwhelming task, but it's very likely that once you go down this path and begin isolating some of the problems, a pattern will begin to emerge which may greatly speed up the process. For example:
If you don't know what they are testing, remove them until you do know. Tests are fluid things, if you remove a feature that is no longer required, then you should expect to have to change the test that tests that feature! So unless you know what the tests are testing, you have no hope of changing the codebase with them in place.
You can get the test system set up on developer's machines and run there so the devs can see what parts the tests are interacting with, hopefully provide this missing documentation, and become more familiar with the codebase that you are either not changing correctly, or no longer testing correctly.
In short - if your old tests are failing when you make changes, your code changes are not good. Use those tests as a means of education in how the system works.
@Ignore annotation -- you can keep your tests, but not execute them. Then it's simply a matter of re-enabling them and fixing them one at a time. It allows you to narrow your focus down to just a handful of tests at a time, instead of being overwhelmed with thousands of failures.
The most important thing I would do is go back to the fundamentals of what testing is supposed to do, and what the business needs to keep moving. Testing's job is to identify issues before they become expensive to fix later. I think the key word in that sentence is "expensive." These issues need a business solution. Are expensive issues showing up in the field? If so, testing is failing outright.
Your management and you need to come to a reality check. You are finding that development costs are skyrocketing because of a legacy set of tests. How do those costs compare to the costs of delivering a faulty product because you disabled tests? How do they compare to the onerous task of actually figuring out what behaviors users need (which are the things that should be tested)?
These are issues that need business solutions because they touch the business side of the job. You are delivering product to a customer, and that is a boundary which business is very interested in. They may be able to identify solutions that you, as a developer, cannot. For example, it may be reasonable for them to provide two products: one "legacy" product for those who need reliability and are willing to forgo new features, with one "visionary" product which may have more faults, but is pioneering ahead. This would give you an opportunity to develop two independent sets of tests... one legacy one with 4000 tests, and one with more of the tests you think need to be done (and document them so this process doesn't repeat).
Then, the art begins: how can you manage this two-headed beast so that advances in one branch also help the other branch? How can your updates to the "visonary" branch flow back towards the "legacy" branch, despite rigid testing requirements. How can continued customer requests on the "legacy" branch better shape your understanding of the requirements your legacy customers would need if you eventually re-merged the products?
We cannot remove the old tests because we didn't know what they are testing!
That's exactly why you should remove the old tests! If you don't know what they are doing, then failure is meaningless and running them is a waste of time. Throw them out and start over.
