Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We have three sets of test suites:

  • A "small" suite, taking only a couple of hours to run
  • A "medium" suite that takes multiple hours, usually ran every night (nightly)
  • A "large" suite that takes a week+ to run

We also have a bunch of shorter test suites, but I'm not focusing on them here.

The current methodology is to run the small suite before each commit to the trunk. Then, the medium suite runs every night, and if in the morning it turned out it failed, we try to isolate which of yesterday's commits was to blame, rollback that commit and retry the tests. A similar process, only at a weekly instead of nightly frequency, is done for the large suite.

Unfortunately, the medium suite does fail pretty frequently. That means that the trunk is often unstable, which is extremely annoying when you want to make modifications and test them. It's annoying because when I check out from the trunk, I cannot know for certain it's stable, and if a test fails I cannot know for certain if it's my fault or not.

My question is, is there some known methodology for handling these kinds of situations in a way which will leave the trunk always in top shape? e.g. "commit into a special precommit branch which will then periodically update the trunk every time the nightly passes".

And does it matter if it's a centralized source control system like SVN or a distributed one like git?

By the way I am a junior developer with a limited ability to change things, I'm just trying to understand if there's a way to handle this pain I am experiencing.

share|improve this question
6  
I have no idea what software you are working on, but a small test suite which takes hours to run is some WTF. If they would run faster this would be easier, no way to streamline your tests? –  Benjamin Bannier Nov 20 '12 at 8:58
2  
what's so "extremely annoying" about trunk being unstable? Don't know if you know, but one of the most popular branching strategies is even called unstable trunk –  gnat Nov 20 '12 at 9:54
1  
There are many ways to optimize a test suite (like for any other software). I have no idea why yours take so long, but you might e.g. be able to reuse some of the test environment or just use better algorithms/data structures when running (profiling helps). Might also be that nobody has take the time to identify representative samples and you just test every possible input/output against some reference. Maybe your buildsystem allows you to encode code-test dependencies so you don't need to run the full set. And I know this wasn't your question, that's why I made it a comment, not an answer. –  Benjamin Bannier Nov 20 '12 at 10:08
1  
...hmm, your better option is likely to improve tests and application logging so that it's easier to find the failure reason. That way, one would need to find and fix the cause of the failure instead of wasting efforts on "detective investigations" looking for who, when and why changed particular line of code... –  gnat Nov 20 '12 at 10:19
1  
@honk Some tests just take a long time to run. I work for a company that makes data acquisition equipment and our "partial" test run takes about an hour. The tests need to do various types of measurements and that just takes time. –  Velociraptors Nov 21 '12 at 1:50

7 Answers 7

up vote 1 down vote accepted

The only way to fix the root cause of the instability is to decouple the code so changes are more isolated, as other answers have suggested.

However, as an individual developer, if you want a more stable build for you personally to work on, that's relatively easy to solve. Instead of working off of the tip, you only pull the last build that passed the overnight test suite into your working tree. If you can create feature branches for each change, then branch off of the last stable build.

Yes, your tree will be a few days behind, but most of the time that doesn't matter. Do your work against the stable build, so you know your changes are the ones that broke any tests, then before you check in, update to the latest and do your normal integration. Then after you check in, back up to the last stable build again.

You still have to do the messy integration work, but what I like about this method is it isolates the integration work to a time more convenient for me, and gives me a stable code base for development when it isn't convenient. I have a much better idea when it's my changes that likely broke the build versus someone else's.

share|improve this answer
1  
-1 working from branches is viable option, but recommending it without suggestion to test could do more harm than good. Only testing may show whether it's feasible for specific project or not. Eg in a project I did about 2 years ago, such a testing has shown working from branches was taking ~7x times more efforts compared to unstable trunk –  gnat Nov 20 '12 at 16:25
    
Thanks Karl! While this is not what I was hoping to learn, this is a very practical approach which could help me solve the issue at hand. And I agree that working a few days behind the trunk will only rarely cause integration issues. –  Oak Nov 21 '12 at 14:05

I know you're trying to avoid this, but the real insight here is to realize that something is seriously wrong with your codebase: you need to run a full suite of tests that takes a week just to be sure your code is stable!

The most advantageous way to fix this problem is to start separating your code base and tests into (independent) sub-units.
There are huge advantages to this:

  • The tests for each of those units will run faster (there are simply less of them), and they will not break if something goes wrong in one of the independent or downstream units.
  • A failed test will be pinpointed to a particular unit which will make it much easier to find the source of the problem.
  • You can separate the VCS locations of the different units so that your "stable" branch can be a pick 'n mix of the latest successfully tested build of each unit, so that a broken unit or two doesn't destabilize your "stable" version.

On the flipside management of your VCS structure will get more complicated, but at a full week for your full test, I think you can take the pain!

I still recommend using a "stable" and "development" branches strategy in some form or other, but there are many ways to go about that and you can pick the one that works best for your organization (meta-repositories with fixed revisions pointing to separate repositories for each unit, a stable branch and a dev branch, feature branches....)

share|improve this answer
    
I never said the large test is an atomic test, it's a test suite. When an individual developer makes a modification to element X, he runs the tests relevant to X - no matter which test suite they originated from. This is in addition to the weekly test, which is ran to make sure that a change in one place didn't unexpectedly affect another place. But you make an interesting point that at least separating things this way will help speed up the tests for specific modules, while keeping the risk at roughly the same level. –  Oak Nov 20 '12 at 10:31
2  
@oak - Well, in a way the suite IS atomic if running all of it is the only way you're actually confident that the code is stable, but you do make a good point, so I've edited my answer. –  Joris Timmermans Nov 20 '12 at 11:07
4  
We have huge testsuites for our compilers, some of which takes several days to run, and I don't think that is uncommon for software as complex as a C++ compiler. It's not that the suite defines what is to be considered as "stable", but rather that there are millions of different cornercases of code generation that it is impossible to test them all every day. –  JesperE Nov 20 '12 at 11:48
1  
@JesperE - that's understandable, if the huge test suite does not define "stable" but is a gigantic sanity test. I would not expect the full suite (or even the medium suite) to fail very often. –  Joris Timmermans Nov 20 '12 at 12:37

For SVN, I don't know about such a thing like "pre-commit". I think it's likely to produce commits and rollbacks when the test fails. As doc-brown says, the only way there is to commit on a temporary branch and merge it with trunk later on.

Using a distributed one like git or mercurial, I think it would be possible. Using a "testing" repository and a "stable" repository. You push on the test rep, test it nightly, and if everything runs fine, you push from test to stable. Otherwise, you rollback the testing rep. I'm a bit unsure how the version history would look like when you push from testing to stable, but I think it's possible to exclude the broken rollbacked stuff when doing so. A bit experimenting first would be the safest.

An alternative would also be to test each person's local trunk nightly. Then, the people with passed tests are allowed to push it to the central server in the morning.

share|improve this answer

IMHO this has nothing to do with the VCS you are using. Using an "under test" branch may be solution, which can be realized with centralized or distributed VCS as well. But honestly, I think the best thing in your situation is trying to optimize the medium test suite (seems that it contains the most important tests) so that it runs much faster, so and you can use it for pre-commit-to-trunk tests, just like you do it now with your "small suite".

share|improve this answer
    
I'm mostly asking about methodology here - i.e., is there a common way to deal with such a situation. Let's assume, at least for the sake of this discussion, that the tests cannot be optimized beyond what they already are. –  Oak Nov 20 '12 at 10:04
    
@Oak: someone here (you?) voted my answer down - but sometimes the things you don't want to hear are the only things that will help. As you see in the discussion below your question, others suggested the same, so my suggestion does not seem to be so bad at all. –  Doc Brown Nov 20 '12 at 13:38
    
+1, this is the right answer. The OP's real question is "Help, I'm drowning in crap, what methodology can I use to help myself?" and the answer is that really methodology is not what you should be worrying about. –  MrFox Nov 20 '12 at 16:04

The failing medium tests: Is it true that most of the time the same tests fail?

If there is a failure are there always the same related tests that fail?

If true: May be you can selectively pick some medium tests that often fail (one test for every class of error) and execute them within the small set.

Are most of the tests integration-tests that use a real database? If so is it possible to replace them with a unittest that has a mocked-database?

share|improve this answer

You need to make your tests run faster, there is no other way to square this circle.

Consider the problem: you want to be sure that when you check out, you have working code. Sure, you can delay commits and do branching until before the release, but that will only delay the onset of the problem until integration. As in, will you have to run the week-long suite after every merge? Methodology is not the solution, the solution is purely technical.

Here is what I suggest:

1) Make the tests as atmomic as possible, and maximize environment reuse.

2) Get a test-suite farm to run them. If rather than 8 big modules you end up with 50, you can spin up a bunch of an Amazon EC2 spot instances and run the whole suite in parallel. I'm sure this will cost some money, but it will save huge amounts of developer time.

share|improve this answer

The key thing you are taking for granted in your question is that all commits must pass tests. While this is a nice rule to follow and it seems to make some sense, sometimes it's not practical. Your case is an example (although MadKeithV does make a point), and I can imagine keeping a VCS branch so pristine could be difficult if there isn't sufficient cooperation among devlopers.

In reality what you want is to somehow know which commits pass or fail. A "pre-commit branch" as you suggested would work, but that might require extra effort from developers when they make commits, which might be hard to sell.

A similar approach that could be easier is to leave the trunk for people to break as they please, and have a branch for commits that aren't broken. An automated script could go through commits as they are made to the trunk, run the tests on them and add them to the branch if they pass.

Or you could be absurdly simplistic and have a script that lists the passing commits in a text file (which may or may not itself be version controlled).

Or have a batch system that accepts requests for branches/revisions to test (from anywhere in the tree), and tests them and commits them to the trunk (or another branch) if they pass.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.