Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

When having a continuous integration executing the tests at each commit, a common best practice is to have all the tests passing at all times (aka "do not break the build").

I find some problems with that:

For example one cannot help an open source project by creating tests corresponding to tickets. I know if I propose a Pull Request to an open source project containing failing test, the build will be marked as failed and the project will not want that merged into its repository because it would "break the build".

And I don't believe it is a bad thing to have failing tests in your repo, it's like having open issues in your tracker. These are just things waiting to be fixed.

The same goes in a company. If you work with TDD, you can't write tests, commit and then write the logic code that fulfills the test. That means if I have written 4-5 tests on my laptop, I can't commit them before going on holidays. Nobody can take back my work. I can't even "share" them with a colleague except by sending them by email for example. It also prevents working with one person writing the tests, the other one writing the model.

All that to say, am I misusing/misunderstanding the build process/continuous integration? It seems to me that "passing"/"not passing" is a too narrow indicator.

Is there a way to make continuous integration and TDD compatible?

Maybe there is a standard solution/practice to distinguish "new tests" (that can fail) and "regression tests" (that should not fail because they used to work)?

share|improve this question
1  
Have an indicator that shows if the number of failed tests went up (red) or down (green) in the last hour (or so). –  Joachim Sauer Feb 8 '13 at 13:53
2  
I'm no TDD/ALM specialist (hence the comment, rather than an answer) but I think your problem can be solved with private branches/feature branches. Are you working on Feature A? Branch it off, work on the branch (with colleagues), and once you're done - merge it into the continuously integrated trunk. –  Avner Shahar-Kashtan Feb 8 '13 at 13:53
    
@JoachimSauer Yes but is such metric standardized/used in any major project? I'm trying to understand why most of the projects (and CI tools) work that way. –  Matthieu Napoli Feb 8 '13 at 14:02
    
I think the correct criterion for "tests that can fail" is not "new tests", but rather "tests for known open issues". I can see how those tests are useful to have - I can also how those tests are NOT useful in the CI build because they pollute the meaning of the test pass/fail there (you only want to run tests for which someone has actually spent time to make them pass). –  Joris Timmermans Feb 8 '13 at 15:20
    
@MadKeithV Exactly –  Matthieu Napoli Feb 8 '13 at 15:39

8 Answers 8

I see where you're getting at, but these types of problems are typically solved in other ways. There is a good reason why this is standard protocol. If someone submits code that doesn't compile, everyone updating his/her code will have a program that doesn't compile. That includes programmers who are currently working on something completely different and somehow find themselves in a situation in which they need to wait before they can compile and test what they're working on.

Standard protocol is that you can commit changes even for complete or even incomplete work, so long as it compiles so that programmers can update their code every day if necessary.

However, I still see what you're getting at. Sometimes you want to commit in order to simply save your code. For this, most source repositories support branching. This allows you to create a private branch, work on it without disturbing others, then merge into the trunk when the work is completed. This allows you to commit when you want without any of the backlash associated with causing the build to break.

If that's not suitable, GIT allows you to commit (pushing) to repositories on your local machine, but conceivably the repository could be anywhere. You could create a repository for potentially partial/incomplete work and another repository for finished work, and on that repository you can add a nightly build.

Again, I can't stress the importance enough. Do not commit broken code to trunk ever! Your contributions cannot impact the work of other programmers.

Edit

I see that you intended broken tests, but in my humble opinion, there is little difference. The whole point of a test is to determine if a particular aspect of a program passes or fails. If it always fails and you do nothing, then the test, in the traditional usage of unit testing, serves nothing. If you use it to perform some other metric which doesn't necessarily entail a "failed" commit if one such test fails, then I would strongly recommend you find another way to do the same thing.

Otherwise you risk that the test is never taken into consideration or if it causes your build to fail, that your fellow programmers ignore failed builds. It's more important that programmers realize when they've broken a build than to perform a test which offers no real insight and may only result in bad practices.

share|improve this answer
1  
Indeed you make a point with topic branches. But I'm never talking about commiting broken code, just failing tests. For example I could help an open source project by creating tests for incoming tickets even if I don't know how to fix them. That saves a some time for the maintainers. –  Matthieu Napoli Feb 8 '13 at 14:05
    
If I'm working on the same project as you, and you upload a failing test, I now have a build that has a failing test. I might end up deleting your test, since that functionality isn't yet implemented, or decide to implement the feature and end up stomping on your code and wasting time. If there were a culture of doing this, that sort of response could be avoided, but then everyone would be doing it, and so even when all your tests pass, not all of mine do. In that case, the build would always have failing tests. I don't see an upside. –  Michael Shaw Feb 8 '13 at 14:25
    
the build would always have failing tests precisely! But is that such a bad thing? Our only metric is "the build is broken or not", but your code could be full of known bugs, so that doesn't really mean anything except there is no regression. In a perfect world, every tracker issue would have a test (reproducing is easier than fixing). So the upside would be to see that 35 tests/70% of all the tests are passing, that Branch-A improves it to 40 tests (80%) with no regression, and that Branch-B has regressions. Today you could only say Master and Branch-A are OK and Branch-B is broken. –  Matthieu Napoli Feb 8 '13 at 14:34
    
@Matthieu I see what you're getting at. Sounds like you need a special category or something that says "hey, if this test is failing, it's ok. we know about it. But we still want it ran and if it passes then that's even better and we should remove the special category because we now care if it breaks" –  Earlz Feb 8 '13 at 16:20
    
@Earlz Exactly! What I'm wondering is "is it done by someone, somewhere? And are there tools that support that (CI and unit testing libraries?" Because if I just categorized those tests with classic CI and unit-tests tools, the build will always fail anyway and I won't see a difference between which tests failed, and so it won't be useful :/ –  Matthieu Napoli Feb 11 '13 at 8:30

Given a master branch with failing tests, how can you be sure - without comparing that list against previous builds - that you haven't introduced bugs?

Simply tracking the number of failing tests is insufficient: you might fix one test, and break another. And if you're on holiday, it won't be clear to others looking at the failing build.

Keep your master branch clean and green at all times. Work in a branch. Keep the branch under CI, in a separate job, and have failing tests to your heart's content. Just don't break master.

Have the branch's reviewer only merge your branch if it passes all tests. (More strongly: have the reviewer only able to merge your branch if the result of merging the branch into master passes all tests!)

share|improve this answer
2  
Simply tracking the number of failing tests is insufficient that's not the only possible metric. For example: Branch-A improves it to 40 tests (80% passing) with no regression. No regression means previously passing tests always pass. In short, a test would be allowed to fail as long as it has never passed. It seems to me that we are missing on good things by constraining to forbid failing tests in main branches. (of course it would require tools working differently: unit tests, CI, ...) –  Matthieu Napoli Feb 8 '13 at 15:55
    
I still stand by my point: master should always be green, because it's clear and unambiguous. By all means have marker tests failing... in a feature branch. CI can keep reminding people of outstanding bugs. –  Frank Shearar Feb 8 '13 at 16:31
    
I think what Matthieu is proposing is a slightly different definition of "green", not deviating from master always being green. It's not obvious to me that it doesn't make sense - it would need some not entirely trivial tooling for tracking, of course. (Need to roll back a change that made that test pass? Tough luck if that means the state is suddenly red ...) –  Christopher Creutzig Mar 9 '13 at 21:47
    
NUnit has the concept of an ignored test. That might be an alternative: they don't get run so they don't fail, but they're still reported as being ignored. –  Frank Shearar Mar 9 '13 at 22:07

There are ways to solve your problems without throwing away well understood and accepted practices about continuous integration.

I will start with the problem of committing a 'broken test' that corresponds to a ticket. One solution is to create one or more breaking tests exposing the problem, then actually fix the problem, so they can be merged back to the main code line together. Second solution is to have the broken tests, but use some type of ignore flag so they do not actually run and break the build. Possibly add a comment or a special annotation that makes it very obvious that this is a broken test for Ticket#N. Also attach a note to the ticket itself that refers to the created tests that are waiting to be unignored and ran. This would help a person fixing the ticket, but also would not be a red-flag for someone who comes across the test.

And onto your next problem with TDD. TDD is about writing a small test and then writing a small chunk of code to make that test pass. Then keep iterating until you have a small functional module. I feel that if you write 4-5 tests, then you go on vacation, you might be doing it wrong. You can pair program with someone in a way that one of you write the test, the other the corresponding code. You should however not use the main code line repository to share this code between you two before a completed module is ready to be committed. As others suggested, a shared branch would solve your problems there.

Trying to break the continuous integration mantra can lead down to unexpected and scary paths. For example, what would code coverage mean in this type of environment? How would developers not feel that the system has many "Broken Windows"? How would one make a change, run the test and know whether they are actually breaking anything new, or its just the old stuff?

share|improve this answer
    
You don't need any tooling to share with the person you are pair programming with - just hand over the keyboard. If you're using different computers, well, there's nothing wrong with that, it's just not "pair programming". –  Christopher Creutzig Mar 9 '13 at 21:50

I think your fundamental problem is that you are including test RESULTS as part of the build. While obviously some people agree with you, others don't. Breaking the build occurs when it doesn't build. Not when it doesn't build without errors.

Consider a major project like Windows or Linux, or even something like Firefox -- do you think they ship bug free? Of course not. Now these projects aren't doing TDD, but that's really irrelevant -- TDD doesn't change two fundemental facts: bugs exist, and it takes time to fix them. Time which a project (open source or not) just can't afford to waste on low priority bugs. KDE recently had a bug that was over a decade old fixed. When was the last time you heard someone say "I'm glad we waited a decade to ship our project"?

TDD, in a way, probably makes it EASIER to ship with bugs -- because you have a better understanding of what the flaw is. If you can precisely define what causes the bug, you have an excellent basis for weighing the cost of fixing it.

My recommendation is to find a project that doesn't mind some red amoung the green.

share|improve this answer
 > a common best practice is to have all the tests passing (green) at all times.

I prefer to have all tests not failing (not red).

With this slightly different definition you can also define tests that are

  • not implemented yet (grey in nunit if there is a NotImplementedException)
  • known to be failing="needs a to do" by marking/annotating the test as ignored (yellow)

I you check these into the repository your continous build is not broken and therefore valid.

share|improve this answer

You could consider two different CI build "concepts".

  1. Regular CI builds. The regular CI build should compile and run only those tests for which code has been written to make them pass, so that CI reports of test pass/fails is a clear, unambiguous indicator of regression against the previously accepted state of the code.
  2. A "future" CI build. This build compiles and runs only those tests for which no code has been written specifically to make them pass. There can be several reasons to have such a test:

    • Tests can added for specific failure cases from the issue tracker, for which no fix has been attempted yet. It's clearly useful to already have a codified, running test for an issue even without a fix.

    • Tests added for new required functionality that has not been implemented yet.

    • Often it's easier to know how to test a failure or feature than to know how to implement the feature or fix, and separating the two steps by committing the test to source control can be useful to ensure no information is lost.

As in "standard" CI, in the regular development regime the team would only be looking at the daily build results of the regular build.

The team can also keep an eye on the evolution of test cases from the "future" CI build - specifically to see if any changes made for the regular CI actually fix issues from the "future" build, which can be an indication of an important underlying issue or design improvement.

Finally, if a team member has extra time on their hands they could take a look at fixing one of the issues from "future", and if they succeed migrate it to "regular" (while updating the issue tracker status).

share|improve this answer

And I don't believe it is a bad thing to have failing tests in your repo, it's like having open issues in your tracker. These are just things waiting to be fixed.

The problem isn't failing tests, it's a simple, context less indicator of regression. Aka: as a developer, can I check a single indicator and know if I introduced a regression or breaking code.

The moment you introduce the concept of 'soft' failures (it's ok, we're working on it/not implemented yet/we're waiting for the new version/it's going to pass again once this other build is fixed), you require everyone that might run or look at the test to know the expected state. Which in a large enough team is going to change by the hour: your indicator becomes meaningless. In a smaller context (team private integration test for instance), then I think it's like technical debt and is ok - just needs to be managed.

The way most tool address that is that 'green/passing' reflects what the expected result is - not that the code is working:

  • Fitness has the concept of expected failure.
  • JUnit has @Ignored / @Test(expect=)
  • Cucumber has the 'not yet implemented' (or whatever it's called) status

Those come with their own issues (how do you distinguish between 'yes, we know it's broken, working on it' and 'the correct behavior is an exception') - but they help.

share|improve this answer

I use skipped tests.

In the particular unit testing framework I use, I can raise a SkipTest exception. The test isn't actually run, and its failure won't break the build. However, I can see the number of skipped tests and see if there is work to be done in that area.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.