Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Continuous Delivery sounds good, but my years of software development experience suggest that in practice it can't work.

(Edit: To make it clear, I always have lots of tests running automatically. My question is about how to get the confidence to deliver on each checkin, which I understand is the full form of CD. The alternative is not year-long cycles. It is iterations every week (which some might consider still CD if done correctly), two weeks, or month; including an old-fashioned QA at the end of each one, supplementing the automated tests.)

  • Full test coverage is impossible. You have to put in lots of time -- and time is money -- for every little thing. This is valuable, but the time could be spent contributing to quality in other ways.
  • Some things are hard to test automatically. E.g. GUI. Even Selenium won't tell you if your GUI is wonky. Database access is hard to test without bulky fixtures, and even that won't cover weird corner cases in your data storage. Likewise security and many other things. Only business-layer code is effectively unit-testable.
  • Even in the business layer, most code out there is not simple functions whose arguments and return values can be easily isolated for test purposes. You can spend lots of time building mock objects, which might not correspond to the real implementations.
  • Integration/functional tests supplement unit tests, but these take a lot of time to run because they usually involve reinitializing the entire system on each test. (If you don't reinitialize, the test environment is inconsistent.)
  • Refactoring or any other changes break lots of tests. You spend lots of time fixing them. If it's a matter of validating meaningful spec changes, that's fine, but often tests break because of meaningless low-level implementation details, not stuff that really provides important information. Often the tweaking is focused on reworking the internals of the test, not on truly checking the functionality that is being tested.
  • Field reports on bugs cannot easily be matched with the precise micro-version of the code.
share|improve this question
    
It works very fine for Etsy slideshare.net/OReillyOSCON/… ! –  YoTsumi Dec 9 '11 at 12:09
4  
Continuous Delivery does not require testing (but it helps). It refers to the fact that the things you build on a regular basis COULD be shipped to the customer if needed. –  user1249 Dec 9 '11 at 14:22
    
It's interesting that your objections to continuous delivery focus overwhelmingly on testing as an element of CD. However, that's only one part of the puzzle: You need competent internal tooling, developers committed to rigorous quality inspections, a breadth-over-depth prioritization approach in your automated tests, not to mention strong organizational support. It can be done, but it takes a lot of people to commit to the cause. –  Stephen Gross Dec 9 '11 at 16:16
1  
@Stephen Yes, I am focusing on testing, because I am in agreement on all the other aspects. I am in favor of testing, too. I just don't see how you can have enough confidence to deploy every checkin. –  Joshua Fox Dec 11 '11 at 8:10
    
@Thorbjørn Ravn Andersen . Certainly, CD seems to require testing. How can you have the confidence to automatically ship every checkin without it. –  Joshua Fox Dec 11 '11 at 8:10
show 5 more comments

6 Answers

up vote 14 down vote accepted

my years of software development experience suggest that in practice it can't work.

Have you tried it? Dave and I wrote the book based on many collective years of experience, both of ourselves and of other senior people in ThoughtWorks, actually doing the things we discuss. Nothing in the book is speculative. Everything we discuss has been tried and tested even on large, distributed projects. But we don't suggest you take it on faith. Of course you should try it yourself, and please write up what you find works and what doesn't, including the relevant context, so that others can learn from your experiences.

Continuous Delivery has a big focus on automated testing. We spend about 1/3 of the book talking about it. We do this because the alternative - manual testing - is expensive and error-prone, and actually not a great way to build high quality software (as Deming said, "Cease dependence on mass inspection to achieve quality. Improve the process and build quality into the product in the first place")

Full test coverage is impossible. You have to put in lots of time -- and time is money -- for every little thing. This is valuable, but the time could be spent contributing to quality in other ways.

Of course full test coverage is impossible, but what's the alternative: zero test coverage? There is a trade-off. Somewhere in between is the correct answer for your project. We find that in general you should expect to spend about 50% of your time creating or maintaining automated tests. That might sound expensive until you consider the cost of comprehensive manual testing, and of fixing the bugs that get out to users.

Some things are hard to test automatically. E.g. GUI. Even Selenium won't tell you if your GUI is wonky.

Of course. Check out Brian Marick's test quadrant. You still need to perform exploratory testing and usability testing manually. But that's what you should be using your expensive and valuable human beings for - not regression testing. The key is that you need to put a deployment pipeline in place so that you only bother running expensive manual validations against builds that have passed a comprehensive suite of automated tests. Thus you both reduce the amount of money you spend on manual testing, and the number of bugs that ever make it to manual test or production (by which time they are very expensive to fix). Automated testing done right is much cheaper over the lifecycle of the product, but of course it's a capital expenditure that amortizes itself over time.

Database access is hard to test without bulky fixtures, and even that won't cover weird corner cases in your data storage. Likewise security and many other things. Only business-layer code is effectively unit-testable.

Database access is tested implicitly by your end-to-end scenario based functional acceptance tests. Security will require a combination of automated and manual testing - automated penetration testing and static analysis to find (e.g.) buffer overruns.

Even in the business layer, most code out there is not simple functions whose arguments and return values can be easily isolated for test purposes. You can spend lots of time building mock objects, which might not correspond to the real implementations.

Of course automated tests are expensive if you build your software and your tests badly. I highly recommend checking out the book "growing object-oriented software, guided by tests" to understand how to do it right so that your tests and code are maintainable over time.

Integration/functional tests supplement unit tests, but these take a lot of time to run because they usually involve reinitializing the entire system on each test. (If you don't reinitialize, the test environment is inconsistent.)

One of the products I used to work on has a suite of 3,500 end-to-end acceptance tests that takes 18h to run. We run it in parallel on a grid of 70 boxes and get feedback in 45m. Still longer than ideal really, which is why we run it as the second stage in the pipeline after the unit tests have run in a few minutes so we don't waste our resources on a build that we don't have some basic level of confidence in.

Refactoring or any other changes break lots of tests. You spend lots of time fixing them. If it's a matter of validating meaningful spec changes, that's fine, but often tests break because of meaningless low-level implementation details, not stuff that really provides important information. Often the tweaking is focused on reworking the internals of the test, not on truly checking the functionality that is being tested.

If your code and tests are well encapsulated and loosely coupled, refactoring will not break lots of tests. We describe in our book how to do the same thing for functional tests too. If your acceptance tests break, that's a sign that you're missing one or more unit tests, so part of CD involves constantly improving your test coverage to try and find bugs earlier in the delivery process where the tests are more fine-grained and the bugs are cheaper to fix.

Field reports on bugs cannot easily be matched with the precise micro-version of the code.

If you're testing and releasing more frequently (part of the point of CD) then it is relatively straightforward to identify the change that caused the bug. The whole point of CD is to optimize the feedback cycle so you can identify bugs as soon as possible after they are checked in to version control - and indeed, preferably before they're checked in (which is why we run the build and unit tests before check-in).

share|improve this answer
    
Thanks for your answer. Yes, I believe in testing. My projects have had good code coverage from automated tests run with the daily build. I'm just saying that you need some sort of iteration before you release. "You still need to perform exploratory testing ... manually." I don't understand. A full CD system deploys on every checkin. How can you do that if you include manual testing? –  Joshua Fox Dec 10 '11 at 16:24
1  
I like to distinguish between continuous delivery and continuous deployment. Here's the difference. Continuous delivery means you keep the system production-ready at all times and can release on demand at the push of a button. Release is a business decision. Continuous deployment is a limiting case where you release every good build (note not every check-in - some check-ins don't result in a releasable build). In both cases you can include manual validations: the key is the concept of the deployment pipeline. –  Jez Humble Dec 12 '11 at 19:15
add comment

First, CD takes one big mental adjustment -- you have to admit that sometimes things will go out broken no matter what you do. At the end of the day, you cannot prove a negative.

Once you get past this, you realize that you need tools and procedures to a) catch these errors very quickly and b) either roll back or deploy the update very efficiently. Moreover, if you are truly drinking the CD cocktail, you are really delivering lots of small, pointed changes that are easy to roll back and should not be able to introduce major, application-wide bugs. Even the guys really practicing CD are handling major changeovers in a more traditional manner.

share|improve this answer
    
"small...changes ... should not be able to introduce application-wide bugs". Even in well-factored code, this can happen. For example, you add a div which pushes another div out of view in one particular browser. For example, a null value appears in an unexpected corner case, throwing an exception and preventing the GUI from rendering at all. Yes, you should test everything possible, as I do, but inevitably, bugs happen, and small bugs can disrupt the whole app. –  Joshua Fox Dec 11 '11 at 10:54
    
But they are still easy to find and fix which is the greater point of emphasis. –  Wyatt Barnett Dec 11 '11 at 15:44
add comment

Full test coverage is impossible. You have to put in lots of time -- and time is money -- for every little thing. This is valuable, but the time could be spent contributing to quality in other ways.

You don't need 100% coverage, you need enough to be confident in your system, that changes to one place won't break things you've previously proven work.

Some things are hard to test automatically. E.g. GUI. Even Selenium won't
tell you if your GUI is wonky. Database access is hard to test without bulky fixtures, and even that won't cover weird corner cases in your data storage.

Database access is trivial to write however. You shouldn't need many tests on your data layer as it's just getting and setting data. The most important thing is ensuring that when it fails, it rollsback and logs the issue so you can fix it.

Likewise security and many other things. Only business-layer code is effectively unit-testable. Even in the business layer, most code out there is not simple functions whose arguments and return values can be easily isolated for test purposes.

Then many of your functions/classes are too large. They should be written to be testable.

You can spend lots of time building mock objects, which might not correspond to the real implementations.

The I/O of the mock object should match what is expected however. What happens inside it unimportant.

Integration/functional tests supplement unit tests, but these take a lot of time to run because they usually involve reinitializing the entire system on each test. (If you don't reinitialize, the test environment is inconsistent.)

These shouldn't be run all the time. Just as needed.

Refactoring or any other changes break lots of tests. You spend lots of time fixing them. If it's a matter of validating meaningful spec changes, that's fine, but often tests break because of meaningless low-level implementation details, not stuff that really provides important information. Often the tweaking is focused on reworking the internals of the test, not on truly checking the functionality that is being tested.

Then your code is too tightly coupled, and your tests are poorly written.

Field reports on bugs cannot easily be matched with the precise micro-version of the code.

Not sure what you're getting at here? If there's a bug, write a test to show it's existence, then fix it.

share|improve this answer
    
"The I/O of the mock object should match what is expected"." Building MOs that fully implement an interface-spec is time-consuming. Worse, one must update them continually, so that one is effectively writing all code twice (once for production and once for MOs). –  Joshua Fox Dec 10 '11 at 16:26
add comment

Works fine for us, but our customers are mostly internal. Multiple builds done during the day, broken builds are not tolerated, web start mechanism used so users get the latest version every launch. One thing is that CD makes a lot of problems go away. Yes, you have compatibility concerns all the time, however, since you're only deploying small changes each time, the concerns are really easy to get a handle on. Stress level of CD is MUCH lower than when we made big updates and had to hope that everything was just right since there'd be so much code to review in the case of a breakage.

share|improve this answer
add comment

Every system has risks, and every risk has potential costs. If the cost of some tiny risk, of the kind that may take months or years to find in extensive testing and QA, is high enough (the software in your heart pacemaker), you don't ship without an extensive period of testing of a frozen release. If the cost of failure is small enough, maybe you ship continuously with zero testing (your cat's Facebook page).

share|improve this answer
    
Yes. For most production applications, the risk is somewhere in between. And it seems to me that the risk is such that we would not want to deploy on each checkin, even with automated testing. It seems that a round of QA is always needed. But roughly weekly releases seem feasible to me. –  Joshua Fox Dec 11 '11 at 21:49
add comment

To be honest, ALL software is in continuous delivery! The most important thing is to get it out the door! Get your users using it and prioritize the feature requests and bug squashing after that.

"Real artists ship"
-- Steve Jobs.

share|improve this answer
    
the alternative to CD is not year-long cycles. It is iterations every week, two weeks, or month. –  Joshua Fox Dec 10 '11 at 20:35
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.