Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm studying an approach to understand better how the continuous integration workflow fits better in a software development company with the scrum method.

I'm thinking something like this:

enter image description here

Would that be a nice workflow?

share|improve this question

6 Answers 6

You're some of the way there, but I'd expand your diagram somewhat:

An expanded version of the CI workflow. It's fairly hard to explain in a short alt-tag.

Basically (if your version control will allow it, i.e. if you're on hg/git), you want each developer/dev pair to have their own "personal" branch, which contains a single user story they're working on. When they complete the feature, they need to push into a central branch, the "Release" branch. At this point, you want the dev to get a new branch, for the next thing they need to work on. The original feature branch should be left as-is, so any changes that need to be made to it can be made in isolation (this is not always applicable, but it's a good starting point). Before a dev returns to work on an old feature branch, you should pull in the latest release branch, to avoid weird merge issues.

At this point, we've got a possible release candidate in the form of the "Release" branch, and we're ready to run our CI process (on that branch, obviously you can do this on each developer branch, but this is fairly rare in larger dev teams at it clutters up the CI server). This might be a constant process (this is ideally the case, the CI should run whenever the "Release" branch is changed), or it might be nightly.

At this point, you'll want to run a build, and get a viable build artefact from the CI server (i.e. something you could feasibly deploy). You can skip this step if you're using a dynamic language! Once you're built, you're going to want to run your Unit Tests, as they are the foundation of all the automated tests in the system; they're likely to be quick (which is good, as the entire point of CI is to shorten the feedback loop between development and testing), and they're unlikely to need a deploy. If they pass, you're going to want to automatically deploy your application to a test server (if possible), and run any integration tests you have available. The integration tests may be automated UI tests, BDD tests, or standard integration tests using a Unit Testing framework (i.e. "unit" tests that require more state).

By this point, you should have a fairly comprehensive indication of whether the build is viable. The final step I'd normally setup with a "Release" branch is to have it automatically deploy the release candidate to a test server, so your QA department can do manual smoke tests (this is often done nightly instead of per-checkin so as to avoid messing up a test cycle). This just gives a quick human indication of whether the build is really suitable for a live release, as it's fairly easy to miss things if your test pack is less than comprehensive, and even with 100% test coverage it's easy to miss something that you can't (shouldn't) test automatically (such as a mis-aligned image, or a spelling mistake).

This is, really, a combination of Continuous Integration and Continuous Deployment, but given that the focus in Agile is on lean coding and automated testing as a first-class process, you want to aim to get as comprehensive an approach as possible.

The process I've outlined is an ideal-case scenario, there are many reasons why you might abandon parts of it (for example, developer branches are simply not feasible in SVN), but you want to aim for as much of it as possible.

As for how the Scrum sprint cycle fits into this, ideally you want to have your releases happen as often as possible, and not leave them until the end of the sprint, as getting quick feedback as to whether a feature (and build as a whole) is viable for a move to production is a key technique for shortening your feedback loop to your Product Owner.

share|improve this answer
1  
I agree with a lot of what you say, but I (personally) would try to encourage people together in the same branch whenever possible. –  William Payne May 25 '12 at 15:57
    
@WilliamPayne Realistically, if you're using a DVCS people are never in the same branch (unless they're literally working on the same machine), since you "branch" every time you clone. The equivalent "Branch" to what you'd have in a standard VCS is the merge branch you will have per-feature (so Dev B and Dev C could work on branches of Branch C then check into Branch C as they are done, if they were both working on the C feature). The benefits of having more branches is less time spent fixing merge issues, since merging in HG/Git is almost completely painless, but a push/pull race is not. –  Ed Woodcock May 25 '12 at 16:30
1  
At the risk of this turning into a flame war, I believe that problems due to merge conflicts are frequently exaggerated (although I concede that they do exist). Furthermore, I believe that the move to DVCS tools (though warranted by other considerations) should not be justified purely on the basis of avoiding merge conflicts - In anything other than an open/community development effort, these are symptomatic of poor communication and coordination in the development team. Switching to a DVCS only masks a deeper and more significant problem with team communication. –  William Payne May 25 '12 at 17:40
1  
I still strongly believe in the necessity of a "daily" (trunk) repo that developers commit to as they work. Integrating your work continuously is what Continuous Integration is all about, after all. For me, branching is something that is done as a last resort - not because of the danger of merge conflicts, but because developers loose track of what the other is doing, and start working in silos. It is about teamwork, communication and human factors, not the limitations of the tools that we use. To do continuous integration, you need continuous communication, which branching fails to support. –  William Payne May 25 '12 at 17:50
1  
@WilliamPayne I'm not going to debate the relative merits of DVCS vs. VCS with you (at least not via the comments), there are significant benefits to using a DVCS if you get the right workflow, whereas with a more traditional workflow a VCS will do a sterling job (in my experience). I will tell you, however, that since switching to a DVCS my team has experienced significantly quicker development turnaround and more robust deployments. The end-of-project merge hell that many people still experience to this day has been almost completely removed from our development process. –  Ed Woodcock May 26 '12 at 10:21

Conceptually yes. A diagram isn't capturing a lot of important points though like:

  1. unit testing
  2. incremental commits
  3. staging gets deployed to often, production usually doesn't.
share|improve this answer

You may want to draw a wider system for the diagram. I would consider adding the following elements:

Show your inputs to the system, which are fed to the developers. Call them requirements, bug fixes, stories, or whatever. But currently your workflow assumes the viewer knows how those inputs are inserted.

Show the control points along the workflow. Who / what decides when a change is allowed into trunk / main / release-branch / etc...? What codetrees / projects are build on the CIS? Is there a checkpoint to see if the build was broken? Who releases from CIS to staging / production?

Related to the control points is identifying what your branching methodology is and how it fits into this workflow.

Is there a Test team? When are they involved or notified? Is there automated testing being performed on the CIS? How are breakages fed back into the system?

Consider how you would map this workflow to a traditional flowchart with decision points and inputs. Have you captured all of the high-level touch points that are needed to adequately describe your workflow?

Your original question is attempting to make a comparison, I think, but I'm not certain on which aspect(s) you're trying to compare. Continuous Integration has decision points just like other SDLC models, but they may be at different points in the process.

share|improve this answer

I use the term "Development Automation" to encompass all automated build, documentation generation, test, performance measurement and deployment activities.

A "development automation server" therefore has a similar, but somewhat broader remit than a continuous integration server.

I prefer to use development automation scripts driven by post-commit hooks that allow both private branches and the central development trunk to be automated, without requiring additional configuration on the CI server. (This precludes the use of most of the off-the-shelf CI server GUIs that I am aware of).

The post-commit script determines which automation activities to run based on the content of the branch itself; either by reading a post-commit configuration file in a fixed location in the branch, or by detecting a particular word (I use /auto/) as a component of the path to the branch in the repository (with Svn)).

(This is easier to set up with Svn than Hg).

This approach enables the development team to be more flexible about how they organise their workflow, allowing CI to support development on branches with minimal (close to zero) administrative overhead.

share|improve this answer

There is a good series of posts on continuous integration up on asp.net that you may find useful, it covers quite a bit of ground and workflows that fit with what it looks like you're after doing.

Your diagram doesn't make mention of the work done by the CI server (unit testing, code coverage and other metrics, integration testing or nightly builds), but I assume that's all covered in the "Continuous Integration server" stage. I'm not clear on why the CI box would be pushing back to the central repository though? Obviously it needs to get the code but why would it ever need to send it back?

CI is one of those practices recommended by various disciplines, it isn't unique to scrum (or XP) but in fact I would say it's benefits are available to any flow even the non-agile such as waterfall (maybe wet-agile?). For me the key benefits are the tight feedback loop, you know pretty quickly whether the code you just committed works with the rest of the code base. If you're working in sprints and having your daily stand-ups then being able to refer to the status, or metrics from last nights build in the CI server is definitely a plus and helps focus people. If your product owner can see the status of the build - a big monitor in a shared area showing the status of your build projects - then you've really tightened up that feedback loop. If your development team are committing frequently (more than once a day and ideally more than once an hour) then the chances that you'll run into an integration issue that takes a long time to resolve are reduced, but if they do it's clear to all and you can take whatever measures you need, everyone stopping to deal with the broken build for example. In practice you'll probably not hit many failed builds that take more than a few minutes to figure out if you're integrating often.

Depending on your resources / network you may want to consider adding different end servers. We have a CI build that's triggered by a commit to the repo and assuming that builds and passes all it's tests then it's deployed to the the development server so devs can make sure that it plays nicely (you could include selenium or other UI testing here?). Not every commit is a stable build though, so to trigger a build to the staging server we have to tag the revision (we use mercurial) we want to be built and deployed, again this is all automated and triggered simply by committing with a particular tag. To go to production is a manual process; you can leave it as simple as forcing a build the trick is knowing which revision / build you want to use, but if you were to tag the revision appropriately then the CI server can checkout the correct version and do whatever is needed. You could be using MS Deploy to sync the changes to the production server(s), or to package it up and put the zip somewhere ready for an admin to deploy manually... it depends on how comfortable you are with that.

As well as going up a version you should also consider how you might deal with failure and go down a version. Hopefully it wont happen but there could be some change made to your servers that means that what works on UAT doesn't work on production, so you release your approved version and it fails... you can always take the approach that you identify the bug, add some code, commit, test, deploy to production to fix it... or you could wrap some further tests around your automated release to production and if it fails those then it automatically rolls back.

CruiseControl.Net uses xml to configure the builds, TeamCity uses wizards, if you're aiming to avoid specialists in your team then the complexity of the xml configs can be something else to bear in mind.

share|improve this answer
    
+1 for the comment about feedback loops. :-) –  William Payne May 25 '12 at 18:11

First, a caveat: Scrum is a pretty rigorous methodology. I have worked for a couple of organisations that have tried to use Scrum, or Scrum-like approaches but neither of them really came close to using the full discipline in its entirety. From my experiences I am an Agile-enthusiast, but a (reluctant) Scrum-skeptic.

As I understand it, Scrum & other Agile methods have two main objectives:

  • The first is to provide an explicit mechanism for managing risks and the ongoing discovery of risks.
  • The second is to provide a structured mechanism for stakeholder communication, requirements discovery and requirements management.

The first (risk management) objective is achieved through iterative development; making mistakes and learning lessons quickly, allowing the team to build the understanding and the intellectual capability to reduce risk and move towards a reduced-risk solution with a low-risk "austere" solution already in the bag.

Development Automation, including continuous integration, is the most critical factor in the success of this approach. Risk discovery & Lesson-learning must be fast, friction-free, and free of confounding social factors. (People learn MUCH faster when it is a machine that tells them they are wrong rather than another human - egos only get in the way of learning).

As you can probably tell - I am also a fan of test-driven development. :-)

The second objective has less to do with development automation, and more to do with Human Factors. It is harder to implement because it requires buy-in from the front end of the business, who are unlikely to see the need for the formality.

Development Automation can have a role here, in that automatically generated documentation and progress reports can be used to keep stakeholders outside of the development team continuously updated with progress, and information radiators showing build status and passing/failing test suites can be used to communicate progress on feature development, helping (hopefully) support the adoption of the Scrum communications process.

So, in summary:

The diagram that you used to illustrate your question only captures part of the process. If you wanted to study agile/scrum and CI, I would argue that it is important to consider the broader social and human-factors aspects of the process.

I must end by banging the same drum that I always do. If you are trying to implement an agile process in a real-world project, the best predictor of your chance of success is the level of automation that has been deployed; it reduces friction, increases velocity & paves the road to success.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.