Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We currently are using CCNet as our continous integration server. Most projects check for changes every 30 seconds (the default) and if needed perform a build (unit tests, stylecop, fxcop, etc).

We've gotten quite a few projects now, and the server spends most of its time near 100% cpu utilization. This has alarmed some of the development team, even though the server is responsive and builds are still about the same length of time they've always been.

Its been suggested that we lower the check interval to about five minutes. To me that seems too long, and we risk people committing code and then going home for the weekend and now there's a broken build possibly holding up others. In response, the suggestion is that if someone needs to know the results they can force the build. But that seems to defeat the purpose of CI, as I thought it was supposed to be automated.

My proposed solution is just to get another build server and split the builds amongst the servers.

Am I thinking about this the wrong way, or is there a point where if integration isn't often enough you're not really doing CI anymore?

share|improve this question
    
How often do people check code into the integration branch? Also, are developers running any kind of build/test cycle prior to merging into the branch that's under CI? –  Thomas Owens Oct 6 '12 at 13:42
    
Fairly often, we normally all work in trunk. Feature branches are rarely used, pretty much only when the incremental changes being committed would certainly cause things to break. Most changes don't break anything though. –  Andy Oct 7 '12 at 16:24
    
Well, that's just a terrible idea. I'd search around for questions about branching strategies - I know others have said why working in the trunk isn't a good idea, and I'm pretty sure I've mentioned it in some of my answers. –  Thomas Owens Oct 7 '12 at 16:26
    
trunk checkins are ok.. if you understand what you're doing and your workload allows for it, but in general you should be using branches, or even a branch. –  gbjbaanb Oct 8 '12 at 21:27
    
@ThomasOwens That's your opinion, and not relevent to the question. FYI, there are a lot of valid strategies, and each team needs to find what works best with them. We've already experience some problems blogged about here which is why we went to dev in trunk (but stable): codinghorror.com/blog/2007/10/… –  Andy Oct 9 '12 at 12:57
show 2 more comments

4 Answers

up vote 4 down vote accepted

The CPU is supposed to be near 100% utilization. Otherwise you're just wasting CPU. Same with memory.

Continuous means as frequently as you can manage. If your server is maxed out, then crank out the builds as fast as it will go. It might sometimes get busy enough to drag your response time down, but if you lower the check interval, your response time will always be dragged down. It makes no sense to try to "conserve" idle time on your server. If your actual response time frequently drops to unacceptable levels, then you add additional servers.

share|improve this answer
    
I tend to agree with this as well; we're not noticing any performance problems, this issue was raised when someone logged on to fix a busted build and looked at task manager. Even if we were, I'd rather just add a second server to the mix. –  Andy Oct 7 '12 at 16:29
    
CPU at 100% means you're just wasting electricity. And not only is electricity expensive (plus cooling for those hot little chips) but it also is killing the planet with co2 emissions, dude. –  gbjbaanb Oct 8 '12 at 21:29
    
Accepted as this is the answer I feel best fits within our evironment / constraints. –  Andy Oct 9 '12 at 13:47
1  
@gbjbaanb are you saying that it's more power efficient to run two hosts at 50% CPU, than it is to run one host at 100% CPU? Interesting if true. Would love to see some data to back up that claim. –  Noah Sussman Jan 3 '13 at 22:05
add comment

You should not be checking for changes - let the source control system 'push' checkin notifications to the build system instead (you may need to get a new SCM or build system, I seriously recommend Jenkins)

So, obviously this would stop a lot of problems on the server.

The next problem is how much checking in goes on? If people are committing every 5 minutes, it isn't going to help much. Then the last question is - how long do these builds take? If you're spending 20 minutes building a checked-in component and running tests and analysis and creating installer packages, and it depends on others, are those built automatically too, even though they haven't changed?

Monitor the build system to see what it is doing. I've found that it is easy to set up a build server that does far too much building that is necessary. That woudl be the first step I'd take.

share|improve this answer
    
I agree, build on each commit / push instead of checking periodically. We use Bamboo from Attlasian for this. The tool integrates nicely with Mercurial (what we use) and of course it knows about all the other versioning tools. –  Patkos Csaba Oct 6 '12 at 13:56
    
Migrating to a new CI or SCM isn't really practical. There are frequent commits all day, the builds are relatively fast. I belive the slowest is five minutes. Most of the CPU usage is from CCNet firing off the SVN command line client to see if there have been changes; obviously an actual build process will use more resources. –  Andy Oct 7 '12 at 16:28
    
If most of the CPU is from CCNet doing SVN commands just to check for changes, AND someone's decided that 'something must be done', then I'd say find a way to trigger CCNet differently. Alternately, you could see if there is any way to make the svn commands complete faster (network? disks? SVN server upgrades?) –  Michael Kohne Oct 7 '12 at 16:54
    
the polling is your problem - CCNet runs the SVN command every 5 minutes... well there's your problem. Make SVN push a "check now" notification to CCNet (if it can accept such a thing) from a post-commit hook which will reduce the load dramatically. See: jenkins-ci.org/node/212 for a little discussion. –  gbjbaanb Oct 8 '12 at 21:25
    
@gbjbaanb In order for SVN to push changes, it would require that we allow all developers change the post commit hooks on the server for each and every item under CI. Its not practical to allow that level of access, and indeed we had to cut back access. Any developer can (and should) be able to setup a CI build. –  Andy Oct 9 '12 at 13:00
show 1 more comment

In my team, our CI configuration enables us to trigger a job each time changes are detected in the SCM; this is what continuous means to us.

If you have the resources, consider implementing a distributed CI system to offload the workload to multiple servers and keep resource usage to a minimum on the master.

We have multiple jobs for doing specific things. Subsequently, we do not need to trigger all of the jobs, per se. Depending on where the changes were made, we may trigger one or more jobs as needed.

Because the jobs are focused on a specific task and specialized, they generally complete execution in a reasonable time.

As already noted, the best practice solution is to configure svn with a post-commit hook so it will notify hudson when changes have been committed. This reduce stress on the SCM and will still facilitate CI.

share|improve this answer
    
This would be my perferred solution; just add another server. We've already got other build servers to do long running integration tests and actual deployment of code, this is literally just doing a build and running some code quality tools. –  Andy Oct 7 '12 at 16:32
add comment

The first thing that seems odd is that you're polling for changes, rather than receiving change notifications. I've never used CruiseControl .NET, but the other continuous integration tools that I've used supported receiving a notification or trigger when the integration branch was updated. This seems more efficient, since you won't be wasting cycles checking for updates that don't exist (I would suspect this would reduce power consumption). If, after this, your build/test cycle in your CI environment is too slow or queues of work are building up, then I'd consider moving to a multiple server environment.

The next thing I would recommend is ensuring some level of confidence that what your CI tools build will always build. Depending on the size and complexity of the system, it might not make sense to build and then run every test in the development environment. However, prior to merging, the developer should at least build the system to ensure that the tests can be run. It might also be a good idea to have smoke tests that hit key areas of the system, but that can be run quickly in a development environment - if they don't pass, the developer doesn't merge changes. Having a more extensive level of unit testing over only changed components is also an option. Just leave the extensive testing to continuous integration, while maintaining sufficient confidence that you'll be left with a good build after the merge.

To me, the idea that one person not being in the office is a problem is also part of the problem. This problem would be partially reduced by what I described above - developer build and test cycles to decrease problems discovered in continuous integration. However, making sure that more than one person can fix issues discovered in parts of the system are also better. Consider code reviews, pair programming, or having people work in different subsystems to spread knowledge around.

I think that very rarely is "throw more technology at the problem" an appropriate solution. Look at how you are using the technology first, along with the goals you want to achieve. It'll probably end up being most cost-effective in the long run.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.