Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

The build server at our company sucks. The build agents (there are a dozen of them, each is a separate VM, and apparently they are all living on their own ESX server) are SLOW. The web interface is SLOW and is running on a Windows VM, also on ESX. Everything is SLOW, SLOW, SLOW. Artifact upload sometimes fails for unknown reasons. Sometimes the web interface reports in the UI that it has run out of memory and it seizes up then crashes. It seems like IT is rebooting it at least once every day. Today they are rolling back a minor update because it has caused problems.

Currently IT owns the hardware that the build servers run on and we have zero visibility into that. We do have admin control over the software. We (the developers) are constantly complaining about it but nothing seems to really improve. Does it make sense for developers to own the build server(s)? IT doesn't really have as much incentive to make them reliable and fast as developers do. In order to get IT to fix the problems we have to complain to our managers and then they have to talk to their managers who then tell IT to fix it. We can report problems to IT directly but that just doesn't seem to work to get things fixed because IT has lots of other things they need to deal with. Also, the developers are the ones who are actually using the build system and are probably a lot more familiar with it than IT.

How does it work at other companies?

We are about a 150 person company. So probably half of that is engineering, but a third of that is probably hardware, so we have maybe 40 or so devs.

share|improve this question

closed as primarily opinion-based by GlenH7, Blrfl, Robert Harvey, gnat, Bart van Ingen Schenau Apr 7 at 11:46

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise.If this question can be reworded to fit the rules in the help center, please edit the question.

1  
As a developer, I do not want to be in the business of MIS. I do not want to be fixing hardware, OS or Jenkins installs, etc. Give me access to configure my builds and I will be happy. It seems like your issue is more with the performance, which is, in the end, a money issue. Talk to management about appropriating funds to fix that. –  Snowman Apr 4 at 20:40
5  
Having been on both sides of that fence, I can tell you from experience that the grass is equally brown on both sides. –  Blrfl Apr 4 at 20:54
    
In today's world of Azure, Amazon, and other cloud services there is no reason to host your own your hardware. At most have a test server to test the build images –  Ramhound Apr 5 at 1:15

2 Answers 2

Ideally developers should not have to own the build server or even maintain the build process themselves. Every build in a company should go through common parameterized pipelines that have been defined for building projects at the company. This provides several benefits: First, developers don't have to worry about the build process, a dedicated team maintains it. Second, new changes to the pipeline can benefit everyone at once, rather than having to roll it out to dozens of individual build processes. Lastly, it allows the organization to derive powerful metrics about the quality of their builds across the board.

This isn't a pipe dream; we actually do this at my company. 300 engineers build several hundred projects in the same pipeline on a regular basis, and we sometimes have thousands of runs through the pipeline in a single day, all building and automatically deploying to production. It works amazingly well, because we have a strong emphasis on Continuous Delivery principles, as espoused by the canonical book on the subject.

My organization didn't arrive there overnight. It required buy-in from executives at the top. We were at the same point as most people about two years ago: Individual builds for projects and manual configuration and deployment to environments. Following principles of Continuous Delivery has yielded enormous benefits for developers.

So, like I said, in my mind that's the ideal builds and deployments strategy. But it sounds like your IT department hasn't bought into the concept of providing quality build systems and processes yet. If you can't get support from your IT dept., then my suggestion would be to try to roll your own process for now with your own build servers if you have the organizational freedom to do so. Leaving IT to do your own thing is a surprisingly effective way to get them to offer better support to you. :)

share|improve this answer

In the past, I had very positive experiences with having a dedicated build team consisting of just a few developers who reported to the QA department. The build team's builds were the company's gold standard. If your build was broken, they wouldn't accept it. The code that shipped came from their servers only. This provided a lot of incentive for each team to ensure their builds and unit tests were sane. This worked very well in a company of nearly 120 developers and 40 QA all spanning ~15 teams.

In my view, each team is responsible for their own builds but not the build server. Leave the developers to develop! A build team is responsible for the build servers and for collaborating with each respective team to ensure build and test sanity.

You may want to take a "grass roots" approach. If you have the option to set aside your own hardware, I would set up a personal build server and set the example. Once word gets out that you are able to quickly provide successful and repeatable builds, others will want to adopt your strategy. Hopefully, this will encourage buy-in from those who have the power to make the necessary changes.

share|improve this answer
1  
I would avoid using developer hardware except for a proof of concept. I have seen a regular desktop system turned into the team's build server because it worked, and management was too cheap to buy server-class hardware. It is the hardware version of a software prototype becoming the end product. –  Snowman Apr 4 at 21:05
    
Agreed. Proof of concept only. The approach I described is to demonstrate a better alternative and to show management that it works. Having a prototype already up and running is a much easier sell to the management. –  Christian Groleau Aug 16 at 16:54

Not the answer you're looking for? Browse other questions tagged or ask your own question.