Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I used to coding in C# in a TDD style - write/or change a small chunk of code, re-compile in 10 seconds the whole solution, re-run the tests and again. Easy...

That development methodology worked very well for me for a few years, until a last year when I had to go back to C++ coding and it really feels that my productivity has dramatically decreased since. The C++ as a language is not a problem - I had quite a lot fo C++ dev experience... but in the past.

My productivity is still OK for a small projects, but it gets worse when with the increase of the project size and once compilation time hits 10+ minutes it gets really bad. And if I find the error I have to start compilation again, etc. That is just purely frustrating.

Thus I concluded that in a small chunks (as before) is not acceptable - any recommendations how can I get myself into the old gone habit of coding for an hour or so, when reviewing the code manually (without relying on a fast C# compiler), and only recompiling/re-running unit tests once in a couple of hours.

With a C# and TDD it was very easy to write a code in a evolutionary way - after a dozen of iterations whatever crap I started with was ending up in a good code, but it just does not work for me anymore (in a slow compilation environment).

Would really appreciate your inputs and recos.

p.s. not sure how to tag the question - anyone is welcome to re-tag the question appropriately.

Cheers.

share|improve this question
    
stackoverflow.com/questions/5078409/… These should probably be merged/in one place. –  Ben Feb 22 '11 at 14:20
    
See laso stackoverflow.com/questions/373142/… for speeding up C++ compile times. –  Eclipse Feb 22 '11 at 21:26
1  
Start a swordfight. ;p –  Steven Jeuris Feb 24 '11 at 12:42

10 Answers 10

Several things come to my mind:

  1. Make use of distributed compilation. You can do this with GCC ("distCC"?) or VC (Xoreax' IncrediBuild is not exactly cheap, but worth every cent spent on it.).

  2. Split your project into dynamically loaded libraries, and carefully try to minimize the dependencies on them. Smaller executables link much faster.

  3. Program against small test projects rather than the whole big application.

  4. Employ template-meta programming to perform algorithms at compile-time. Yes, this will actually increase compilation times, but it will also decrease turnarounds needed for testing: If it compiles fine, it's done.

  5. Invest in hardware. More CPU kernels (in your machine or in others) will do wonder with distributed compilation, and lots of memory plus a fast disk (SSD instead of HDD) will help a lot to. If you have a 64bit system and obscene amounts of RAM, compiling on a RAM disk might provide an incredible speed boost.

share|improve this answer

Another technical solution not yet mentioned by others is switching to Solid State Drives instead of regular hard drives. In a previous project I worked on, SSDs brought down build times from the range of 30 minutes to 3.

Of course, they are costly. For your boss, calculate the price of lost developer time against the price of the one-time investment. The investment probably pays for itself in a few months.

share|improve this answer
4  
That's interesting. That says 90% of build time is I/O disk latency. –  Mike Dunlavey Feb 22 '11 at 14:16
    
I haven't seen a 90% decrease, but a substantial one. It doesn't seem to speed up compilation, but it definitely speeds up linking. If you're making small changes to a large project (so there isn't much compilation in a change), and relinking, you might well get this. (This is Visual Studio 2008, using C++.) –  David Thornley Feb 22 '11 at 14:37
    
@David: Yeah, or ar. I see a big difference between built-in disk and USB disk. –  Mike Dunlavey Feb 22 '11 at 20:10
1  
This is a good idea. Also, mounting a part of RAM on your filesystem would work fast, and it's cheap. –  Goran Jovic Feb 22 '11 at 20:38
2  
Ramdisk is even faster (and cheaper). –  SK-logic Feb 24 '11 at 13:20

More planning, code in bigger chunks, write integration tests instead of unit-tests and run the build + test suite overnight.

share|improve this answer

I can easily remember when builds took a long time. Some mitigating approaches:

  • Build the system by combining libraries or dlls. That way, when you modify some code, the only part that needs to be recompiled is your part.
  • The number of points in the code you need to edit to implement a feature not only effects how much editing you have to do, but the frequency with which you put in bugs, amplifying the compile-debug-edit-compile loop. Anything that reduces the redundancy of the code, such as DRY, helps.
  • If you are in the debugger, and can edit, recompile, and continue without leaving the debugger, that is really helpful.
share|improve this answer

10+ minutes for a compile? Seriously?

Are you using an IDE that does incremental building (e.g. Eclipse)? If not, you probably should be, it'll do the basic compiling in seconds rather than minutes.

Or are you talking about integration stuff, where you need to build the entire app to test your change? If so, look at smaller tests to ensure the major bugs are out of your code before having to do the full build.

share|improve this answer
3  
10mins is small. I've worked for a project that took an hour to compile and link from scratch on a single-core machine, PCHs and all. Of course, except for automatic release builds nobody would build it on just one processor core, but still... If you had to change something in a header that's included just about everywhere (string manipulation, error handling), you could go nuts. –  sbi Feb 22 '11 at 13:14
1  
Used to work (years ago) on a system that took, for a complete build, 48 hours to compile. Of course a full build was only ever started on friday evening, hopefully to be done when we got back to the office on monday. We instead built small modules as needed (say a single DLL). –  jwenting Feb 22 '11 at 13:45
1  
-1 to all the comments above if I could +1 to TrueDub. Yes, if you are recompiling everything then it can take a very long time. However, if any thought at all is given to dependency management then 10 hour entire recompile projects can have incremental recompiles in less than a minute be the norm. You all should be ashamed of yourselves wasting your employers time waiting for your recompiles when applying a bit of intelligence will save vast amounts of time. –  Dunk Feb 22 '11 at 17:27
1  
And if you work in a small shop that happens to sit on a several-MLoC project, then that means that a considerable part of the code is old, started out a decade ago as multiple small projects, where compilation speed never was an issue, and the dependency management is abominably bad. So are you going to tell such a company to throw all this away and spend another decade re-writing it? –  sbi Feb 22 '11 at 23:49
1  
@sbi: Yes I have worked on 100+ developer projects with several million lines of code and the key to not wasting 100's of man hours every week was managing the dependencies and managing the interfaces. If you were creating low level stuff that was used by everyone then you should have nailed down those interfaces VERY EARLY in the process. Then it should have taken a near act of god to change those interfaces since it will cause 10's of hours of wasted developer time. –  Dunk Feb 23 '11 at 3:13

Long compilation times are a problem sometimes, but the already mentioned modularisation can help overcome that (mostly).

Far more serious is being stuck in an environment where you cannot compile at all, where every code change has to be submitted to another department on another continent for application to the test/development environment, a process that can take days to complete.

I'm now working in such an environment, and this system has already cost me over a week of time (and the project only has budget for 4 weeks total time before the money runs out) just to get the initial version of our changes installed (and then they made mistakes which cause part of the files to not be picked up by the application server, so we're looking at several more days of delays). Each minor change now (say we find something in testing that needs fixing, like a missed error condition) can cause a delay of another day or more.

In such conditions you try to make as certain as you can that there are no errors whatsoever before even trying to get your code compiled. It feels almost like I'm back to mainframe programming, where we had 5 minutes of CPU time per month available for all compile and test work.

share|improve this answer
1  
Boy, that's a situation from hell. I do remember the mainframe days. We did a lot of "desk checking" of code. It's amazing how much got done that way, such as endless simulations of flights to the moon. –  Mike Dunlavey Feb 22 '11 at 14:11

First, why does it take so long to compile in the first place?

  • Does your environment (IDE, make, whatever) support incremental builds? Make sure that you are only recompiling the changes, rather than the whole thing.
  • If you have a multi-core machine, your IDE may support parallel compilation. I know for a fact that Visual Studio does that. Apparently so does gcc. So get a better machine, and enable parallel compilation.
  • Consider using precompiled headers.
  • If you try all that, and compilation is still slow, review your code. Look for unnecessary dependencies. Are you including a header where a forward declaration would be sufficient? Consider using the PIMPL idiom to reduce dependency on headers.

If after all of this your build time is still slow, then break up the problem: create many small test projects and work on each individually. Make sure you have a automated nightly build system that does a fresh checkout, builds everything, and runs all the unit tests automatically.

Finally, if it still takes you a long time to test your changes, then put more thought into them. Be sure to do a diff in your version control system and carefully review all the changes before testing. In short, this is very much like embedded systems development, where the turnaround time for a test is long, and your ability to examine the state of the system is limited.

This leads me to another thought: instrument your code to use logging. This way you may be able to see what the problem is without rebuilding and re-running a dozen times.

share|improve this answer
2  
I'm not sure if GCC supports parallel compiles so much as the fact that make or similar tools will start several copies of GCC if you tell it too. –  Zachary K Feb 24 '11 at 14:31

You probably need a multi-prong approach:

1) Faster build systems. As many cores/ram/fast disk as you can afford. For larger C++ projects you'll find that disk is often a limiter, so make sure you have fast ones.

2) More modularization of the project. Break stuff up so that changes can't easily cause full re-compiles of everything. Frankly, push as much basic stuff as possible into separate dll/so files so that part of the project can be completely divorced from the rest.

3) Incremental builds/distributed builds/caching as appropriate to your environment. On some systems, distcc (distributed building) and ccache (caching of partially built stuff) can save a lot of compile time.

4) Make sure your build can be well parallelized. In a makefile environment especially, it's not hard to get into a situation where you've accidentally setup the Makefiles such that you can't do parallel building.

share|improve this answer

Extensive logging and internal validation has been helpful for long turnaround times. Once your build is done, a single run can reveal a large set of possible issues at once.

When dealing with rather complex algorithms or bookkeeping, it can be helpful to include a highly simplified version in parallel with the 'real' one. In any run, you have useful reference data included.

share|improve this answer

What @sbi and @Michael Kohne said.

Spend time and energy on the build process itself. Once upon a time we had a stately, mature product that took more than an hour for a full build. Much time and energy was spent fixing what the build dependencies claimed to be, and then later, fixing/reducing what they actually were. Build time dropped to ~30 minutes.

Changing build tools dropped it more. For a multi-part project, 'scons' can do all compiles before doing any links. 'make' using multiple makefiles does one single project's compiles before that project's links, then moves on.

That brought us to the point that all the individual compile commands could be done massively parallel. 'distcc' on slow machines, make/scons -j8 on multicore machines. That brought full builds down to a handful of minutes.

In a different light, create an automated nightly build process. That way if something problematic gets committed to your source repository, the first person to arrive at work, see and fix the problem, may prevent multiple people from (re)doing multiple failed builds.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.