Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We ran into an unfortunate situation at work recently and I've been wondering what we can do to avoid similar problems in the future.

We make embedded systems. The FPGA code is in one SVN repository while the firmware & software code is in a different repository. Building the firmware requires the built FPGA images, so these binary files are also stored in the firmware/software repo. The FPGA build can take 2-3 hours and may randomly fail. To my understanding, these late failures occur because of the complex timing issues in the FPGA logic. Developer errors such as bad syntax will cause the build to fail within minutes.

Because the builds take so long, the FPGA developers generally make their changes, build, give the new image to the firmware developers, and then commit to SVN. A few weeks ago, this caused a lot of issues as we were preparing for a release. One FPGA developer made a small change and started a build. Another FPGA developer made a commit (after he had been told that his changes did not need to be included in this release and that he should wait to commit). After this commit, the initial build failed. The first FPGA developer then had to revert the other's changes, commit his changes, and spend a significant amount of time inspecting the code in SVN to make sure it reflected the code used to build the image required for the firmware build before we tagged for the release.

If we were using git or another DVCS this wouldn't be a problem, since each FPGA developer could commit locally before building and push to a server after a successful build. Switching our version control is currently not an option. We're in the process of throwing more hardware at the problem of long FPGA builds and we're investigating whether the build time can be improved by changing settings in the build tools.

Is there anything else we can do to improve our processes so we don't repeat this situation or discover other problems with versioning and conflicts?

share|improve this question
    
What prevented the person from repeating the build, using the previous revision? Using -r flag and svn cp or merge. –  CaptainMurphy Mar 19 '13 at 18:19
1  
did these "FPGA" guys consider using a dedicated branch to separate the code intended to be delivered to you from their everyday stream of work? –  gnat Mar 19 '13 at 18:25
    
Repeating the build wasn't the problem. I think that the developer just restarted the build without updating. The problem is that we needed to be able to create a tag that accurately reflected the code used in the image we released. –  Velociraptors Mar 19 '13 at 18:26
    
gnat, I'm not sure I understand what you mean. The code that they deliver to the firmware developers is pretty much all of their everyday stream of work. They don't do a lot of multitasking, except for working on products that have separate code bases. –  Velociraptors Mar 19 '13 at 18:30
1  
I might be misunderstanding what you are doing - but it should never, ever be the case that a new commit breaks a running build! At the very least you should be committing before making builds and making builds from specific revisions (that you track), so you can branch from that revision if a problem occurs and more commits have happened since the start of that build. –  Joris Timmermans Mar 20 '13 at 10:29
show 2 more comments

2 Answers

One FPGA developer made a small change and started a build. Another FPGA developer made a commit (after he had been told that his changes did not need to be included in this release and that he should wait to commit). After this commit, the initial build failed. The first FPGA developer then had to revert the other's changes, commit his changes, and spend a significant amount of time inspecting the code in SVN to make sure it reflected the code used to build the image required for the firmware build before we tagged for the release.

The problem is that you are using a single branch for multiple purposes that are conflicting. You are doing release packaging, low risk fixes, and high risk fixes in the same branch.

There is nothing fundamentally wrong with checking in code, its just that it needs to be checked in the right spot. Yes, DCVS makes some aspects of this easier, though it is not the right answer for every solution.

Consider having the trunk or mainline (where you currently check everything in to) be the low risk fixes and low risk development spot - don't break things for other people. Have another branch be the release branch. Once the release is ready, merge all the changes from the mainline into the release branch and build from there. You can still build from the mainline, but that is a dev/nightly release, not one that is ready.

Upon having the release build fail, either check the code into the trunk again and cherry pick (this is key - so that the second developer's code doesn't need to be reverted) that change into the release branch, or check it into the release branch and merge it back into the main line. Different people will have different opinions on which of these is the best-est practice.

Advanced SCM Branching Strategies is a good read on this topic.

In the environment that I work on, there is a branch for the release (release), a branch for the dev build (the trunk), and a branch (or two or three - as necessary) for various high risk changes. Each of these branches has a separate build in the build server.

share|improve this answer
1  
Section on the release (packaging) role in the article you mention isn't just "good read", it's best of all I've read on this topic so far (and I've read a lot:). "Using a separate branch to insulate the release effort from the ongoing development and maintenance, and vice versa, is recommended... restrict efforts to only those fixes necessary to release the product on all target platforms. Strategies were addressed to allow the release effort to proceed in parallel with ongoing maintenance and development activities..." –  gnat Mar 19 '13 at 18:54
1  
@gnat ok, so I understated its importance a bit... –  MichaelT Mar 19 '13 at 18:57
add comment

I would recommend using a code review system in addition to your source control system. This would enforce that each commit would have to be reviewed and approved by other members of the team before being accepted to a repository. Other benefits would include improved knowledge sharing and development process transparency.

edit: Slowing things down is a worry but if you choose the right software review tool and use it correctly it can be fast and greatly beneficial. slightly slowing the development process down in the short term in this manner would spare long debugging/fixing sessions when something silly gets into the central repository, saving time in the long term.

share|improve this answer
1  
While you're right, that would fix the problem, having to constantly review code would become tedious at best and probably require a dedicated programmer assuming the project is big enough which it seems it is. –  Neil Mar 20 '13 at 8:27
    
Yes that is a worry but if you choose the right software review tool and use it correctly it can be fast and greatly beneficial. slightly slowing the development process down in the short term in this manner would spare long debugging/fixing sessions when something silly gets into the central repository, saving time in the long term. It would be advised to have a team of reviewers/approvers so that people are always free to do some reviewing. In fact we allow our whole team to review everyone else's work but each commit must be reviewed by at least two other people before being accepted –  hotpotato Mar 20 '13 at 9:23
    
You're right. It would be greatly beneficial, but this type of thing is very difficult to properly organize, and so you only risk to lose time and resources. I think it is better to assume there will be human error and to handle it in such a way that it won't cause damage even if it does happen. Otherwise, you risk that both the coder and the reviewer make a human error and you publish a bug. –  Neil Mar 20 '13 at 10:17
    
In what way is it difficult to organize? In our experience all we had to do was install the code review system software, add your repository and define a group of 'reviewers'. Then any future commits are caught and the reviewers are notified by email (automated) that they have some reviewing to do, which is performed quickly in a web browser. A cursory glance over the change diffs is all that is required to catch a lot of errors, especially of the type that the original poster is having. –  hotpotato Mar 20 '13 at 13:17
    
A cursory glance is not all that is required to catch a lot of errors. Most errors are extremely convoluted and do not have an obvious syntax error that you can fix. Why not use a nightly and release branch then? Code reviewers can approve code committed to the release branch instead of approving every little code commit. –  Neil Mar 20 '13 at 13:41
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.