Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

To give you a little background: I work for a company with roughly twelve Ruby on Rails developers (+/- interns). Remote work is common. Our product is made out of two parts: a rather fat core, and thin up to big customer projects built upon it. Customer projects usually expand the core. Overwriting of key features does not happen. I might add that the core has some rather bad parts that are in urgent need of refactorings. There are specs, but mostly for the customer projects. The worst part of the core are untested (not as it should be...).

The developers are split into two teams, working with one or two PO for each sprint. Usually, one customer project is strictly associated with one of the teams and POs.

Now our problem: Rather frequently, we break each other's stuff. Someone from Team A expands or refactors the core feature Y, causing unexpected errors for one of Team B's customer projects. Mostly, the changes are not announced over the teams, so the bugs hit almost always unexpected. Team B, including the PO, thought about feature Y to be stable and did not test it before releasing, unaware of the changes.

How to get rid of those problems? What kind of 'announcement technique' can you recommend me?

share|improve this question
32  
The obvious answer is TDD. –  mouviciel Jun 12 at 8:19
1  
How comes you state that "Overwriting of key features does not happen", and then your problem is that it does happen? Do you differentiate in your team between "core" and "key features", and how do you do that? Just trying to understand the situation ... –  logc Jun 12 at 10:32
4  
@mouvciel That and don't use dynamic typing, but that particular bit of advice comes a bit too late in this case. –  Doval Jun 12 at 11:29
3  
Use a strongly typed language like OCaml. –  Gaius Jun 12 at 12:31
    
@logc May be I wasnt clear, sorry. We dont overwrite a core feature like the filter library itself, but add new filters to the classes we use in our customer projects. One common scenario can be, that changes in the filter library destroy those added filters in the customer project. –  SDD64 Jun 12 at 13:00

10 Answers 10

up vote 23 down vote accepted

I would recommend reading Working Effectively with Legacy Code by Michael C. Feathers. It explains that you really need automated tests, how you can easily add them, if you don't already have them, and what "code smells" to refactor in what way.

Besides that, another core problem in your situation seems a lack of communication between the two teams. How big are these teams? Are they working on different backlogs?

It's almost always bad practice to split up teams according to your architecture. E.g. a core team and a non-core team. Instead, I would create teams on functional domain, but cross-component.

share|improve this answer
    
I have read in "The Mythical Man-Month" that code structure usually follows team/organisation structure. Thus, this is not really "bad practice", but just the way things usually go. –  Marcel Jun 12 at 8:22
    
I think in "Dynamics of software development", the manager behind Visual C++ recommends vividly having feature teams; I have not read "The Mythical Man-Month", @Marcel, but AFAIK it lists bad practices in the industry ... –  logc Jun 12 at 10:43
    
Marcel, it is true that this is the way things usually go or went, but more and more teams are doing it different, e.g. feature teams. Having component based teams results in a lack of communication when working on cross component features. Next to that it will almost always result in architectural discussions not based on the purpose of a good architecture, but people trying to push responsibilities to other teams/components. Hence, you'll get the situation described by the author of this question. See also mountaingoatsoftware.com/blog/the-benefits-of-feature-teams. –  Tonni Tielens Jun 12 at 12:01
    
Well, as far as I understood the OP, he stated that the teams are not split into a core and non-core team. The teams are split-up "per customer", which is essentially "per functional domain". And that's part of the problem: since all teams are allowed to change the common core, changes from one team affect the other. –  Doc Brown Jun 12 at 12:17
    
@DocBrown You are right. Each team may change the core. Of course, those changes are supposed to be beneficial for each project. However, they work on different backlogs. We have one for each customer and one for the core. –  SDD64 Jun 12 at 13:22

Primarily, you have a communication problem (probably also linked with a team building problem), so I think a solution to your case should be focused on... well, communication, instead of development techniques.

I take for granted that it is not possible to freeze or fork the core module when starting a customer project (otherwise then you simply need to integrate in your company schedules some non-customer-related projects that aim at updating the core module).

So we're left with the issue of trying to improve the communication between the teams. This can be addressed in two ways:

  • with human beings. This means that your company designates someone as the core module architect (or whatever lingo is good for the top management) who will be responsible for the quality and availability of the code. This person will incarnate the core. Thus, she will be shared by all the teams and ensure proper sync between them. Furthermore, she should also act as a reviewer of code committed to the core module to maintain its coherence;
  • with tools and workflows. By imposing Continuous Integration on the core, you will make the core code itself the communication medium. This will require some effort first (by the addition of automated test suites on it) but then the nightly CI reports will be a gross status update of the core module.

You can find more about CI as a communication process here.

Finally, you stil have a problem with the lack of team work at the company level. I'm not a big fan of team building events, but this seems like a case where they would be useful. Do you have developer-wide meetings on a regular basis ? Can you invote people from other teams to your project retrospectives ? Or perhaps have some Friday evening beer sometimes ?

share|improve this answer

While every system needs effective test suites (which means, among other things, automation), and while these tests, if used effectively, will catch these conflicts sooner than they are now, this does not address the underlying problems.

The question reveals at least two underlying problems: the practice of modifying the 'core' in order to satisfy requirements for individual customers, and the failure of the teams to communicate and coordinate their intent to make changes. Neither of these are root causes, and you will need to understand why this is being done before you can fix it.

One of the first things to be determined is whether both the developers and the managers realize there is a problem here. If at least some do, then you need to find out why they either think they cannot do anything about it, or choose not to. For those who do not, you might try to increase their ability to anticipate how their current actions may create future problems, or replace them with people who can. Until you have a workforce that is aware of how things are going wrong, you are unlikely to be able to fix the problem (and perhaps not even then, at least in the short term.)

It may be difficult to analyze the problem in abstract terms, at least initially, so focus on a specific incident that resulted in a problem, and try to determine how it happened. As the people involved are likely to be defensive, you will need to be alert for self-serving and post-hoc justifications in order to find out what is really happening.

There is one possibility that I hesitate to mention because it is so unlikely: the customers' requirements are so disparate that there is insufficient commonality to justify shared core code. If this is so, then you actually have multiple separate products, and you should manage them as such, and not create an artificial coupling between them.

share|improve this answer
    
Before we migrated our product from Java to RoR, we actually did like you suggested. We one had a Java core for all customers, but their requirements 'broke' it one day and we had to split it up. During that situation, we faced problems like: 'Dude, customer Y does have such a nice core feature. Too bad we cant port it to customer Z, because their core is incompatible'. With Rails, we strictly want to go for a 'one core for all' policy. If it has to be, we still offer drastic changes, but those disattach the customer from any further updates. –  SDD64 Jun 16 at 6:37
    
Just calling TDD seems not enough for me. So, besides the splitting of the core suggestion, I like your answer the most. Sadly, the core is not perfectly tested, but that would not solve all our problems. Adding new core features for one customer may seem perfectly fine and even give a green build, for them, because only the core specs are shared between the customers. One does not notice, what happens to every possible customer. So, I like your suggestion to find out the problems and to talk about what caused them. –  SDD64 Jun 16 at 6:50

"Mostly, the changes are not announced over the teams, so the bugs hit almost always unexpected"

Communications problem anyone? What about (in addition to what everyone else has already pointed out, that you should be rigorous testing) making sure that there is proper communication? That people are made aware that the interface they're writing to is going to change in the next release and what those changes will be?
And give them access to at least a dummy interace (with empty implementation) as soon as possible during development so they can start writing their own code.

Without all that, unit tests won't do much except point out during the final stages that there's something out of whack between parts of the system. You want to know that, but you want to know it early, very early, and having the teams talk to each other, coordinate efforts, and actually have frequent access to the work the other team is doing (so regular commits, not one massive commit after several weeks or months, 1-2 days before delivery).
Your bug is NOT in the code, certainly not in the code of the other team that didn't know you were messing around with the interface they're writing against. Your bug is in your development process, the lack of communication and collaboration between people. Just because you're sitting in different rooms doesn't mean you should isolate yourselves from the other guys.

share|improve this answer

We all know that unit tests are the way to go. But we also know that realistically retro-fitting these to a core is difficult.

A specific technique that might be useful for you when extending the functionality is to try to temporarily and locally verify that existing functionality has not been changed. This can be done like this:

Original pseudo code:

def someFunction
   do original stuff
   return result
end

Temporary in-place test code:

def someFunctionNew
   new do stuff
   return result
end

def someFunctionOld
   do original stuff
   return result
end

def someFunction
   oldResult = someFunctionOld
   newResult = someFunctionNew
   check oldResult = newResult
   return newResult
end

Run this version through whatever system level tests that exist. If everything is fine, you know you have not broken things and can then proceed to remove the old code. Note that when you check the old and new results match, you might also add code to analyse differences to capture cases that you know should be different because of an intended change such as a bug fix.

share|improve this answer

As a more long term fix, you also need better and more timely communication between teams. Each of the teams that will ever utilize, for example, core feature Y, need to be involved in building the planned testcases for the feature. This planning, in and of itself, will highlight the different use cases inherent in feature Y between the two teams. Once how the feature should work is nailed down, and the testcases are implemented and agreed upon, there's an additional change in your implementation scheme that's required. The team releasing the feature is required to run the testcase, not the team that's about to use it. The task, if any, that should cause collisions, is the addition of a new testcase from either of the teams. When a team member thinks of a new aspect of the feature that is not tested, they should be free to add a testcase that they have verified passing in their own sandbox. In this way, the only collisions that will occur will be at the intent level, and should be nailed down before the refactored feature is released into the wild.

share|improve this answer

Other answers have highlighted important points (more unit tests, feature teams, clean interfaces to the core components), but there is one point I find missing, which is versioning.

If you freeze the behavior of your core by doing a release1 and you put that release into a private artifact management system2, then any customer project can declare its dependency on core version X, and it will not be broken by the next release X + 1.

The "announcement policy" then just reduces to having a CHANGES file along with each release, or having a team meeting to announce all features of each new core release.

Also, I think you need to better define what is "core", and what subset of that is "key". You seem to (correctly) avoid making many changes to "key components", but you allow frequent changes to "core". In order to rely on something, you have to keep it stable; if something is not stable, do not call it core. Maybe I could suggest calling it "helper" components?

EDIT: If you follow the conventions in the Semantic Versioning system, then any incompatible change in the core's API must be marked by a major version change. That is, when you change the previously existing core's behavior, or remove something, not just add something new. With that convention, developers know that updating from version '1.1' to '1.2' is safe, but going from '1.X' to '2.0' is risky and has to be carefully reviewed.

1: I think this is called a gem, in the world of Ruby
2: The equivalent to Nexus in Java or PyPI in Python

share|improve this answer
    
"Versioning" is important, indeed, but when one tries to solve the described problem by freezing the core before a release, then you easily end up with the need for sophisticated branching & merging. The reasoning is that during a "release build" phase of team A, A might have to change the core (at least for bugfixing), but won't accept changes to the core from other teams - so you end up with one branch of the core per team, to be merged "later", which is a form of technical debt. That's sometimes ok, but often it just postpones the described problem to a later point in time. –  Doc Brown Jun 12 at 13:49
    
@DocBrown: I agree with you, but I wrote under the assumption that all developers are cooperative and grown-up. This is not to say I have not seen what you describe. But a key part of making a system reliable is, well, striving for stability. Furthermore, if team A needs to change X in the core, and team B needs to change X in the core, then maybe X does not belong in the core; I think that is my other point. :) –  logc Jun 12 at 13:58
    
@DocBrown Yes, we learned to use one branch of the core for each customer project. This caused some other problems. For example we don't like to 'touch' already deployed customer systems. As a result, they might encounter several minor version jumps of their used core after each deployment. –  SDD64 Jun 12 at 13:58
    
@SDD64: that's exactly what I am saying - not integrating changes immediately to a common core is no solution in the long term as well. What you need is a better testing strategy for your core - with automatic and manual tests as well . –  Doc Brown Jun 12 at 14:04
1  
For the record, I am not advocating a separate core for each team, nor denying that tests are required -- but a core test and its implementation can change at the same time, as I commented before. Only a frozen core, marked by a release string or a commit tag, can be relied upon by a project that builds on top of it (excluding bug fixes and provided that the versioning policy is sound). –  logc Jun 12 at 14:25

Like other people said, a good suite of unit tests wont resolve your problem : you will have problem while merging changes, even if each team test suite passes.

Same for TDD. I dont see how it can solve this.

Your solution is non technical. You need to clearly define the "core" boundaries and assign a "watch dog" role to someone, be it the lead dev or architect. Any changes to the core must pass through this watchdog. He is responsible to make sure every output from all teams will merge without too much collateral damages.

share|improve this answer
    
We had a "watch dog", since he wrote most of the core. Sadly, he was also responsible for most of the untested parts. He was YAGNI impersonated and has been replaced half a year ago by two other guys. We still struggle to refactore those 'dark parts'. –  SDD64 Jun 12 at 13:51
2  
The idea is to have a unit test suite for the core, which is part of the core, with contributions from all teams, not separate test suites for each team. –  Doc Brown Jun 12 at 13:54
2  
@SDD64: you seem to confuse "You Ain't Gonna need it (yet)" (which is an very good thing) with "You Ain't need to clean up your code (yet)" - which is an extremly bad habit, and IMHO quite the opposite. –  Doc Brown Jun 12 at 13:57
    
The watchdog solution is really, really suboptimal, IMHO. It is like building a single point of failure into your system, and on top of that a very slow one, because it involves a person and politics. Otherwise, TDD can of course help with this problem: each core test is an example to customer project devs how the current core is supposed to be used. But I think you gave your answer in good faith ... –  logc Jun 12 at 14:19
    
@DocBrown: Okay, maybe our understandings differ. The core features, written by him, are overly complicated to satisfy even the strangest possibilities. Most of them, we never encountered. The complexity slows us down to refactor them, on the other side. –  SDD64 Jun 12 at 14:31

The previous answers that point you to better unit tests are good, but I feel that there might be more fundamental issues to adress. You need clear interfaces for accessing core code from the code for the customer projects. This way if you refactor the core code without altering the behaviour as observed through the interfaces, the other team's code will not break. This will make it much easier to know what can be "safely" refactored, and what needs a, possibly interface breaking, redesign.

share|improve this answer
    
Spot on. More automated testing will bring nothing but benefits and is entirely worth doing, but it won't solve the core problem here which is a failure to communicate core changes. Decoupling by wrapping interfaces round important features will be a huge improvement. –  Matt Thrower Jun 13 at 8:09

We worst part of the core are untested (as it should be...).

This is the problem. Efficient refactoring depends heavily on suite of automated test. If you don't have those, the problems you are describing begin to appear. This is especially important if you use dynamic language like Ruby, where there is no compiler to catch basic errors related to passing parameters to methods.

share|improve this answer
10  
That and refactoring in baby steps and committing very frequently. –  Stefan Billiet Jun 12 at 7:19
1  
There's probably reams of advice that could add advice here, but it'll all come down to this point. Whatever about the OPs "as it should be" joke showing they know it's a problem in itself, the impact of scripted testing on refactoring is immense: If a pass has become a fail, then the refactoring hasn't worked. If all passes remain passes, then the refactoring might have worked (moving fails to passes would obviously be a plus, but keeping all passes as passes is more important than even a nett gain; a change that breaks one test and fixes five might be an improvement, but not a refactoring) –  Jon Hanna Jun 12 at 10:25
    
I gave you a "+1", but I think "automated tests" are not the only approach to solve this. A better manual, but systematic QA, maybe by a separate QA team could solve the quality issues too (and it makes probably sense to have both - automatic and manual tests). –  Doc Brown Jun 12 at 12:25
    
A good point, but if the core and the customer projects are separate modules (and moreover in a dynamic language like Ruby), then the core can change both a test and its associated implementation, and break a dependent module without failing its own tests. –  logc Jun 12 at 13:31
    
As others have commented. TDD. You probably already recognize you should have unit tests for as much of the code as possible. While writing unit tests just for the sake of it is a waste of resources, when you start refactoring any component you should start with extensive test writing before touching the core code. –  jb510 Jun 12 at 22:30

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.