Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm working on a large software project which is highly customized for various customers arround the world. This means that we have maybe 80% code which is common between the various customers, but also a lot of code which has to change from one customer to the other. In the past we did our development in separate repositories (SVN) and when a new project started (we have few, but large customers) created another repository based on whatever past project has the best code basis for our needs. This has worked in the past, but we ran into several problems:

  • Bugs which are fixed in one repository are not patched in other repositories. This might be a problem of organization, but I find it difficult to fix and patch a bug in 5 different repositories, keeping in mind that the team maintaining this repository might be in another part of the world and we don't have their test environment, neither know their schedule or what requirements they have (a "bug" in one country might be a "feature" in another).
  • Features and improvements made for one project, which might also be useful for another project are lost or if they are used in another project often cause big headaches merging them from one code base to another (since both branches might have been developed independently for a year).
  • Refactorings and code improvements made in one development branch are either lost or cause more harm than good if you have to merge all these changes between the branches.

We are now discussing how to solve these problems and so far came up with the following ideas on how to solve this:

  1. Keep development in separate branches but organizing better by having a central repository where general bug fixes are merged into and having all projects merge changes from this central repository into their own on a regular (e.g. daily) basis. This requires huge discipline and a lot of effort for merging between the branches. So I'm not convinced that it will work and we can keep this discipline, especially when time pressure puts in.

  2. Abandon the separate development branches and have a central code repository where all our code lives and do our customization by having pluggable modules and configuration options. We are already using Dependency Injection containers to resolve dependencies in our code and we are following the MVVM pattern in most of our code to cleanly separate business logic from our UI.

The second approach seems to be more elegant, but we have many unsolved problems in this approach. For example: how do handle changes/additions in your model/database. We are using .NET with Entity Framework to have strongly typed entities. I don't see how we can handle properties which are required for one customer but useless for another customer without cluttering our data model. We are thinking of solving this in the database by using satellite tables (having a separate tables where our extra columns for a specifiy entity live with a 1:1 mapping to the original entity), but this is only the database. How do you handle this in code? Our data model lives in a central library which we would not be able to extend for each customer using this approach.

I'm sure that we are not the only team struggling with this problem and I'm shocked to find so little material on the topic.

So my questions are the following:

  1. What experience do you have with highly customized software, what approach did you choose and how did it work for you?
  2. What approach do you recommend and why? Is there a better approach?
  3. Are there any good books or articles on the topic that you can recommend?
  4. Do you have specific recommendations for our technical environment (.NET, Entity Framework, WPF, DI)?

Edit:

Thanks for all the suggestions. Most of the ideas match those that we already had in our team, but it is really helpful to see the experience you had with them and tips to better implement them.

I'm still not sure which way we will go and I'm not making the decision (alone), but I will pass this along in my team and I'm sure it will be helpful.

At the moment the tenor seems to be a single repository using various customer specific modules. I'm not sure our architecture is up to this or how much we have to invest to make it fit, so some things might live in separate repositories for a while, but I think it's the only longterm solution that will work.

So, thanks again for all responses!

share|improve this question
    
Consider treating database tables as code. –  user1249 Aug 10 '12 at 12:23
    
We are doing this already in the sense that we have our database scripts in our subversion repository, but it doesn't really solve the problems mentioned above. We don't want to have Key-Value style tables in our database model because they come with a lot of problems. So how do you allow additions to your model for individual customers while still maintaining a shared code repository for all of them? –  aKzenT Aug 10 '12 at 12:30

9 Answers 9

up vote 6 down vote accepted

It sounds like the fundamental problem is not just code repository maintenance, but a lack of suitable architecture.

  1. What is the core/essence of the system, that will always be shared by all systems?
  2. What enhancements/deviations are required by each customer?

A framework or standard library encompasses the former, while the latter would be implemented as add-ons (plugins, subclasses, DI, whatever makes sense for the code structure).

A source control system that manages branches and distributed development would probably help also; I'm a fan of Mercurial, others prefer Git. The framework would be the main branch, each customized system would be sub-branches, for example.

The specific technologies used to implement the system (.NET, WPF, whatever) are largely unimportant.

Getting this right is not easy, but it is critical for long-term viability. And of course the longer you wait, the greater the technical debt you'll have to deal with.

You may find the book Software Architecture: Organizational Principles and Patterns useful.

Good luck!

share|improve this answer
1  
Yup. Architecture is often more of a psychological thing than a technical thing though. If you focus completely on the product, then you will end up with a work-breakdown where the components are only useful for that product. If, on the other hand, you focus on building a set of libraries that are going to be more generally useful, then you build a set of capabilities that can be deployed in a wider range of situations. The key, of course, is to find the right balance between these two extremes for your particular situation. –  William Payne Aug 10 '12 at 23:34
    
I think there is a large shared part between many of our branches (>90%), but the last 10% are always in different places, so there are very few components where I cannot imagine some customer specific changes except for some utility libraries which don't contain any business logic. –  aKzenT Aug 11 '12 at 15:15
    
@aKzenT: hmmm... don't imagine, measure instead. Survey the code and look at all the places where customization has occurred, make a list of the components modified, note how often each component has been modified, and think about the kinds and patterns of modifications that have actually been done. Are they cosmetic or algorithmic? Are they adding or changing base functionality? What is the reason for each type of change? Again, this is hard work, and you may not like the implications of what you discover. –  Steven A. Lowe Aug 11 '12 at 21:03
1  
I accepted this as the answer, because we can't take this decision before thinking more about our architecture, identify parts which are common or that SHOULD be common and then see if we can live with a single repository or if forking is necessary (for the moment at least). This answer reflects this best IMO. Thanks for all the other posts, though! –  aKzenT Aug 15 '12 at 9:47

One company I've worked for had the same problem, and the approach to tackle the problem was this: A common framework for all new projects was created; this includes all stuff that has to be the same in every project. E.g. form generating tools, export to Excel, logging. Effort was taken to make sure that this common framework is only improved (when a new project needs new features), but never forked.

Based on that framework, customer-specific code was maintained in seperate repositories. When useful or necessary, bug fixes and improvements are copy-pasted between projects (with all the caveats described in the question). Globally useful improvements go into the common framework, though.

Having everything in a common codebase for all customers has some advantages, but on the other hand, reading the code becomes difficult when there are countless ifs to make the program behave differently for each customer.

EDIT: One anecdote to make this more understandable:

The domain of that company is warehouse management, and one task of a warehouse management system is to find a free storage location for incoming goods. Sounds easy, but in practice, a lot of constraints and strategies have to be observed.

At one point in time, management asked a programmer to make a flexible, parameterisable module to find storage locations, which implemented several different strategies and should have been used in all subsequent projects. The noble effort resulted in a complex module, which was very difficult to understand and maintain. In the next project, the project lead couldn't figure out how to make it work in that warehouse, and the developer of said module was gone, so he eventually ignored it and wrote a custom algorithm for that task.

A few years later, the layout of the warehouse where this module was originally used changed, and the module with all its flexibility didn't match the new requirements; so I replaced it with a custom algorithm there, too.

I know LOC is not a good measurement, but anyway: the size of "flexible" module was ~3000 LOC (PL/SQL), while a custom module for the same task takes ~100..250 LOC. Therefore, trying to be flexible extremely increased the size of the code base, without gaining the reusability we had hoped for.

share|improve this answer
    
Thanks for your feedback. This is an extension to the first solution described and we thought about this also. However since most of our libraries use some core entities and these core entities are usually extended for one customer or another I think we could only put very few libraries in this core repository. E.g. we have a "Customer" entity defined and ORM mapped in one library which is used by almost all of our other libraries and programs. But every client has some extra fields they need to add to a "Customer" so we would need to fork this library and hence all libraries depending on it. –  aKzenT Aug 10 '12 at 12:54
3  
Our idea to avoid countless "ifs" is to make extensive use of dependency injection and exchanging complete modules for different customers. Not sure how this will work out though. How did this approach work for you? –  aKzenT Aug 10 '12 at 12:55
    
+1, this basically matches my experience of projects that have handled this well. If you're going to use the 'ifs for different customers' approach, 2 points: 1. Don't do if (customer1) ..., instead do if (configurationOption1) ... and have per-customer config options. 2. Try not to do it! Maybe 1% of the time that will be the better (more understandable/easily maintainable) option than having configuration-specific modules. –  Baqueta Aug 10 '12 at 13:10
    
@Baqueta: Just to clarify: you recommend using per-customer modules instead of configuration options (ifs), right? I like your idea to differentiate between features instead of customers. So the combination of both would be to have various "feature modules" which are controlled by configuration options. A customer is then only represented as a set of independent feature modules. I like this approach a lot, but I'm not sure how to architect this. DI solves the problem of module loading and exchanging, but how do you manage different data models between customers? –  aKzenT Aug 10 '12 at 13:37
    
Yup, per-feature modules and then per-customer configuration/selection of features. DI would be ideal. Unfortunately I've never had to combine your requirements of substantial per-customer customisation with a single data library though, so I'm not sure I can be much help there... –  Baqueta Aug 10 '12 at 13:51

I worked for many years on a Pension Administration application which had similar issues. Pension plans are vastly different between companies, and require highly specialized knowledge for implementing calculation logic and reports and also very different data design. I can only give a brief description of part of the architecture, but maybe it will give enough of the idea.

We had 2 separate teams: a core development team, which was responsible for the core system code (which would be your 80% shared code above), and an implementation team, which had domain expertise in pension systems, and was responsible for learning client requirements and coding scripts and reports for the client.

We had all of our tables defined by Xml (this before the time when entity frameworks were time-tested and common). The implementation team would design all the tables in Xml, and the core application could be prompted to generate all the tables in Xml. There were also associated VB script files, Crystal Reports, Word docs etc. for each client. (There was also an inheritance model built into the Xml to enable reusing other implementations).

The core application (one application for all clients), would cache all the client specific stuff when a request for that client came, and it generated a common data object (kind of like a remote ADO record set), which could be serialized and passed around.

This data model is less slick that entity/domain objects, but it is highly flexible, universal, and can be processed by one set of core code. Perhaps in your case, you could define your base entity objects with only the common fields, and have an additional Dictionary for custom fields (add some kind of set of data descriptors to your entity object so that it has meta data for the custom fields.)

We had separate source repositories for the core system code and for the implementation code.

Our core system actually had very little business logic, other than some very standard common calculation modules. The core system functioned as: screen generator, script runner, report generator, data access and transport layer.

Segmenting core logic and customized logic is a tough challenge. However, we always felt it was better to have one core system running multiple clients, rather than multiple copies of the system running for each client.

share|improve this answer
    
Thanks for the feedback. I like the idea of having additional fields in a dictionary. This would allow us to have a single definition of an entity and put all client specific stuff in a dictionary. I'm not sure though if there is a good way to make it work with our ORM wrapper (Entity Framework). And I'm also not sure if it's really a good idea to have a globally shared data model instead of having a model for every feature/module. –  aKzenT Aug 10 '12 at 15:08

One of the projects I've worked on supported multiple platforms (more than 5) across a large number of product releases. A lot of the challenges you are describing were things we faced, albeit in a slightly different way. We had a proprietary DB, so we didn't have the same types of problems in that arena.

Our structure was similar to yours, but we had a single repository for our code. Platform specific code went into their own project folders within the code tree. Common code lived within the tree based upon what layer it belonged to.

We had conditional compilation, based upon the platform being built. Maintaining that was kind of a pain, but it only had to be done when new modules were added at the platform specific layer.

Having all of the code in a single repository made it easy for us to make bug fixes across multiple platforms and releases at the same time. We had an automated build environment for all the platforms to serve as a backstop in case new code broke a presumed unrelated platform.

We tried to discourage it, but there would be cases where a platform needed a fix based upon a platform-specific bug that was in otherwise common code. If we could conditionally override the compile without making the module look fugly, we'd do that first. If not, we would move the module out of common territory and push it into platform specific.

For the database, we had a few tables that had platform specific columns / modifications. We would make sure that every platform version of the table met a baseline level of functionality so common code could reference it without worrying about platform dependencies. Platform specific queries / manipulations were pushed into the platform project layers.

So, to answer your questions:

  1. Lots, and that was one of the best teams I've worked with. Codebase at that time was around 1M loc. I didn't get to choose the approach, but it worked out pretty dang well. Even in hindsight, I haven't seen a better way of handling things.
  2. I recommend the second approach you suggested with the nuances I mention in my answer.
  3. No books that I can think of, but I would research multi-platform development as a starter.
  4. Institute some strong governance. It's the key to making sure your coding standards are followed. Following those standards is the only way to keep things manageable and maintainable. We had our share of impassioned pleas to break the model we were following, but none of those appeals ever swayed the entire senior development team.
share|improve this answer
    
Thanks for sharing your experience. Just to understand better: How is a platform defined in your case. Windows, Linux, X86/x64? Or something more related to different customers/environments? You are making a good point in 4) I think it's one of the problems we have. We have a team of a lot of very smart and qualified people, but everyone has slightly different ideas on how to do things. Without someone clearly in charge of architecture it's hard to agree on a common strategy and you risk loosing yourself in discussions without changing anything. –  aKzenT Aug 10 '12 at 15:15
    
@aKzenT - yes, I meant physical hardware and OS as platforms. We had a large feature set, some of which was selectable by a licensing module. And we supported a broad array of hardware. Related to that, we had a common layer directory that had a bunch of devices with their own directories in that layer for the code tree. So our circumstances really weren't all that far off from where you are. Our senior devs would have spirited debates with each other, but once a collective decision was made everyone agreed to toe the line. –  GlenH7 Aug 10 '12 at 15:45

I've worked on a smaller system (20 kloc), and found that DI and configuration are both great ways to manage differences between clients, but not enough to avoid forking the system. The database is split between an application specific part, which has a fixed schema, and the client dependent part, which is defined through a custom XML configuration document.

We've kept a single branch in mercurial that's configured as if it was deliverable, but branded and configured for a fictional client. Bug fixes are mainlined into that project, and new development of core functionality only happens there. Releases to actual clients are branches off of that, stored in their own repositories. We keep track of large changes to the code through manually written version numbers and track bug fixes using commit numbers.

share|improve this answer
    
did you differentiate between core libraries which are the same between the customers or do you fork the complete tree? Do you regulary merge from the mainline to the individual forks? And if yes how much time did the daily merge routine cost and how did you avoid that this procedure falls apart when time pressure starts (like it always does at one point in the project)? Sorry for so many questions :p –  aKzenT Aug 10 '12 at 15:04
    
We fork the whole tree, mainly because client requests come before build system integrity. Merging from the main system happens manually using mercurial and other tools, and is typically limited to critical bugs or large feature updates. We try to keep updates to large infrequent chunks, both because of the cost of the merge, and because many clients host their own systems and we don't want to put installation costs onto them without providing value. –  Dan Monego Aug 10 '12 at 15:20
    
I'm curious: IIRC mercurial is a DVCS similar to git. Did you notice any advantages doing these merges between branches in comparison to Subversion or other traditional VCS? I just finished a very painful merge process between 2 completely separate developed branches using subversion and was thinking if it would had been easier if we used git. –  aKzenT Aug 10 '12 at 15:27
    
Merging with mercurial has been much, much easier than merging with our prior tool, which was Vault. One of the main advantages is that mercurial is really good about putting the change history front and center in the application, which makes it easy to track what was done where. The most difficult part for us was getting the existing branches moved over - if you're merging two branches, mercurial requires that they both have the same root, so getting that set up required one last completely manual merge. –  Dan Monego Aug 10 '12 at 15:40

I am afraid that I do not have direct experience of the problem that you describe, but I do have some comments.

The second option, of bringing the code together into a central repository (as much as practicable), and architecting for customization (again, as much as practicable) is almost certainly the way to go in the long term.

The problem is how you plan to get there, and how long it is going to take.

In this situation, it is probably OK to (temporarily) have more than one copy of the application in the repository at a time.

This will enable you to gradually move to an architecture that directly supports customization without having to do it in one fell swoop.

share|improve this answer

The second approach seems to be more elegant, but we have many unsolved problems in this approach.

I am sure any of those problems can be solved, one after a another. If you get stuck, ask here on or on SO about the specific problem.

As others have pointed out, having one central codebase / one repository is the option you should prefer. I try to answer your example question.

For example: how do handle changes/additions in your model/database. We are using .NET with Entity Framework to have strongly typed entities. I don't see how we can handle properties which are required for one customer but useless for another customer without cluttering our data model.

There are some possibilities, all of them I have seen in real-world systems. Which one to choose depends on your situation:

  • live with the cluttering to a certain degree
  • introduce a tables "CustomAttributes" (describing names and types) and "CustomAttributeValues" (for the values, for example stored as a string representation, even if they are numbers). That will allow to add such attributes at install time or run time, having individual values for each customer. Don't insist on having each custom attribute modeled "visibly" in your data model.

  • now it should be clear how to use that in code: have just general code for accessing those tables, and individual code (perhaps in a separate plug-in DLL, which is up to you) for interpreting that attributes correctly

  • another alternative is to give each entity table a big string field where you can add an individual XML-string.
  • try to generalize some concepts, so they could be more easily reused across different customers. I recommend Martin Fowler's book "Analysis patterns". Though this book is not about customizing software pre se, it may be helpful for you either.

And for specific code: you can also try to introduce a scripting language into your product, especially for adding customer-specific scripts. That way you do not only create a clear line between your code and customer-specific code, you can also allow your customers to customize the system to some degree by themselves.

share|improve this answer
    
The problems I see with adding CustomAttributes or XML columns to store custom properties is that they are very limited in their abilities. For example doing sorting, grouping or filtering based on these attributes is very challenging. I once worked on a system which used an extra attribute table and it become increasingly complex to maintain and handle these custom attributes. For this reasons I was thinking instead of putting these attributes as columns in an extra table which is mapped 1:1 to the original. The problem on how to define, query and manage these is still the same though. –  aKzenT Aug 12 '12 at 9:31
    
@aKzenT: Customizability does not come for free, you will have to trade off ease of use vs. customizability. My general suggestion is that you don't introduce dependencies where the core part of the system depends in any way from any custom part, only the other way round. For example, when introducing these "extra tables" for customer 1, can you avoid to deploy that tables and related code for customer 2? If the answer is yes, then the solution is ok. –  Doc Brown Aug 12 '12 at 15:37

I have only built one such application. I'd say that 90% of the units sold were sold as is, no modifications. Each customer had their own customized skin and we served up the system within that skin. When a mod did come in that affected the core sections we tried using IF branching. When mod #2 came in for the same section we switched to CASE logic whiched allowed for future expansion. This seemed to handle most of the minor requests.

Any further minor custom requests were handled by implementing Case logic.

If the mods were two radical, we built a clone (separate include) and wrapped a CASE around it to include the different module.

Bug fixes and modifications on the core effected all users. We tested thoroughly in development before going to production. We always sent out email notifications that accompanied any change and NEVER, NEVER, NEVER posted production changes on Fridays... NEVER.

Our environment was Classic ASP and SQL Server. We were NOT a spaghetti code shop... Everything was modular using Includes, Subroutines and Functions.

share|improve this answer

When I'm asked to start the development of B which is sharing 80% functionality with A, I will either:

  1. Clone A and modify it.
  2. Extract the functionality that both A and B share into C which they will use.
  3. Make A configurable enough to fulfill the needs of both B and itself (therefore B is embedded in A).

You chose 1, and it doesn't seem to fit your situation well. Your mission is to predict which of 2 and 3 is a better fit.

share|improve this answer
1  
Sounds easy, but how to do this in practice? How do you make your software so configurable without cluttering it with "if(customer1)", which becomes unmaintainable after some time. –  aKzenT Aug 10 '12 at 14:44
    
@aKzenT That's why I left 2 and 3 for you to choose. If the kind of changes needed to make customer1's project support customer2's needs through configuration is going to make the code unmaintainable then don't do 3. –  Zippoxer Aug 13 '12 at 16:14
    
I prefer doing "if (option1)" instead of "if (customer1)". This way with N options I can manage a lot of possible customers. E.g. with N boolean options you can manage 2^N customers, having only N 'if'... impossible to manage with "if (customer1)" strategy which would require 2^N 'if'. –  Fil Jun 12 at 10:16

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.