Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm a big fan of Git sub-modules. I like to be able to track a dependency along with its version, so that you can roll-back to a previous version of your project and have the corresponding version of the dependency to build safely and cleanly. Moreover, it's easier to release our libraries as open source projects as the history for libraries is separate from that of the applications that depend on them (and which are not going to be open sourced).

I'm setting up workflow for multiple projects at work, and I was wondering how it would be if we took this approach a bit of an extreme instead of having a single monolithic project. I quickly realized there is a potential can of worms in really using sub-modules.

Supposing a pair of applications: studio and player, and dependent libraries core, graph and network, where dependencies are as follows:

  • core is standalone
  • graph depends on core (sub-module at ./libs/core)
  • network depdends on core (sub-module at ./libs/core)
  • studio depends on graph and network (sub-modules at ./libs/graph and ./libs/network)
  • player depends on graph and network (sub-modules at ./libs/graph and ./libs/network)

Suppose that we're using CMake and that each of these projects has unit tests and all the works. Each project (including studio and player) must be able to be compiled standalone to perform code metrics, unit testing, etc.

The thing is, a recursive git submodule fetch, then you get the following directory structure:

studio/
studio/libs/                    (sub-module depth: 1)
studio/libs/graph/
studio/libs/graph/libs/         (sub-module depth: 2)
studio/libs/graph/libs/core/
studio/libs/network/
studio/libs/network/libs/       (sub-module depth: 2)
studio/libs/network/libs/core/

Notice that core is cloned twice in the studio project. Aside from this wasting disk space, I have a build system problem because I'm building core twice and I potentially get two different versions of core.

Question

How do I organize sub-modules so that I get the versioned dependency and standalone build without getting multiple copies of common nested sub-modules?

Possible solution

If the the library dependency is somewhat of a suggestion (i.e. in a "known to work with version X" or "only version X is officially supported" fashion) and potential dependent applications or libraries are responsible for building with whatever version they like, then I could imagine the following scenario:

  • Have the build system for graph and network tell them where to find core (e.g. via a compiler include path). Define two build targets, "standalone" and "dependency", where "standalone" is based on "dependency" and adds the include path to point to the local core sub-module.
  • Introduce an extra dependency: studio on core. Then, studio builds core, sets the include path to its own copy of the core sub-module, then builds graph and network in "dependency" mode.

The resulting folder structure looks like:

studio/
studio/libs/                    (sub-module depth: 1)
studio/libs/core/
studio/libs/graph/
studio/libs/graph/libs/         (empty folder, sub-modules not fetched)
studio/libs/network/
studio/libs/network/libs/       (empty folder, sub-modules not fetched)

However, this requires some build system magic (I'm pretty confident this can be done with CMake) and a bit of manual work on the part of version updates (updating graph might also require updating core and network to get a compatible version of core in all projects).

Any thoughts on this?

share|improve this question

3 Answers 3

I would not use submodules.

It's tempting, same as used to be the case with svn-externals. However, can you be sure all of those projects you link are still at the same place in a year? What about in five?

Therefore, I am simply copying all required dependencies into my project. This means that as long as my repo is valid, I can checkout the exact state.

Basically, I have a folder structure as follows:

myproject/... [sources etc]
ext/ [third-party dependencies]


e.g. ext/boost, ext/cppunit

While this is not very nice from a disk-space perspective, I value the guarantee that I can check out every state recorded as long as the repo is available much higher.

In addition, there are a bunch of problems with submodules as described here

share|improve this answer
    
I'm sure they're in the right location because I'm maintaining all of them :-) Also, be careful about copying projects because of redistribution conditions. –  André Caron May 1 '13 at 0:54
    
OK, that reduces the problem. And licensing: Yes, you have to be careful, but that's a completely different problem. –  Wilbert May 2 '13 at 8:14

I suppose that when you integrate both graph and network submodules into studio, you always must have the same version of core at a given time in the history of studio. I would simlink the studio/libs/core submodule into studio/libs/{graph,network}/libs.

Update:

I creted multiple repositories with the dependencies you stated:

./core      <--- (v2)
./graph
./graph/libs
./graph/libs/core  <--- (v2)
./graph/.gitmodules
./network
./network/libs
./network/libs/core  <--- (v1)
./network/.gitmodules
./studio
./studio/libs
./studio/libs/graph
./studio/libs/graph/libs
./studio/libs/graph/libs/core <--- (v1)
./studio/libs/graph/.gitmodules
./studio/libs/network
./studio/libs/network/libs
./studio/libs/network/libs/core  <--- (v1)
./studio/libs/network/.gitmodules
./studio/studio
./studio/.gitmodules

v1 and v2 are two different versions of core. graph handles version 2, whereas network needs some work and is stuck at version 1. In studio, the local embedded versions of core both points to v1 in order to have a working program. Now, apart from the build perspective, everything works well with submodules.

I can now remove the following directory :

./studio/libs/network/libs/core

And replace it with a symbolic link :

./studio/libs/network/libs/core@ -> ../../graph/libs/core/

I commit locally this change and lose the ability to have two separate versions of core inside studio, but I only build core once. When I'am ready to upgrade to v2, I can do :

 git submodule update # (--rebase ?)

... inside studio/libs/network.

share|improve this answer
    
The symbolic link idea did cross my mind, but it's a non-solution. If you link from graph/libs/core to the outside, you're not using the submodule. If you link from studio/libs/core to one of the sub-module's own libraries, then which one do you choose, graph or network? Moreover, what happens when it's three or more layers deep? Finally, what if core can be a range of revisions. It's not obvious that you want to link to either version of core that graph and network are using. –  André Caron Nov 10 '11 at 22:35
    
"which one do you choose ?" : core would be a submodule fetched from the original core library, updated to a version that is compatible to both graph and network (you must decide which one is good). The symbolic links would be added in the local graph and network submodules (unfetched). –  coredump Nov 11 '11 at 0:03
1  
The symbolic links you propose to add in graph and network would point outside their own repository (e.g. somewhere else in the studio project). How do they know when to use their own sub-module versus when to use the symbolic link? Perhaps you should add an example to demonstrate your line of thinking. –  André Caron Nov 11 '11 at 3:04

I would flatten it down to have sub-module depth of just one and have a repository which would hold all the modules as sub-modules and nothing else apart from README and the build scripts. There would be a separate build script for each of the packages linking its dependencies. Otherwise you can have a separate repo for a package.

share|improve this answer
1  
I'm not sure if this was clear in my post, but I have multiple applications that depend on the same libraries and I don't want to duplicate the build scripts for libraries across the applications. –  André Caron Oct 17 '11 at 20:54
3  
You should elaborate on your answer to demonstrate how it addresses the different issues. It's not clear to me exactly how you link dependencies given that, depending on the context, the dependent libraries are not in the same location. –  André Caron Oct 17 '11 at 20:56

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.