Managing 3rd party sources and binaries used by code under source control

Question

I have a large code base under source control (was subversion, now git). To compile the code and run the tests I use a set of 3rd party libraries. These libraries can be divided into few categoriesL

• Binaries only
• 3rd party sources
• 3rd party sources + local modifications

Each library has its {Windows, Linux} X {debug, release} X {32bit, 64bit} configurations. In addition these libraries evolve with time and different versions of my project use different versions/builds of these libraries.

My question is what is the best way to store these 3rd parties?

Here is my set of preferences:

1. Keep the size of the project source repository small
2. Keep the project source in sync with the 3rd parties so I can always compile and run and old version
3. Simple to manage
4. Cross platform

I tried and thought of several solutions but neither was satisfactory:

1. Use a versioned script to fetch the binaries from a manually managed ftp server that holds all versions of the libraries. This works, but requires careful management of the directory structure on the server. It is error prone as someone might overwrite one of the binaries with a new build.
2. SVN externals - At the time SVN externals could not refer to a specific tag. Today I am using git.
3. Git submodules - Pulls the entire external repository which can be huge. Alternatively it requires managing a separate repository for every single library. The submodule points at a specific tag which means either I get all the externals, when I need only some, or I mimic some weird file system in the git tree.

It is clear to me that the 3rd party sources need to be stored in git in a vendor branch, but the binaries and headers are a different story.

Answer 1

A fair solution for my problem is git-subtree which was recently merged into mainline git. It provides a fair balance between my requirements and the platform limitations. Now I have multiple repositories for the externals (each has a vendor branch as well as local changes branch) and each project repository takes parts of these externals into sub-folders. To keep things organized I maintain a 'bin' and 'lib' folders which contains soft links to the appropriate files/folders in the externals sub-folder.

git-subtree allows to merge a sub-tree from an external repository into a sub-folder. The sub-folder can be merged back and forth with the external repository.

Pros/Cons:

1. Small repository - The repository is not as small as I would like it to be but it contains only the necessary parts from the external repositories. To save space I try to keep the external trees small. I consider it a good price to pay when in return I get simplicity and robustness; as loading and updating a project is a simple git pull and all project related data is contained in a single repository

2. Project/Externals sync - As the project and externals are versioned in the same repository, I can checkout any branch/tag I want and expect it to be working.

3. Simplicity - Day-by-day work is straight forward. Updating external repository, creating a new one or switching to a different version of the external may be tricky and requires special syntax. However this does happen too much. The best thing is that one can add a new external to this project first and only afterwards split it (using git-subtree) into its own repository.

4. Cross platform - Well it's git

5. Binaries - I decided to avoid holding binaries and provide Makefiles instead. I came to this decision as some of my externals depend on other externals which makes it very hard to build a binary that doesn't change very often. For some externals I do store the binaries due to very long build times.

Structure:

/root
   /External
      /External1 (git-subtree from git@git.domain.com:External1 v1.0)
      /External2 (git-subtree from git@git.domain.com:External2 v0.7)
   /lib
      /libExternal1.a -> ../External/External1/libExternal1.a
      /libExternal2.a -> ../External/External1/libExternal2.a
   /include
      /External1 -> ../External/External1/include
      /External2 -> ../External/External2/include

Answer 2

We have gone for a variant of your option 3. Option 1 seems to me to be equivalent to Option 3, but with more implementation/testing effort on your part and hence more potential to go wrong.

Ultimately, if you want to be able to exactly re-create a build, you'll need your externals (including binaries) to be versioned along with the code itself and hosted locally. And git submodules will do a good job of doing this for you.

Answer 3

For the third-party sources, I think submodules are exactly what you're looking for. If you don't want to require the entire upstream history in every clone, frontend the upstream repo with your own, containing a handcrafted branch with just the necessary history. Look at git commit-tree to see how to make those, it's easy. The commit id's won't match the authoritative upstream, but the tree id's will.

For binaries, git annex seems to be the most-recommended way to store content that's doesn't fit well with git's source-diffing focus. I haven't used it myself but the design looks ready for production use, it also supports multiple independent `repositories and looks about as convenient as it could reasonably be.

This isn't turnkey, so it doesn't really meet your third, but it nails the rest and what you need is straightforward use of basic tools.