Submodules, subtrees or something else for dependencies in Git?

Question

I have a situation with a larger project that has lots of modules/libraries with their respective repos. Most of these modules are dependencies of other modules which are than dependencies of a project. And now it has come to the point where the main project has several sub-projects and many of modules are being shared. Some dependencies are more than 3-4 levels deep.

I have read that it is possible to update/pull submodules inside of a project, but that works only for 1st level of submodules. Let's say that those submodules have their own submodules (2nd level) and that some 1st level submodules share the same 2nd level submodules. Also, 2nd lvl submodules have their submodules (lvl3), etc. Now what I should do is to firstly push changes made in 3rd level, than update submodules in 2nd level modules and push those, now I can go to 1st level, update and push, and finally update my project submodules and push those.

This is now not only more work, but it still doesn't solve my problem why I need something like this and that is to be able to easily push and pull multiple repositories when changes were being made to those that dependent on each other. It can easily happen that someone in a team pushes changes in 4 of 5 repos, and when other members pull all except this one production line stops until error has been found.

What can I do about this? Maybe some advices about workflow, has anyone else encountered this problem or is there some feature in Git that solves this.

Answer 1

I would recommend the repo tool that android uses. It is generic enough to work with any git hosting environment, and doesn't require super-project commits to update sub-projects like submodules does.

First, install the client as described here: https://source.android.com/source/downloading.html#installing-repo

Then create a manifest repository. A manifest is an xml file that describes git repository locations and paths they should be checked out to. Like this:

mkdir manifests
cd manifests
git init

Create a manifest file default.xml:

<?xml version="1.0" encoding="UTF-8"?>
<manifest>
  <remote name="github" fetch="ssh://git@github.com" />
  <default remote="github" revision="master" />
  <project name="git/git.git" path="git" />
  <project name="libgit2/libgit2.git" path="vendor/libgit2" />
</manifest>

Then add, commit the manifest, and push somewhere:

git add default.xml
git commit -m "My first try at a manifest file"
git push git@github.com:myusername/manifests.git master

Now you are ready to use the repo command.

mkdir myproject
cd myproject
repo init -u git@github.com:myusername/manifests.git
repo sync -j2

Your git repositories will be cloned. You can now work in each one like normal. After you push to any of the projects, all anyone else needs to do is a repo sync and they will be updated to the latest revision (also see repo start).

Caveats

You may have to reorganize your project. Typically you might have other modules as sub-directories (myproject/vendor/dependency). While you can still maintain this layout using repo, it will cause a git repository to be checked out with another repo. With .gitignore trickery it might be workable, but I would recommend reorganizing your project so repositories do not need to be checked out within each other.

A short explanation on manifest files

See https://gerrit.googlesource.com/git-repo/+/master/docs/manifest-format.txt for a complete explanation of each item in the xml file.

See https://source.android.com/source/using-repo.html for a simple command reference. repo help is also very useful. Note: you should ignore repo upload unless you are using Gerrit.

<remote name="github" fetch="ssh://git@github.com" />

This is just like adding a remote in git. It means we can refer to the url with a given name.

<default remote="github" revision="master" />

The default element specifies default options for the projects. This is equivalent of adding the remote and revision items on every project. This just saves some typing.

<project name="git/git.git" path="git" />

This is where the real work is happening. repo sync will take the name and append it to the remote using a slash. In this case the remote is the default github, so it will get the url ssh://git@github.com/git/git.git. It will checkout the project to the path git at the specified revision (in this case the default is master). Subsequent repo syncs will checkout the latest revision (in the case of a branch).