Convert a Git folder to a submodule retrospectively?

Question

Quite often it is the case that you\'re writing a project of some kind, and after a while it becomes clear that some component of the project is actually useful as a standal

Answer 1

Status quo

Let's assume we have a repository called repo-old which contains a subdirectory sub that we would like to convert into a submodule with its own repo repo-sub.

It is further intended that the original repo repo-old should be converted into a modified repo repo-new where all commits touching the previously existing subdirectory sub shall now point to the corresponding commits of our extracted submodule repo repo-sub.

Let's change

It is possible to achieve this with the help of git filter-branch in a two step process:

1. Subdirectory extraction from repo-old to repo-sub (already mentioned in the accepted answer)
2. Subdirectory replacement from repo-old to repo-new (with proper commit mapping)

Remark: I know that this question is old and it has already been mentioned that git filter-branch is kind of deprecated and might be dangerous. But on the other hand it might help others with personal repositories that are easy to validate after conversion. So be warned! And please let me know if there is any other tool that does the same thing without being deprecated and is safe to use!

I'll explain how I realized both steps on linux with git version 2.26.2 below. Older versions might work to some extend but that needs to be tested.

For the sake of simplicity I will restrict myself to the case where there is just a master branch and a origin remote in the original repo repo-old. Also be warned that I rely on temporary git tags with the prefix temp_ which are going to be removed in the process. So if there are already tags named similarily you might want to adjust the prefix below. And finally please be aware that I have not extensively tested this and there might be corner cases where the recipe fails. So please backup everything before proceeding!

The following bash snippets can be concatenated into one big script which should then be executed in the same folder where the repo repo-org lives. It is not recommended to copy and paste everything directly into a command window (even though I have tested this successfully)!

0. Preparation

Variables

# Root directory where repo-org lives
# and a temporary location for git filter-branch
root="$PWD"
temp='/dev/shm/tmp'

# The old repository and the subdirectory we'd like to extract
repo_old="$root/repo-old"
repo_old_directory='sub'

# The new submodule repository, its url
# and a hash map folder which will be populated
# and later used in the filter script below
repo_sub="$root/repo-sub"
repo_sub_url='https://github.com/somewhere/repo-sub.git'
repo_sub_hashmap="$root/repo-sub.map"

# The new modified repository, its url
# and a filter script which is created as heredoc below
repo_new="$root/repo-new"
repo_new_url='https://github.com/somewhere/repo-new.git'
repo_new_filter="$root/repo-new.sh"

Filter script

# The index filter script which converts our subdirectory into a submodule
cat << EOF > "$repo_new_filter"
#!/bin/bash

# Submodule hash map function
sub ()
{
    local old_commit=\$(git rev-list -1 \$1 -- '$repo_old_directory')

    if [ ! -z "\$old_commit" ]
    then
        echo \$(cat "$repo_sub_hashmap/\$old_commit")
    fi
}

# Submodule config
SUB_COMMIT=\$(sub \$GIT_COMMIT)
SUB_DIR='$repo_old_directory'
SUB_URL='$repo_sub_url'

# Submodule replacement
if [ ! -z "\$SUB_COMMIT" ]
then
    touch '.gitmodules'
    git config --file='.gitmodules' "submodule.\$SUB_DIR.path" "\$SUB_DIR"
    git config --file='.gitmodules' "submodule.\$SUB_DIR.url" "\$SUB_URL"
    git config --file='.gitmodules' "submodule.\$SUB_DIR.branch" 'master'
    git add '.gitmodules'

    git rm --cached -qrf "\$SUB_DIR"
    git update-index --add --cacheinfo 160000 \$SUB_COMMIT "\$SUB_DIR"
fi
EOF
chmod +x "$repo_new_filter"

1. Subdirectory extraction

cd "$root"

# Create a new clone for our new submodule repo
git clone "$repo_old" "$repo_sub"

# Enter the new submodule repo
cd "$repo_sub"

# Remove the old origin remote
git remote remove origin

# Loop over all commits and create temporary tags
for commit in $(git rev-list --all)
do
    git tag "temp_$commit" $commit
done

# Extract the subdirectory and slice commits
mkdir -p "$temp"
git filter-branch --subdirectory-filter "$repo_old_directory" \
                  --tag-name-filter 'cat' \
                  --prune-empty --force -d "$temp" -- --all

# Populate hash map folder from our previously created tag names
mkdir -p "$repo_sub_hashmap"
for tag in $(git tag | grep "^temp_")
do
    old_commit=${tag#'temp_'}
    sub_commit=$(git rev-list -1 $tag)

    echo $sub_commit > "$repo_sub_hashmap/$old_commit"
done
git tag | grep "^temp_" | xargs -d '\n' git tag -d 2>&1 > /dev/null

# Add the new url for this repository (and e.g. push)
git remote add origin "$repo_sub_url"
# git push -u origin master

2. Subdirectory replacement

cd "$root"

# Create a clone for our modified repo
git clone "$repo_old" "$repo_new"

# Enter the new modified repo
cd "$repo_new"

# Remove the old origin remote
git remote remove origin

# Replace the subdirectory and map all sliced submodule commits using
# the filter script from above
mkdir -p "$temp"
git filter-branch --index-filter "$repo_new_filter" \
                  --tag-name-filter 'cat' --force -d "$temp" -- --all

# Add the new url for this repository (and e.g. push)
git remote add origin "$repo_new_url"
# git push -u origin master

# Cleanup (commented for safety reasons)
# rm -rf "$repo_sub_hashmap"
# rm -f "$repo_new_filter"

Remark: If the newly created repo repo-new hangs during git submodule update --init then try to re-clone the repository recursively once instead:

cd "$root"

# Clone the new modified repo recursively
git clone --recursive "$repo_new" "$repo_new-tmp"

# Now use the newly cloned one
mv "$repo_new" "$repo_new-bak"
mv "$repo_new-tmp" "$repo_new"

# Cleanup (commented for safety reasons)
# rm -rf "$repo_new-bak"