Restore git submodules from .gitmodules

Question

I have a folder, which was a git repo. It contains some files and .gitmodules file. Now, when I do git init and then git submodule init, the latter

Answer 1

Extending excellent @Mark Longair's answer to add submodule respecting branch and repo name.

#!/bin/sh

set -e

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
    while read path_key path
    do
        name=$(echo $path_key | sed 's/\submodule\.\(.*\)\.path/\1/')
        url_key=$(echo $path_key | sed 's/\.path/.url/')
        branch_key=$(echo $path_key | sed 's/\.path/.branch/')
        url=$(git config -f .gitmodules --get "$url_key")
        branch=$(git config -f .gitmodules --get "$branch_key" || echo "master")
        git submodule add -b $branch --name $name $url $path || continue
    done

Answer 2

I know its been a while, but I want to share this version that calls git config only once, doesn't requires a script and also handles branches:

git config -f .gitmodules --get-regexp '^submodule\.' | perl -lane'
$conf{$F[0]} = $F[1]}{
@mods = map {s,\.path$,,; $_} grep {/\.path$/} keys(%conf);
sub expand{$i = shift; map {$conf{$i . $_}} qw(.path .url .branch)}
for $i (@mods){
    ($path, $url, $branch) = expand($i);
    print(qq{rm -rf $path});
    print(qq{git submodule add -b $branch $url $path});
}
'

The only side effect is the output of the commands, nothing gets executed, so you can audit before committing to them.

This works with a simple copy and paste at the console, but should be trivial to put in a shell script.

example output:

rm -rf third-party/dht
git submodule add -b post-0.25-transmission https://github.com/transmission/dht third-party/dht
rm -rf third-party/libutp
git submodule add -b post-3.3-transmission https://github.com/transmission/libutp third-party/libutp
rm -rf third-party/libb64
git submodule add -b post-1.2.1-transmission https://github.com/transmission/libb64 third-party/libb64
rm -rf third-party/libnatpmp
git submodule add -b post-20151025-transmission https://github.com/transmission/libnatpmp third-party/libnatpmp
rm -rf third-party/miniupnpc
git submodule add -b post-2.0.20170509-transmission https://github.com/transmission/miniupnpc third-party/miniupnpc
rm -rf third-party/libevent
git submodule add -b post-2.0.22-transmission https://github.com/transmission/libevent third-party/libevent

Answer 3

Expanding on @Mark Longair's answer, I wrote a bash script to automate steps 2 & 3 of the following process:

1. Clone a 'boilerplate' repo to begin a new project
2. Remove the .git folder and re-initialize as a new repo
3. Re-initialize the submodules, prompting for input before deleting folders

---

#!/bin/bash

set -e
rm -rf .git
git init

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' > tempfile

while read -u 3 path_key path
do
    url_key=$(echo $path_key | sed 's/\.path/.url/')
    url=$(git config -f .gitmodules --get "$url_key")

    read -p "Are you sure you want to delete $path and re-initialize as a new submodule? " yn
    case $yn in
        [Yy]* ) rm -rf $path; git submodule add $url $path; echo "$path has been initialized";;
        [Nn]* ) exit;;
        * ) echo "Please answer yes or no.";;
    esac

done 3<tempfile

rm tempfile

Note: the submodules will be checked out at the tip of their master branch instead of the same commit as the boilerplate repo, so you'll need to do that manually.

Piping the output from git config into the read loop was causing problems with the prompt for input, so it outputs it to a temp file instead. Any improvements on my first bash script would be very welcome :)

---

Big thanks to Mark, https://stackoverflow.com/a/226724/193494, bash: nested interactive read within a loop that's also using read, and tnettenba @ chat.freenode.net for helping me arrive at this solution!

Answer 4

git submodule init only considers submodules that already are in the index (i.e. "staged") for initialization. I would write a short script that parses .gitmodules, and for each url and path pair runs:

git submodule add <url> <path>

For example, you could use the following script:

#!/bin/sh

set -e

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
    while read path_key path
    do
        url_key=$(echo $path_key | sed 's/\.path/.url/')
        url=$(git config -f .gitmodules --get "$url_key")
        git submodule add $url $path
    done

This is based on how the git-submodule.sh script itself parses the .gitmodules file.

Answer 5

For zsh users, try my function which has DRY_RUN=1 support to see what commands will be ran and only uses git to parse the file instead of sed.

gsub_file() {(
  set -eu

  cd "$(git rev-parse --show-toplevel)"

  submodule_paths=(
    "${(@f)$(git config --file ./.gitmodules --get-regexp "path" | awk '{ print $2 }')}"
  )
  submodule_urls=(
    "${(@f)$(git config --file ./.gitmodules --get-regexp "url" | awk '{ print $2 }')}"
  )
  submodule_branches=(
    "${(@f)$(git config --file ./.gitmodules --get-regexp "branch" | awk '{ print $2 }')}"
  )

  sh_c() {
    echo + "$*"
    if [ "${DRY_RUN-}" ]; then
      return
    fi
    eval "$@"
  }

  for (( i=1; i <= ${#submodule_paths[@]}; i++ )) do
    p="${submodule_paths[$i]}"
    if [ -d "$p" ]; then
      continue
    fi

    url="${submodule_urls[$i]}"
    unset b
    if [ "${submodule_branches[$i]-}" ]; then
      b="-b ${submodule_branches[$i]}" 
    fi
    sh_c git submodule add "${b-}" "$url" "$p"
  done
)}

Answer 6

An updated version of the script by @mark-longair . This one also supports branches, handles the case where some of the submodules already exist in .git/config, and when necessary backs up existing directories of the same name as the submodule paths.

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
    while read path_key path
    do
        url_key=$(echo $path_key | sed 's/\.path/.url/');
        branch_key=$(echo $path_key | sed 's/\.path/.branch/');
        # If the url_key doesn't yet exist then backup up the existing
        # directory if necessary and add the submodule
        if [ ! $(git config --get "$url_key") ]; then
            if [ -d "$path" ] && [ ! $(git config --get "$url_key") ]; then
                mv "$path" "$path""_backup_""$(date +'%Y%m%d%H%M%S')";
            fi;
            url=$(git config -f .gitmodules --get "$url_key");
            # If a branch is specified then use that one, otherwise
            # default to master
            branch=$(git config -f .gitmodules --get "$branch_key");
            if [ ! "$branch" ]; then branch="master"; fi;
            git submodule add -f -b "$branch" "$url" "$path";
        fi;
    done;

# In case the submodule exists in .git/config but the url is out of date

git submodule sync;

# Now actually pull all the modules. I used to use this...
#
# git submodule update --init --remote --force --recursive
# ...but the submodules would check out in detached HEAD state and I 
# didn't like that, so now I do this...

git submodule foreach --recursive 'git checkout $(git config -f $toplevel/.gitmodules submodule.$name.branch || echo master)';