git pull multiple remotes in parallel

Question

I have a repo with thousands of remotes, and I\'d like to pull from thousands of remotes at the same time, ideally I can specify a maximum number to do at the same time.

Answer 1

Starting from Git 2.24 it it now possible with [--jobs] option.

Some examples:

Fetching 3 remotes, 2 remotes will be fetched in parallel:

git fetch -j2 --multiple remote1 remote2 remote3

Fetching all remotes, 5 remotes will be fetched in parallel:

git fetch -jobs=5 --all

If you have thousands of remotes and you don't want to download all of them and they form some logical groups. Instead of specifying them in command line (with --multiple) options You can also define remote groups like this in .git/config

[remotes]
    group1 = remote1 remote2 origin
    group2 = remote55 remote66

And then use this group in fetch command.

This command: git fetch --multiple -j4 group1 group2 remote10 fetches remote1 remote2 origin remote55 remote66 remote10 remotes and 4 fetches are done in parallel.