How do I synchronise two remote Git repositories?

Question

I have two repository urls, and I want to synchronise them such that they both contain the same thing. In Mercurial, what I\'m trying to do would be:

hg pull         


        

Answer 1

As others I came here because this SO-question has so right header for this particular problem.
This was few years ago. I studied available answers, but nothing solved my situation.

I took a step further and created several solutions.
And my final solution posted here is for Git repositories only (sorry, I do not use Mercurial right now as it was asked in the body of the question).

I had rather difficult conditions at the start than others here.

• I can't use Git-hooks because I always have a limited access to one or the other remote repository. I consider them a burden dependency.
• Teams on both remote sides are big. They produce huge amount of commits and commits at the same Git-branches too.
• We need 24/7 and fast synchronizing solution. This drastically decreases amount of possible Git-conflicts and converts them to simple local Git-merges.
• It was important to have a capability to do Git-merges from any remote repository for any branches and by any member of both teams.
• Sometimes one remote repository needs to be completely replaced by an empty new repository at another location. And I didn't want to do initial repository filling manually.
• CI/CD managing branches should be migrated quite arbitrary.
• Not all present syncing methods recognize subtle changes right. For example, moving back in history.
• And some methods have a threatening tendency to occasionally delete branches or the entire repository.

And just like the author of this SO question, I also wanted to have

• no user interaction, at all. That is, everything should be fully automated.
• all changes in both repos for all branches should be migrated.

I quickly wrote some bash script and quickly realized that with Git this would not be enough.
The main problem is conflict solving.
Long story short, you can only solve conflicts by some conventions or by asking to repeat a conflicting merge (commit).

So, my script became a compiled application.

Later arrived some requirements.

• It was very desirable to remain some branches invisible to the other side repository.
• We cannot allow arbitrary updating for all branches. Some branches should be updated in a limited way on one or the other side.
• Some Git-servers like to create util- (garbage) branches and tags. No one wants them to be migrated.
• More importantly, some Git servers, like Gitlab or Bitbucket, like to completely block some branches and tags. This is a catastrophe.

The other thing is that people tend to forget the fact that they have two synchronized remote Git-repositories. They do it really fast and do not want to remember any rules.
Some teams may have some level of employee rotation.
A big application requires more documentation and support. And etc.

Finally, I threw out some complicated conflict solving and state analysis.
A remaining subset of logic was pretty simple, which allowed me to convert my app into a bunch of bash and gAWK scripts.

And as a result of all, my tool only synchronizes branches with defined prefixes.
You can say, let's synchronize branches that begin with the @.
Or branches that start with my-company- and your-company-.

Of course, my tool has a little learning curve.
But it is quite mature and gives a possibility to forget about some sync problems, at all.

Actually I forgot about my tool and came here two years later, because I've implemented a wish list.

My latest tool is here - git-repo-sync. I hope this will help someone else.

Answer 2

Here's a tested solution for the issue: http://www.tikalk.com/devops/sync-remote-repositories/

The commands to run:

#!/bin/bash

# REPO_NAME=<repo>.git
# ORIGIN_URL=git@<host>:<project>/$REPO_NAME
# REPO1_URL=git@<host>:<project>/$REPO_NAME

rm -rf $REPO_NAME
git clone --bare $ORIGIN_URL
cd $REPO_NAME
git remote add --mirror=fetch repo1 $REPO1_URL
git fetch origin --tags ; git fetch repo1 --tags
git push origin --all ; git push origin --tags
git push repo1 --all ; git push repo1 --tags

Answer 3

You might not have seen that the fetch did in fact work when you used git clone --mirror --bare, because by default git does not list it's remote branches. You can list them with git branch -a.

I don't quite have the syntax worked out for unnamed remotes, but you could automatically add remotes based on some scheme from the url... in any case, it'll probably work best if you choose some unique and consistent name for each repo, so you can know what changes came from where

However, you could try something like this:

git clone --bare --mirror --origin thing1 {repo1} repo.git
cd repo.git
git fetch thing2 --mirror
git push thing1 --mirror
git push thing2 --mirror

After this was done, thing1 would have all of thing2's branches available to merge at any time, as remote branches. You can list the remote branches with git branch -a.

On github or bitbucket, you will not be able to see these remote branches via the web interfaces, however you can see them if you clone with --mirror, so they do exist.

Answer 4

Try git reset --hard HEAD after git fetch. However, I'm not sure I understand exactly what your goal is. You will need to cd into the separate repository directories before running the fetch, reset, and push commands.

Answer 5

Git branches do not have "heads" in the Mercurial sense. There is only one thing called HEAD, and it's effectively a symlink to the commit you currently have checked out. In the case of hosted repositories like GitHub, there is no commit checked out—there's just the repository history itself. (Called a "bare" repo.)

The reason for this difference is that Git branch names are completely arbitrary; they don't have to match between copies of a repository, and you can create and destroy them on a whim.[1] Git branches are like Python variable names, which can be shuffled around and stuck to any value as you like; Mercurial branches are like C variables, which refer to fixed preallocated memory locations you then fill with data.

So when you pull in Mercurial, you have two histories for the same branch, because the branch name is a fixed meaningful thing in both repositories. The leaf of each history is a "head", and you'd normally merge them to create a single head.

But in Git, fetching a remote branch doesn't actually affect your branch at all. If you fetch the master branch from origin, it just goes into a branch called origin/master.[2] git pull origin master is just thin sugar for two steps: fetching the remote branch into origin/master, and then merging that other branch into your current branch. But they don't have to have the same name; your branch could be called development or trunk or whatever else. You can pull or merge any other branch into it, and you can push it to any other branch. Git doesn't care.

Which brings me back to your problem: you can't push a "second" branch head to a remote Git repository, because the concept doesn't exist. You could push to branches with mangled names (bitbucket_master?), but as far as I'm aware, you can't update a remote's remotes remotely.

I don't think your plan makes a lot of sense, though, since with unmerged branches exposed to both repositories, you'd either have to merge them both, or you'd merge one and then mirror it on top of the other... in which case you left the second repository in a useless state for no reason.

Is there a reason you can't just do this:

1. Pick a repository to be canonical—I assume BitBucket. Clone it. It becomes origin.

2. Add the other repository as a remote called, say, github.

3. Have a simple script periodically fetch both remotes and attempt to merge the github branch(es) into the origin branches. If the merge fails, abort and send you an email or whatever. If the merge is trivial, push the result to both remotes.

Of course, if you just do all your work on feature branches, this all becomes much less of a problem. :)

---

[1] It gets even better: you can merge together branches from different repositories that have no history whatsoever in common. I've done this to consolidate projects that were started separatedly; they used different directory structures, so it works fine. GitHub uses a similar trick for its Pages feature: the history of your Pages is stored in a branch called gh-pages that lives in the same repository but has absolutely no history in common with the rest of your project.

[2] This is a white lie. The branch is still called master, but it belongs to the remote called origin, and the slash is syntax for referring to it. The distinction can matter because Git has no qualms about slashes in branch names, so you could have a local branch named origin/master, and that would shadow the remote branch.

Answer 6

For something similar I use this simple code trigerred by webhook in both repositories to sync GitLab and Bitbucket master branch:

git pull origin master
git pull gitlab master
git push origin master
git push gitlab master

It propably is not what you need in question, but it could be helpful for somebody else who needs to sync just one branch.