Proper Git workflow for combined OS and Private code?

Question

I have a closed source project that is built on my open source framework. I want to know how I should structure my workflow. Below is my best guess using git with submodules

Answer 1

Make the public repo a submodule inside the private one. When pushing, remember you have to push them both. Also remember to check in the submodule itself in the private repo, so it tracks what revisions of the submodule it is using.

Answer 2

There's two approach here:

1. You could use branch's of the same git repo. In your private repo create a branch with a reference to your public repo and handle both like that.

2. If the components using in your private project are sub-project of your public stuff, then you should use submodules. The handling of submodule is in a kind-of early stage on git at version 1.6.6, but could be useful as your using subproject.

What is seems to me you can't loose if which project tribute to each project, so if you have that clear, then no matter what you choose it'll work !!!!!!. Besides git is easy.

Answer 3

You can have a 'public' and 'private' branch in your local repository. When you push, each branch gets pushed to a separate remote repository (look up the 'git push' syntax). Then, you can freely merge from public to private.

I'm sure there's a way you could merge selected changes from private to public, too, though I'd have to look it up.

Answer 4

git submodules allows you to define a configuration (see this question), that is a reference to one commit of another component (in another repo).

You can develop both codes (your and the submodules) within the same repo, but when you are talking about multiple private directories within your public code, that calls for a subtree merge strategy.
It will allow you to consider your directories (the private and public ones) as one natural working tree.

And to better manage the push and pull of parts of your global repo to a private one, I would recommend the git subtree script tool.

Answer 5

To summarize, I recommend this workflow:

1. keep it simple; have one working copy for each repository (don't use git submodules)
2. use your language's tools to package up your framework
3. setup scripts or light tooling to make context switching fast or automatic

I've used git submodules in the past. I don't think they are a good fit for your use case. The big downsides that jump out at me are:

• It helps to eat your own dog food when you build (or extract) a framework. Do you expect your framework users to also setup git submodules when they use your framework? I'm guessing not.
• There is some risk of accidentally publishing your private source code into your open source framework.
• Git submodules have improved quite a bit in the last year or so, but they still are relatively less well understood. Even competent gitters may struggle with submodules.

Here is one sub-question that I will admit is not so clear cut: "Which workflow makes it easier to bounce back and forth between the OSS framework and the private project?"

• There is a certain allure to using submodules and having both projects in one tree. This will speed you up perhaps in your text editing, but probably will slow you down (or cause more mistakes than usual) when it comes to committing and pushing.

• There is a certain allure to having the projects separated. The context switch (from one text editor window to another) may help remind you that the OSS project is for public consumption. For example, it may help discipline you to not to break backwards compatibility and to keep a good changelog. Committing and pushing will be easy relative to the submodule alternative.

One you have decided on your working copies, you'll want to figure out your day to day workflow. It will depend on your language of course. (In Ruby, for example, you might package up your OSS framework as a gem, build it, then have your private code depend on it.) Whatever you pick, setup some scripts (or editor shortcuts perhaps) to help you build your libraries (or packages) quickly, perhaps even automatically when files change, so that you can bounce between your framework and project effortlessly.

Answer 6

I recommend not to use git submodules, but 2 different repositories that are not connected on github.

You could build the relationship between them using symlinks on the checked out copies, which is basic and simple. The symlinks only have to be created once per location (production, development, coworkers).

The advantage is that nobody has to do the extra effort to learn and maintain git submodules, and you avoid the risk and complexity it brings.

It could be done by keeping a working copy of the os and of the private git repo somewhere on your local machine:

/repos/myproject-os
/repos/myproject-priv

Then you could create create your directory structure where the project actually will live and be worked on somewhere else on this machine (not inside the /repos/ tree) and create symblinks for the subdirectories you use:

ln -s /repos/myproject-os/dir1 /wrk/myproject/base/dir1
ln -s /repos/myproject-os/dir2 /wrk/myproject/base/dir2
ln -s /repos/myproject-priv/dir1 /wrk/myproject/base/dir3
ln -s /repos/myproject-priv/dir2 /wrk/myproject/base/someother/dir4
mkdir /wrk/myproject/base/config
mkdir /wrk/myproject/base/tmp

That way you have the repository structure always clean and can mix and arrange the directories from both repositories the way you want them, and you have also a space for local configs or temp files that do not go into the repositories.

You would do the git commits and everything from the /repos/ tree and your project would run and you would edit the files from the /wrk/ tree. Please note that the .git diretory where the git data lives would not be available form the /wrk/ tree, because you only link to subdirectories (or possibly single files from the root directory).

Part2: You say you want to make sure that you do not accidently push private code into the public repository. You could set up an additional git repository between your working OS repository and the github repository, let's say you put it into /repos/gatekeeper, then your tree looks like this:

/repos/gatekeeper/myproject-os
/repos/myproject-os
/repos/myproject-priv

Every time you push from /repos/myproject-os it goes to /repos/gatekeeper/myproject-os. But from /repos/myproject-priv you push directly to your private github repo.

That way you have the same workflow in both /repos/myproject-os and /repos/myproject-priv and you don't need to worry so much. From time to time when you want to push your changes to the real OS codebase, you go to /repos/gatekeeper/myproject-os and push from there to github.

You could do additional code review before that and look at the diffs so you are sure that only that what you really want goes public.

If you want additional security the /repos/gatekeeper/myproject-os could also be on a different machine or even different location.