I have been using Git for a while now. The projects that I have been working on have not been too complicated in regards to branches/tags.
The problem is that you need to get back the right binaries:
The last point is not overly important if you keep developing the latest versions of your 2 branches (and accept to rebuilt everything if you checkout an old tag and branch from there).
But still, if after a build you automatically publish those '.o' files into a repository made for managing binaries, that would solve your problem neatly.
A local Nexus repo for instance would be appropriate.