I tried:
The Git Extension files are stored in your 'My Documents' folder in a "Visual Studio 2012" subfolder. Removing these files will make the toolbar disappear from within MSVC.