Final update and fix: The solution here turned out to be a combination of two things: using Windows Git rather than Cygwin Git as Graham Borland suggest
You might have more luck running a native Windows client such as msysGit, rather than trying to do it inside Cygwin.
Maybe temporarily adding a swap file bigger than life and going for a few cups of coffee elsewhere will help?
I had the same problem, tried the solutions mentioned so far without success. But my problems with git gc began after I added big image files to repo, so I created .gitattributes file and turned off delta compression for those big files:
*.tga -delta
*.psd -delta
It worked.
Some other config options that you might want to try restricting to lower than default values include:
pack.packSizeLimit
core.packedGitWindowSize
core.packedGitLimit
... all of which are documented in the git config documentation. It's particularly worth checking in each case what units are understood, which I've made mistakes with in the past.
The only thing that help to avoid this error on shared Linux hosting was to add
[pack]
packSizeLimit = 64m
threads = 1
to
.gitconfig
Most important was "threads = 1"