I have this error while pushing my project to tfs GIT.
fatal: Out of memory, malloc failed (tried to allocate 889192448 bytes)
After long hours of search and not getting the solution, in frustration I had allocated "999999999" to postbuffer and it worked wonders. ;)
[http] postbuffer = 9999999999
Try this it may work for you also. Good Luck. :)