I have this error while pushing my project to tfs GIT.
fatal: Out of memory, malloc failed (tried to allocate 889192448 bytes)
I had the same problem on windows, exept with checkout another branch. Finally, I couldn't either pull or fetch my project beacause I got out of memory error ("try to allocate..." etc). I did the trick with increasing git pack and core limits but it didn't work for me. Finally, I deleted git from computer and install again. Problem was solved.