I have this error while pushing my project to tfs GIT.
fatal: Out of memory, malloc failed (tried to allocate 889192448 bytes)
In my experience this can happen for a few reasons:
.gitignore file in that new repo and then copy my files over there. Not a good solution but it was my best solution until I can identify a way to completely remove it from all history.I'm not saying these are the only possibilities. But in my experience with at least a few hundred repositories it was one of these two things every time. Usually #1.
To see which files are the largest in your repository execute this command: ls -lS