How does my git repository keep getting corrupted?

Question

I am using ubuntu 14.10 (64bit), git version 2.1.0. This is a repository that has been around for several years, and has recently started having issues.

Sometimes I'll change branches, edit a file, and type git status. It will show a bunch of files as "modified" that actually haven't changed at all.

If I type the following:

$ git commit -am "fixed sms message"

I get errors like this:

error: short read No such file or directory
error: globalstatic/images/console/avatar_f.gif: failed to insert intodatabase
error: unable to index file globalstatic/images/console/avatar_f.gif
fatal: updating files failed

If I do nothing other than press the up arrow key and press enter again, I'll get an error about the next file that showed up in git status that never actually changed. I can do this until I've exhausted the list of files in git status that didn't change, and then it will commit correctly.

git fsck --full does report that it has found problems, but doesn't actually fix anything. I just have to keep trying to commit until it works. Here's output of git fsck --full when it's giving me issues:

Checking object directories: 100% (256/256), done.
Checking objects: 100% (120625/120625), done.
dangling tag c7539416829fb0748bc32dda3beb386bac46ea9a
dangling blob 88a0700db2a75e6ea2b14b9a5af15ece63a80805
dangling blob f4c664b0044f3d5efff1148717dc68b940e08574
dangling blob 1bf6185091177fd5a496f5bf031f4d666fec92da
dangling commit fee4bdcc078789a3745aa1311f128a6b61a81736
dangling blob 9f14e68da29d49895b1ea303ed33cb390fc56b76
dangling blob 784a02e974b81f35952fb7c31bf2dcb1a7bfeda8
dangling blob 9e92d6cf395206152123f9a29edb95652114fd34
dangling commit 1294f626dcb76cafa560f65792517655fb8a52ae
dangling blob bceb1adde285e71109723211a1bcb5b0fa126681
dangling tag 19f4be8e7b53465b13359bc4350b5e87c5942560
dangling blob 93f96a3b5e995032a50723af796dab9ae36fb974
dangling blob a25bdfab82fef920935478ee2cefe4dc2e81bbf6
dangling blob af7187350341f3d7795d35cc1f0cee78eb9f9fdf
dangling blob e2a1db9e3d3d438c8b03cb6254ca492e505be6f8

If I run it more than once, I get the same message. Nothing changes.

What is going on?

Answer 1

No idea what's going on (beyond noting the obvious, that something is corrupting your git repo), but I know how I fix things like this - I'm hoping that was an implicit question!

• push your repo somewhere (a local bare repository will be fine)
• rename your old repository to myrepo.broken or similar
• check out a fresh copy of the repo you pushed to the previous place
• Once you are sure you have lost nothing, delete the old repo

I far prefer the above method to futzing with the repo itself. But if you insist (and I'd make a backup first):

• Note the dangling blobs and commits may well be no problem at all - see e.g. here. The short read thing should not happen.
• git gc --prune=now will prune all your dangling blobs and commits.
• Here's a tutorial on maintenance and data recovery.

As far as what's going on is concerned, I've had problems with

• NFS mounted home directories containing git trees when the NFS server is dodgy
• case insensitive filing systems
• power loss / system crash causing FS corruption, or happening half way through commit
• unidentified gremlins in the system

It's easy enough to avoid the first and second of these repeating, and the third is understandable, but the fourth tends to be tricky. Avoid feeding after midnight.