I cloned a project from github with git clone --mirror. That left me with a repository with a packed-refs file, a .pack and an .idx file.
There is a workaround that worked for me and might work for some of you:
Basically, Delete all the tags locally (removes it from packed-refs) and then fetch it again.
git tag | xargs git tag -d && git fetch --tags