I have a GitHub repo that had two branches - master & release.
The release branch contained binary distribution files that were contributing to a very large repo
Each time your HEAD moves, git tracks this in the reflog. If you removed commits, you still have "dangling commits" because they are still referenced by the reflog for ~30 days. This is the safety-net when you delete commits by accident.
You can use the git reflog command remove specific commits, repack, etc.., or just the high level command:
git gc --prune=now