I created a repository for my home folder. This is my .gitignore file:
.gitignore
!.vim/plugin/* !.vim/doc/* .* * .viminfo .vim/.netrwhist .vim-fuf-data/fi
Try moving the negated lines (those starting with !) to the bottom of the .gitignore file.
!
See the Examples section here