I am running gitlab which uses gitolite on a central repository Ubuntu machine and have several users that connect to this machine after adding their public SSH keys via Git
Check what ssh -Tvvv git@111.222.333.444 returns, in combination with a sshd -d (a debug session of your ssh daemon)
You should see it somehow doesn't find git.pub and git (public and private keys named after the user name) in your ~git/.ssh.
Those public and private keys can also be stored with their default name (id_rsa and id_rsa.pub) on your server (~git/.ssh).