My issue was that I was attempting to use my github username. Apparently when using Github (or is it a Git thing?), if you use key-based authentication, you need to set your username to git.
I'm not sure why you even need a username at all - perhaps someone more knowledgeable can explain that?