Note: I\'m not a newb, and I\'ve done this a gazillion times, but for some reason today it decided not to work.
I keep getting the Permission denied (publickey).>
After creating a config file (~/.ssh/config) it worked. This is what I had to put in it:
Host github.com
User git
Port 22
Hostname github.com
IdentityFile ~/.ssh/id_rsa
TCPKeepAlive yes
IdentitiesOnly yes
Thanks to @VonC for leading me to there in the comments.
I don't get why I never needed this before though.