Local Setup
I created a public and private SSH key via the ssh-keygen command.
I decided to setup the private key locally first
I wouldn't recommend using the same private key. Both for security reasons and because it might cause other problems. Also, make sure the authorized_keys file contains the public key on the server you want to access, otherwise the whole thing can't work.
I'm assuming you've followed a guide along the lines of this one?
Hope this does something.