After successfully cloning my repo from heroku and added another remote
1/ git clone git@heroku.com:[APP].git
2/ git remote add bitbucket ssh://git@bitbucket
I had this happen when I was trying to use a deployment key because that is exactly what I wanted.
I could connect via ssh -T git@bitbucket.org and it would tell me I had access to read the repository I wanted, but git clone would fail.
Clearing out ~/.ssh/known_hosts, generating a new key via ssh-keygen, adding that new key to bitbucket, and retrying fixed it for me.