I installed TortoiseGIT using TortoisePink as my SSH client. But I later found out that that was not the correct one to use for my purposes. So, I uninstalled TortoiseGIT an
I will add this here as I didn't want to reinstall TortoiseGit just for this and the other solutions did not help.
If you want to switch TortoiseGit to OpenSSH, simply run this at the command prompt:
tortoisegitproc /command:firststart