Apparently git uses only the first key provided, if that fails, git clone fails. My ssh config had a wildcard option with a key matching before the git key and therefore it did not work. After moving the git configuration to the top of .sss/config all is fine.