On msysgit, you need first to set the user environment variable HOME (which isn't set by default in Windows).
HOME can be set to any directory you want, you only have to make sure you have the right to write in it (and that the directory exists and a .ssh under it exists too).