So I thought I had finally got everything setup on Windows ... then ran into this issue.
Current setup
URL: ssh://user@host:port/myapp.git
I found out that using ssh.exe from the Git package works every time, as opposed to the ssh that comes with cygwin (the default). Using this exported variable seems to help; it's slower (2x or more) but it's more stable. Take it as another workaround.
$ export GIT_SSH=/cygdrive/c/Program\ Files/Git/bin/ssh.exe
FYI: This version of Msysgit comes with OpenSSH 4.6p1, OpenSSL 0.9.8e. [works] Cygwin's SSH is OpenSSH 5.5p1, OpenSSL 0.9.8n. [doesn't work]