i am using a certificate generated by makecert which has both private and public key. The java side uses this public key to encrypt the data and .net decrypts it back.
I had the same problem with a self-signed cert, the issue was that I was generating the cert with the switch -sky signature
instead of -sky exchange
(you use signature for signing and exchange for encryption/decryption)
Here is my full command to makecert that works:
makecert -r -pe -a sha1 -n "CN=MyName" -ss my -sr CurrentUser -sky exchange