You can both encrypt and decrypt with PrivateKey. PrivateKey infact contains both Private and PublicKey.
Theoretically at least you can encrypt with PublicKey and decrypt with PrivateKey and vice-versa. In VB.net I see the first case works and secondcase throws BadKey error