I know that RSACryptoServiceProvider can encrypt with the public key, then it can be decrypted with the private key.
Is it possible to encrypt with the private key a
Fortunately no. You can however sign with the private key and verify the signature with the public key.
While the math involve makes sense when the key roles are reversed (and this is how signatures work), encrypting for privacy doesn't make much sense when the decryption key is well know and public.