I know that RSACryptoServiceProvider can encrypt with the public key, then it can be decrypted with the private key.
Is it possible to encrypt with the private key a
You can do both; encrypt with private and decypt with public, OR, encrypt with public and decrypt with private. You can not encrypt then decrypt with private key only, nor can you do the same with public keys alone.
Remus nailed it; encrypting with the private key doesn't make much sense when the decryption key is well know and public.
Also, you can derive the public key from the private key, but not vice versa.