Encryption and decryption error 0x0407106B using OpenSSL

Question

I\'m writing a routine in C that reads a base64 string with the public key and proceeds to encrypt a string. I also test the same string\'s decryption but I\'m getting error

Answer 1

The problem is that you're trying to decrypt the base64 encoded result. You should try to decrypt the result of the encryption.

That is, instead of:

int resultDecrypt = RSA_private_decrypt( RSA_size(privKey), retencrypted, decrypted, privKey, PADDING);

You should call:

int resultDecrypt = RSA_private_decrypt( RSA_size(privKey), encrypted, decrypted, privKey, PADDING);

Also, there is a problem in the encryption call:

int resultEncrypt = RSA_public_encrypt(PADDING, str, encrypted, pubKey, PADDING);

Why are you passing PADDING as flen? This should be the length of the string (i.e. 4 or 5, depending on whether you want to encrypt the null character).

If you want to write the encrypted string as ASCII (encoded using base64), that's fine. But you have to decode it back before you decrypt it.