OpenSSL RSA: Unable to encrypt/decrypt messages longer than 16 bytes

Question

I\'m working on a simple program that uses OpenSSL to do basic RSA encryption and decryption. It is working fine for small messages (<16 bytes), but fails for anything ov

Answer 1

This is because you do not properly handle out and outl parameters in EVP_SealUpdate(), EVP_SealFinal(), EVP_OpenUpdate() and EVP_OpenFinal().

Each EVP_XxxxUpdate() and EVP_XxxxFinal() call will contribute to the output buffer. So, you are required to keep track of the seal/open process by summing each outl returned and providing the expected buffer each time (start of buffer + already handled bytes).

unsigned char* rsa_seal(...)
{
  ...
  **enc_msg_len = 0;

  EVP_SealUpdate(ctx, encrypt + **enc_msg_len, &outl, msg, (int)msg_len);
  **enc_msg_len += outl;

  EVP_SealFinal(ctx, encrypt + **enc_msg_len, &outl);
  **enc_msg_len += outl;
  ...
}

char* rsa_open(...)
{
  ...
  dec_len = 0;

  EVP_OpenUpdate(ctx, decrypt + dec_len, &outl, enc_msg, (int)*enc_msg_len);
  dec_len += outl;

  EVP_OpenFinal(ctx, decrypt + dec_len, &outl);
  dec_len += outl;
  ...
}

The program was working with 15-bytes buffer because in that case, the EVP_XxxxUpdate() call is returning 0 in outl (not enough data to seal/open a block), hiding the problem in your code logic.

Note: The data is not directly encrypted using the RSA key but using a generated symetric key (AES-128 in your case). This is why the block size is 16 bytes.