Android 8.0: IllegalBlocksizeException when using RSA/ECB/OAEPWithSHA-512AndMGF1Padding

Question

I usually find answers for most of our issues here but this time I need to ask :-).

We have encountered a problem with RSA encryption / decryption in one of our Apps running on Android 8.0 (API level 26).

We've been using RSA with "RSA/ECB/OAEPWithSHA-256AndMGF1Padding", which works fine on all versions up to Android 7.1. The same code running on Android 8.0 throws an IllegalBlocksizeException when calling Cipher.doFinal().

Here is the code to reproduce the issue:

private KeyStore mKeyStore;

private static final String KEY_ALIAS = "MyKey";
void testEncryption() throws NoSuchProviderException, NoSuchAlgorithmException, InvalidAlgorithmParameterException, KeyStoreException, IOException, CertificateException, BadPaddingException, IllegalBlockSizeException, InvalidKeyException, UnrecoverableEntryException, NoSuchPaddingException {

    mKeyStore = KeyStore.getInstance("AndroidKeyStore");
    mKeyStore.load(null);

    // Generate Key Pair -------------------------------------
    KeyPairGenerator kpg = KeyPairGenerator.getInstance(KeyProperties.KEY_ALGORITHM_RSA, "AndroidKeyStore");
    kpg.initialize(new KeyGenParameterSpec.Builder(
            KEY_ALIAS,
            KeyProperties.PURPOSE_ENCRYPT | KeyProperties.PURPOSE_DECRYPT)
            .setDigests(KeyProperties.DIGEST_SHA256, KeyProperties.DIGEST_SHA512)
            .setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_RSA_OAEP)
            .setKeySize(2048)
            .build());
    KeyPair kp = kpg.generateKeyPair();

    // Encrypt -----------------------------------------------
    KeyStore.PrivateKeyEntry privateKeyEntry = (KeyStore.PrivateKeyEntry)mKeyStore.getEntry(KEY_ALIAS, null);
    PublicKey publicKey = (PublicKey) privateKeyEntry.getCertificate().getPublicKey();
    Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPWithSHA-256AndMGF1Padding");
    cipher.init(Cipher.ENCRYPT_MODE, publicKey);
    String x = "It doesn't have to be perfect, it's just for demonstration.";

    byte [] vals = cipher.doFinal(x.getBytes("UTF-8"));

    byte[] encryptedBytes = Base64.encode(vals, Base64.DEFAULT);
    String encryptedText = new String(encryptedBytes, "UTF-8");


    // Decrypt -----------------------------------------------
    PrivateKey privateKey = privateKeyEntry.getPrivateKey();

    Cipher output = Cipher.getInstance("RSA/ECB/OAEPWithSHA-256AndMGF1Padding");
    output.init(Cipher.DECRYPT_MODE, privateKey/*, spec */);

    byte[] bxx = Base64.decode(encryptedText, Base64.DEFAULT);
    byte[] bytes = output.doFinal(bxx);  // <= throws IllegalBlocksizeException

    String finalText = new String(bytes, 0, bytes.length, "UTF-8");
}

I tried other padding algorithms too. "RSA/ECB/OAEPWithSHA-1AndMGF1Padding" works and also "RSA/ECB/PKCS1Padding" works fine. As a workaround I could change the padding but this might cause problems when updating from a previous version of the App which used "RSA/ECB/OAEPWithSHA-256AndMGF1Padding" because stored data could not be read anymore.

Has anybody here the same problem and maybe an idea how to fix it without changing the padding?

Thanks in advance and greetings from Hamburg, Dimitri

Answer 1

A possible solution is described here in the comment #15 from Sep 8, 2017 07:08PM:

https://issuetracker.google.com/issues/36708951#comment15

I changed the Cipher-Initialisation from

Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPWithSHA-256AndMGF1Padding");
cipher.init(Cipher.DECRYPT_MODE, this.getPrivateKey(context));

to

OAEPParameterSpec sp = new OAEPParameterSpec("SHA-256", "MGF1", new MGF1ParameterSpec("SHA-1"), PSource.PSpecified.DEFAULT);
Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPWithSHA-256AndMGF1Padding");
cipher.init(Cipher.DECRYPT_MODE, this.getPrivateKey(context), sp);

I tested this on Android 6 up to Android 8 (emulator) and the issue seems to be gone. You should also change the Cipher.ENCRYPT_MODE-Implementation.