Decrypting an OpenSSL PEM Encoded RSA private key with Java?

Question

I have an encrypted private key and I know the password.

I need to decrypt it using a Java library.

I\'d prefer not to use BouncyCastle though, unless ther

Answer 1

Java code example below shows how to construct the decryption key to obtain the underlying RSA key from an encrypted private key created using the openssl 1.0.x genrsa command; specifically from the following genrsa options that may have been leveraged:

-des encrypt the generated key with DES in cbc mode

-des3 encrypt the generated key with DES in ede cbc mode (168 bit key)

-aes128, -aes192, -aes256 encrypt PEM output with cbc aes

Above options result in encrypted RSA private key of the form ...

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AAA,BBB
...

Where AAA would be one of:

DES-CBC, DES-EDE3-CBC, AES-128-CBC, AES-192-CBC, AES-256-CBC

AND BBB is the hex-encoded IV value

KeyFactory factory = KeyFactory.getInstance("RSA");
KeySpec keySpec = null;
RSAPrivateKey privateKey = null;

Matcher matcher = OPENSSL_ENCRYPTED_RSA_PRIVATEKEY_PATTERN.matcher(pemContents);
if (matcher.matches())
{
    String encryptionDetails = matcher.group(1).trim(); // e.g. AES-256-CBC,XXXXXXX
    String encryptedKey = matcher.group(2).replaceAll("\\s", ""); // remove tabs / spaces / newlines / carriage return etc

    System.out.println("PEM appears to be OpenSSL Encrypted RSA Private Key; Encryption details : "
        + encryptionDetails + "; Key : " + encryptedKey);

    byte[] encryptedBinaryKey = java.util.Base64.getDecoder().decode(encryptedKey);

    String[] encryptionDetailsParts = encryptionDetails.split(",");
    if (encryptionDetailsParts.length == 2)
    {
        String encryptionAlgorithm = encryptionDetailsParts[0];
        String encryptedAlgorithmParams = encryptionDetailsParts[1]; // i.e. the initialization vector in hex

        byte[] pw = new String(password).getBytes(StandardCharsets.UTF_8);
        byte[] iv = fromHex(encryptedAlgorithmParams);

        MessageDigest digest = MessageDigest.getInstance("MD5");

        // we need to come up with the encryption key
        
        // first round digest based on password and first 8-bytes of IV ..
        digest.update(pw);
        digest.update(iv, 0, 8);

        byte[] round1Digest = digest.digest(); // The digest is reset after this call is made.
        
        // second round digest based on first round digest, password, and first 8-bytes of IV ...
        digest.update(round1Digest);
        digest.update(pw);
        digest.update(iv, 0, 8);

        byte[] round2Digest = digest.digest();

        Cipher cipher = null;
        SecretKey secretKey = null;
        byte[] key = null;
        byte[] pkcs1 = null;

        if ("AES-256-CBC".equals(encryptionAlgorithm))
        {
            cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");

            key = new byte[32]; // 256 bit key  (block size still 128-bit)
            System.arraycopy(round1Digest, 0, key, 0, 16);
            System.arraycopy(round2Digest, 0, key, 16, 16);

            secretKey = new SecretKeySpec(key, "AES");
        }
        else if ("AES-192-CBC".equals(encryptionAlgorithm))
        {
            cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");

            key = new byte[24]; // key size of 24 bytes
            System.arraycopy(round1Digest, 0, key, 0, 16);
            System.arraycopy(round2Digest, 0, key, 16, 8);

            secretKey = new SecretKeySpec(key, "AES");
        }
        else if ("AES-128-CBC".equals(encryptionAlgorithm))
        {
            cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");

            key = new byte[16]; // 128 bit key
            System.arraycopy(round1Digest, 0, key, 0, 16);

            secretKey = new SecretKeySpec(key, "AES");
        }
        else if ("DES-EDE3-CBC".equals(encryptionAlgorithm))
        {
            cipher = Cipher.getInstance("DESede/CBC/PKCS5Padding");
            
            key = new byte[24]; // key size of 24 bytes
            System.arraycopy(round1Digest, 0, key, 0, 16);
            System.arraycopy(round2Digest, 0, key, 16, 8);

            secretKey = new SecretKeySpec(key, "DESede");
        }
        else if ("DES-CBC".equals(encryptionAlgorithm))
        {
            cipher = Cipher.getInstance("DES/CBC/PKCS5Padding");
            
            key = new byte[8]; // key size of 8 bytes
            System.arraycopy(round1Digest, 0, key, 0, 8);

            secretKey = new SecretKeySpec(key, "DES");
        }

        cipher.init(Cipher.DECRYPT_MODE, secretKey, new IvParameterSpec(iv));

        pkcs1 = cipher.doFinal(encryptedBinaryKey);

        keySpec = pkcs1ParsePrivateKey(pkcs1);

        privateKey = (RSAPrivateKey) factory.generatePrivate(keySpec);
    }
}

The regular expression ...

static final String OPENSSL_ENCRYPTED_RSA_PRIVATEKEY_REGEX = "\\s*" 
+ "-----BEGIN RSA PUBLIC KEY-----" + "\\s*"
+ "Proc-Type: 4,ENCRYPTED" + "\\s*"
+ "DEK-Info:" + "\\s*([^\\s]+)" + "\\s+"
+ "([\\s\\S]*)"
+ "-----END RSA PUBLIC KEY-----" + "\\s*";

static final Pattern OPENSSL_ENCRYPTED_RSA_PRIVATEKEY_PATTERN = Pattern.compile(OPENSSL_ENCRYPTED_RSA_PRIVATEKEY_REGEX);

the fromHex(...) method ...

public static byte[] fromHex(String hexString)
{
    byte[] bytes = new byte[hexString.length() / 2];
    for (int i = 0; i < hexString.length(); i += 2)
    {
        bytes[i / 2] = (byte) ((Character.digit(hexString.charAt(i), 16) << 4)
            + Character.digit(hexString.charAt(i + 1), 16));
    }
    return bytes;
}