The new subject hash openssl algorithm differs

Question

I am running into an issue when managing openssl certificates from Java Framework.

openssl x509 -subject_hash ...

output differs to the one that Java framework returns when calling X509_NAME_hash(), see below.

The reason for this is that openssl changed the way it calculates the SHA1. Now, instead of basing the hash in the ASN.1 DER representation of the subject, as it does for MD5, it first calculates the CANONICAL representation and then based on that, it calculates the ASN.1 DER, and then uses that as the input for the SHA1 algorithm.

NativeCrypto.java:

// --- X509_NAME -----------------------------------------------------------
public static int X509_NAME_hash(X500Principal principal) {
    return X509_NAME_hash(principal, "SHA1");
}

private static int X509_NAME_hash(X500Principal principal, String algorithm) {
    try {
        byte[] digest = MessageDigest.getInstance(algorithm).digest(principal.getEncoded());
        return Memory.peekInt(digest, 0, ByteOrder.LITTLE_ENDIAN);
    } catch (NoSuchAlgorithmException e) {
        throw new AssertionError(e);
    }
}

I was looking into x_name.c and x509_cmp.c from openssl library to try to fix it in the Java under-layer. But I had no success.

I understand that I have to modify X509_NAME_hash method in x509_cmp.c. But not sure if that change should be right before or right after i2d_X509_NAME(x,NULL); This method is calculating the CANONICAL representation of the subject name, right? then, I would need to calculate the ASN1 DER based on that output, right? but I just cant make it.

I would be thankful if somebody can guide me or through some light on this to solve this issue.

x509_cmp.c:

    unsigned long X509_NAME_hash(X509_NAME *x)
    {
    unsigned long ret=0;
    unsigned char md[SHA_DIGEST_LENGTH];

    /* Make sure X509_NAME structure contains valid cached encoding */
    i2d_X509_NAME(x,NULL);
    if (!EVP_Digest(x->canon_enc, x->canon_enclen, md, NULL, EVP_sha1(),
            NULL))
            return 0;

    ret=(   ((unsigned long)md[0]     )|((unsigned long)md[1]<<8L)|
            ((unsigned long)md[2]<<16L)|((unsigned long)md[3]<<24L)
            )&0xffffffffL;
    return(ret);
    }

The function x509_name_canon apparently performs the re-encoding. This is in crypto/asn1/x_name.c:

/* This function generates the canonical encoding of the Name structure.
 * In it all strings are converted to UTF8, leading, trailing and
 * multiple spaces collapsed, converted to lower case and the leading
 * SEQUENCE header removed.
 * 
 */

Answer 1

You are not very far from it, if you want the same result as OpenSSL new SubjectHash you must remove the leading sequence of the DN. Thus you have to do something like this :

// --- X509_NAME -----------------------------------------------------------

public static int X509_NAME_hash(X500Principal principal) {
    return X509_NAME_hash(principal, "SHA1");
}

private static int X509_NAME_hash(X500Principal principal, String algorithm) {
    try {

        byte[] princ = principal.getEncoded();
        final ASN1Sequence obj = (ASN1Sequence) ASN1Object.fromByteArray( princ );

        // Remove the leading sequence ...
        final DERSet enc = (DERSet) obj.getObjectAt(0);
        final byte[] toHash = enc.getDEREncoded();

        MessageDigest md = MessageDigest.getInstance(algorithm);
        byte[] digest = md.digest(toHash);
        return Memory.peekInt(digest, 0, ByteOrder.LITTLE_ENDIAN);

    } catch (NoSuchAlgorithmException e) {
        throw new AssertionError(e);
    } catch (IOException e) {
        throw new AssertionError(e);
    }
}

And with this the result is the same as OpenSSL new Subject_hash.