Mismatch in keytool and openssl certificate fingerprint

Question

I was trying to fingerprint android developer certificates, inside META-INF/, for research purposes.

I'm finding certain scenarios in which output from keytool and openssl would give me different SHA1 fingerprints for the same certificate:

Using keytool:

keytool -princert -file META-INF/CERT.RSA
...
    SHA1: 9D:17:FB:AB:67:BB:D0:7B:12:FE:E8:33:7D:66:F1:C4:2B:03:BD:F7
...

Using openssl:

openssl pkcs7 -inform DER -in META-INF/CERT.RSA -print_certs -out CERT.cert
openssl x509 -in CERT.cert -fingerprint -noout
     SHA1 Fingerprint=80:D5:CD:66:6E:44:75:62:A8:B3:7E:5D:AC:00:DE:1D:FF:6B:E6:CA

Is this normal - keytool uses its own way to fingerprint certs, which differs from openssl- , am I doing something wrong, or is there a bug somewhere?

By the way, I'm no expert, so feel free to give any advice, even the the most basic stuff!

If you need an example of this, I can give you the name of an android app in which this happens.

Thanks in advance.

Answer 1

The different SHA1 fingerprints are caused by different encodings of the code signing certificate.

The first fingerprint (by keytool) is calculated over the certificate bytes exactly as they are contained in the PKCS#7 file META-INF/CERT.RSA. The length of the signature (of the certificate, not the signature of the code) is encoded here in two bytes, where actually one byte would be enough. To see this we have to take a look at the ASN.1 dump of CERT.RSA. There are several programs that can do this, but I recommend Peter Gutmann's dumpasn1/GUIdumpASN. The relevant part is this:

    <06 09>
532   9:             OBJECT IDENTIFIER
       :               sha1WithRSAEncryption (1 2 840 113549 1 1 5)
    <05 00>
543   0:             NULL
       :             }
    <03 82 00 81>
545 129:           BIT STRING    
       :             65 26 30 0C 41 32 63 75    e&0.A2cu
       :             2F B7 DF 9A 96 37 72 1B    /....7r.

The bytes 82 00 81 are the length bytes of the BITSTRING (which is the signature of the certificate) encoded in the long form.

According to the Distinguished Encoding Rules DER the "shortest possible length encoding must be used", which would have been 81 81. This means that the code signing certificate is not completely DER encoded.

When you exported the certificate with openssl, it re-encoded the certificate to valid DER:

    <03 81 81>
489 129:   BIT STRING    
        :     65 26 30 0C 41 32 63 75    e&0.A2cu
        :     2F B7 DF 9A 96 37 72 1B    /....7r. 

Therefore the hash value (fingerprint) of the certificate is different, it has changed.

Neither keytool nor openssl did something wrong. The point of DER encoding is that it generates always exactly the same byte representation of ASN.1 structures.

The software that caused this issue is the tool the faulty code signing certificate was created with.