Separate digest & signing using java security provider

Question

Due to some implementation detail, I need to split hashing and signature generation. I tried to achieve this using the 'NONEwithRSA' signature algorithm.

This is a basic working example:

public void rsaSignatureIntegrityTest() {
    KeyPairGenerator gen = KeyPairGenerator.getInstance("RSA");
    gen.initialize(2048, new SecureRandom());
    KeyPair pair = gen.generateKeyPair();

    byte[] digest = MessageDigest.getInstance("SHA-256").digest(MESSAGE);
    Signature signer = Signature.getInstance("NONEwithRSA");
    signer.initSign(pair.getPrivate());
    signer.update(digest);
    byte[] signed = signer.sign();

    Signature verifier = Signature.getInstance("SHA256withRSA");
    verifier.initVerify(pair.getPublic());
    verifier.update(MESSAGE);
    verifier.verify(signed);
}

Running this, the verifier.verify() method throws a Signature exception:

java.security.SignatureException: Signature encoding error
    at sun.security.rsa.RSASignature.engineVerify(RSASignature.java:204)
    at java.security.Signature$Delegate.engineVerify(Signature.java:1219)
    at java.security.Signature.verify(Signature.java:652)
    at testing.rsaSignatureIntegrityTest(testing.java:38)
    ...
Caused by: java.io.IOException: Sequence tag error
    at sun.security.util.DerInputStream.getSequence(DerInputStream.java:297)
    at sun.security.rsa.RSASignature.decodeSignature(RSASignature.java:229)
    at sun.security.rsa.RSASignature.engineVerify(RSASignature.java:195)
    ... 26 more

The verifier object seems to expect some kind of DER encoded structure, which is not produced by the signer object.

Any suggestions on how to get this to work?

Answer 1

As suspected, RSA signatures are generated with a wrapped digest value, containing the hash oid. Using bouncycastle this can be done quite comfortably.

Example:

public void rsaSignatureIntegrityTest() {
    KeyPairGenerator gen = KeyPairGenerator.getInstance("RSA");
    gen.initialize(2048, new SecureRandom());
    KeyPair pair = gen.generateKeyPair();

    byte[] digest = MessageDigest.getInstance("SHA-256").digest(MESSAGE);
    Signature signer = Signature.getInstance("NONEwithRSA");
    signer.initSign(pair.getPrivate());
    signer.update(wrapForRsaSign(digest, "SHA-256"));
    byte[] signed = signer.sign();
    System.out.println(Base64.getEncoder().encodeToString(signed));

    Signature verifier = Signature.getInstance("SHA256withRSA");
    verifier.initVerify(pair.getPublic());
    verifier.update(MESSAGE);
    verifier.verify(signed);
}

private byte[] wrapForRsaSign(byte[] dig, String hashAlgo) {
    ASN1ObjectIdentifier oid = new DefaultDigestAlgorithmIdentifierFinder().find(hashAlgo).getAlgorithm();
    ASN1Sequence oidSeq = new DERSequence(new ASN1Encodable[] { oid, DERNull.INSTANCE });
    ASN1Sequence seq = new DERSequence(new ASN1Encodable[] { oidSeq, new DEROctetString(dig) });
    try {
        return seq.getEncoded();
    } catch (IOException e) {
        throw new DigestException(e);
    }
}