Java Signature.verify results in SignatureException: Signature encoding error Caused by IOException: Sequence tag error

Question

first of all this is not a duplicate question as most people report this exception when creating a Public Key from a Certificate which is missing "---BEGIN RSA CERTIFICATE---" line.

gist of what I am trying to do is 1. Sign a 50Byte message on JCOP Smart card using SHA1withRSA algorithm (RSA Key is 1024 bits). 2. Export the signature from smart card to server. 3. Verify the signature on the server.

Code snippet on the smart card to create the signature. The key point is I am using algorithm Signature.ALG_RSA_SHA_PKCS1 in the Java Card to create the signature.

private void setcustccid(APDU apdu) {

    byte[] buffer = apdu.getBuffer();
    if (buffer[ISO7816.OFFSET_LC] != (byte)24) {
      ISOException.throwIt(ISO7816.SW_WRONG_LENGTH);
    }
    else {
      short bytesLeft = (short) (buffer[ISO7816.OFFSET_LC] & 0x00FF);
      short readCount = apdu.setIncomingAndReceive();
      if (readCount < bytesLeft) {
        ISOException.throwIt(ISO7816.SW_BYTES_REMAINING_00);
      }
      try {
                  Signature signature = Signature.getInstance(Signature.ALG_RSA_SHA_PKCS1, false);
        signature.init(privKey, Signature.MODE_SIGN);
        Util.arrayCopy(buffer, (short)buffer[ISO7816.OFFSET_CDATA], tempStorage, (short) 0, (byte)24);
        Util.arrayCopy(transactionHistory, (short)0, tempStorage, (short)24, (byte)30);
      } 
      catch (Exception ex) {
        ISOException.throwIt(ISO7816.SW_BYTES_REMAINING_00);
      }

      signature.sign(tempStorage, (short)0, (short)50, finalEncryptedMsg, (short)0);
    }
    }

Code snippet on the Server side trying to verify the signature exported from the Java Smart card which is throwing the exception. The key point here is I am using Signature.getInstance("SHA1withRSA") on the server side. I am doing Cipher decrypt of the signed message just to confirm that the public key generated is working and it is.

modulusString = new BigInteger(1, rsaModulus);
exponentString = new BigInteger(1, rsaExponent);
RSAPublicKeySpec keySpec = new RSAPublicKeySpec(modulusString, exponentString);
KeyFactory factor = KeyFactory.getInstance("RSA"); 
PublicKey publicKey = (RSAPublicKey) factor.generatePublic(keySpec);
rsaCipher = Cipher.getInstance("RSA");
rsaCipher.init(Cipher.DECRYPT_MODE, publicKey);
signature = Signature.getInstance("SHA1withRSA");
signature.initVerify(publicKey);
signature.update(resultBytes);
signature.verify(finalEncryptedMsg);
tempStorage = rsaCipher.doFinal(finalEncryptedMsg);
System.out.println("Decrypted Length = " + tempStorage.length);

The exception happens on signature.verify(). Another thread referred to this same exception but the solution was to add Bouncy Castle as the Provider in Signature.getInstance(). Not sure why Bouncy Castle would be required for verification of signature.

Any help would be greatly appreciated. If you need more of the code to identify the issue please let me know.

java.security.SignatureException: Signature encoding error
    at sun.security.rsa.RSASignature.engineVerify(Unknown Source)
    at java.security.Signature$Delegate.engineVerify(Unknown Source)
    at java.security.Signature.verify(Unknown Source)
    at com.mse.reader.SmartCardReader.main(SmartCardReader.java:234)
Caused by: java.io.IOException: Sequence tag error
    at sun.security.util.DerInputStream.getSequence(Unknown Source)
    at sun.security.rsa.RSASignature.decodeSignature(Unknown Source)
    ... 4 more

Here is the Encrypted Msg and Decrypted Msg. (Base4.encodeBase64)

Encrypted Length = 128

JpypH/vKYR4RLjQA4frCab5WljnAoWgNiGUb0k+DCmh8gdWbOtpR/XUec2rW96Nr1k7czNTb2s/2WQDGXe05a3JjNrlErrfijhdWvn9flIzR/5uPrS3VJw+ALESl8NWqR5HF3AgArE6uYIW87EtSjO0iPJTO2N0cITtLghdUSBs=

Decrypted Length = 50

gCUAABgAAAAAO5rJkAAAAAAAvGFOAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=

Answer 1

You've moved your smart card provider in front of the other providers on Java SE, and for some reason it is also trying to verify RSA signatures instead of just using it for RSA private key operations.

There are a few methods of solving this:

1. if you are using the same signature instance for verification, then use a different one for the verification with the public key
2. if that doesn't solve your problem, try and see if you can move the smart card provider down in the list of providers in the Security class (check the JCA documentation on how to do this)
3. otherwise simply provide the correct provider using the Signature class, I would recommend specifying "SunRsaSign" (you may want to make this string configurable)
4. explain to the company behind com.mse that they should implement delayed provider selection correctly and not gobble up software public keys for use in their hardware device

Note that talking about "server side" is very confusing as the smart card acts as a server. "Terminal side" and "card side" would be a much more clear.