MD5-SHA1 hash using CryptoAPI

Question

I have requirement of signing a data using MD5-SHA1 hash (MD5 hash and SHA1 hash of the data are combined and then signed).

Primary requirement:

MD5-SHA1 has

Answer 1

By using CALG_SSL3_SHAMD5 in CryptCreateHash and after that calling CryptSetHashParam and then calling the CryptSignHash did the job for me.

Let me detailed out the requirement: 1. Generally, I gave private key using EVP_PKEY to OpenSSL function so that it can do the signing part. I used to extract the private key from Windows Certificate store (when key is marked as exportable). But in one scenario, private key is not marked as exportable and I was not able to get private key to EVP_PKEY. So, I did modification to OpenSSL code on the case when private key is not available, then the signing part is implemented using CryptoAPI (since CryptoAPI allows the signing using such non-exportable key).

So, in my case, data is completely hashed by OpenSSL (using md5-sha1 hash). Only thing left for me is to do signing.

So, following steps (under signing part) did the job:

1. Created hash using Certificate key handle provider and use CALG_SSL3_SHAMD5 algorithm.
2. Using CryptSetHashParam, I set the initial hash value
3. Signed the hash using CryptSignHash.
4. After that, reverse the bytes order (as OpenSSL signature is Big endian mentioned [here][1] which I found after lots of experimenting).

Rest of code of OpenSSL seems happy with this signature.