Using MS crypto library on server 2012 - CryptCreateHash error code 87: ERROR_INVALID_PARAMETER

Question

I am attempting to host a web application on a new Windows server 2012 environment, however I am receiving an unprecidented error. This code has existed in our codebase for years and no problems have been experienced on any other platform.

The code in question calls the CryptCreateHash function of advapi32.dll - a Microsoft crypto library. When calling the function I am returned a 0 to denote the call failed, subsequently Err.LastDllError returns the error code 87, which is ERROR_INVALID_PARAMETER.

As I said before this code has worked perfectly for many years on a variety of environments - including a Windows Server 2012 test machine used by the Developers. However when put on the live environment which also runs Server 2012 (albeit in a slightly more complex context of a load balanced system) I receive the error. Neither server has yet been updated to Windows Server 2012 R2, it is running the out-of-the box version of the OS.

After creating a handle to the Crypt Provider using:

CryptAcquireContext(hCryptProv, vbNullString, SERVICE_PROVIDER, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT)

I use hCryptProv to call the CryptCreateHas function.

  If CryptCreateHash(hCryptProv, CALG_MD5, 0, 0, hHash) = 0 Then
      Dim _error As Integer = Err.LastDllError
      Throw New CryptoException("Error during CryptCreateHash. Error Code: " & _error.ToString)
  End If

Some example of the data being passed to the method from both live and dev environments:

Dev:
hCryptProv = 4966968
CALG_MD5 = 32771
hHash = 0

Live:
hCryptProv = 1587622576
CALG_MD5 = 32771
hHash = 0

It appears that both of these sets of parameters are basically identical, although the server's hCryptProv tends to be a much larger number (possibly because it has much more RAM?).

I have attempted to use SHA1 instead of MD5 but the same error occured.

Possibly this is a 32/64 bit related issue, presuming advapi32.dll is 32bit?

Any suggestions would be appreciated, thanks.

EDIT:

Prototypes as requested:

Private Declare Function CryptAcquireContext Lib "advapi32.dll" Alias "CryptAcquireContextA" _
    (ByRef phProv As IntPtr, _
     ByVal pszContainer As String, _
     ByVal pszProvider As String, _
     ByVal dwProvType As Integer, _
     ByVal dwFlags As Integer) As Integer


Private Declare Function CryptCreateHash Lib "advapi32.dll" _
    (ByVal hProv As IntPtr, _
     ByVal Algid As Integer, _
     ByVal hKey As Integer, _
     ByVal dwFlags As Integer, _
     ByRef phHash As Integer) As Integer

I have been playing with the datatype of phProv, previosuly it was Integer, I am yet to test using IntPtr. I have tried using ULong because that is how the HCRYPTPROV datatype is defined in the MSDN docs.

typedef ULONG_PTR HCRYPTPROV; 

Also here are the values of the handle returned by CryptAcquireContext in various configs:

LIVE 32: hCryptProv = 606412672 
LIVE 64: hCryptProv = -1480179632 
LOCAL: hCryptProv = 4966968 
DEV 32: hCryptProv = 99009648 
DEV 64: hCryptProv = 918798256 

This is from when I used Integer as the datatype, note the overflow on live.

EDIT 2

This possibly is fixed. Now when I call CryptDecrypt I receive error -2146893820 (NTE_BAD_LEN). possibly to do with the *pdwDataLen variable.

Here is the method definition:

Private Declare Function CryptDecrypt Lib "advapi32.dll" _
                              (ByVal hKey As IntPtr, _
                              ByVal hHash As IntPtr, _
                              ByVal Final As Boolean, _
                              ByVal dwFlags As Integer, _
                              ByVal pbData As String, _
                              ByRef pdwDataLen As Integer) _
                            As <MarshalAs(UnmanagedType.Bool)> Boolean
End Function

And the call:

lLength = Len(strData)
If CryptDecrypt(_hKey, 0, 1, 0, sTemp, lLength) = False Then
                Dim _error As Integer = Err.LastDllError
                Throw New CryptoException("Error during CryptDecrypt. Error Code: " & _error.ToString)
            End If

Answer 1

If you have a type ULONG_PTR, it needs to be defined as IntPtr in .NET. You'll also need a DllImportAttribute Your CryptCreateHash should be:

Declare Auto Function CryptCreateHash Lib "advapi32.dll" _
    (ByVal hProv As IntPtr, _
     ByVal algId As Integer, _
     ByVal hKey As IntPtr, _
     ByVal dwFlags As Integer, _
     ByRef phHast As IntPtr) As <MarshalAs(UnmanagedType.Bool)> Boolean

Also, be sure you've told it to set the last error. In C# we use a DllImportAttribute and make sure that SetLastError=true. Otherwise, calling Marshal.GetLastWin32Error doesn't return what's expected.

Update

Your CryptDecrypt prototype should be:

Declare Function CryptDecrypt Lib "advapi32.dll" 
    (ByVal hkey As IntPtr, _
     ByVal hHash As IntPtr, _
     <MarshalAs(UnmanagedType.Bool)> ByVal final As Boolean, _
     ByVal flags As Integer, _
     ByVal data As Byte(), 
     ByRef dataLen As Integer) As <MarshalAs(UnmanagedType.Bool)> Boolean

You'll need to convert your string to a byte array. Also note that the dataLen parameter is the length of the byte buffer, not the length of the string.

You should check out pinvoke.net, which has managed prototypes and examples for most Windows API calls.