Not understanding why WinHTTP does NOT authenticate certain HTTPS resource

Question

I\'d be extremely grateful for any kind of help that may help me resolving the problem.

From Excel VBA code I need to download & parse CSV file from HTTPS site h

Answer 1

And one more thing (in addition to the above solution) one should be aware of in case of using POST requests similar to the above code: for some obscure reason I still got once in 4-5 times (or even more) the hated 406 response from the website, which means in my case auth is NOT complete. After hours of step-by-step debugging I happily caught the cause: auth token value may have + signs, and from analyzing an arrow of several dozens auth tokens / response codes I discovered that +-containing tokens exactly match 406 codes.

The solution became pretty obvious: safely URL-encode +es for the PostData. With the help of http://www.blooberry.com/indexdot/html/topics/urlencoding.htm I finally came up to the following:

PostData = "authenticity_token=" & Replace(AuthToken, "+", "%2B", vbTextCompare) & _
    "&back_url=https://redmine.itransition.com/projects/" & Trim(RedmineProject) & _
    "/time_entries" & "&username=" & UN & "&password=" & PW & "&login=Login »"

+es are replaced by %2Bs, and this was that - no more 406s!)

The other special chars do not matter in my case, but the lesson was learned. Hope this will save several hours of life for someone else!

Answer 2

The logon at https://redmine.itransition.com/ is just an HTML form that posts a username & password to a script at /login.

This is not compatible with SetCredentials which is designed for server based authentication schemes like basic/digest/ntlm.

You need to load that page with no credentials, grab what looks like the volatile field authenticity_token from the generated form & post that along with username & password to /login.

If its a session based system it will response with the set-cookie header + data you need to use in subsequent request.

Answer 3

The above @Alex K. response was exactly what I was looking for soooo long! With the help of Firebug and MSDN I finished with 3 requests:

• GET request to collect authenticity_token data from login page using RegEx
• POST request to authenticate & collect required Cookie string from response
• GET request to finally obtain my beloved CSV

The following piece of code which is working as expected:

Set RegX_AuthToken = CreateObject("VBScript.RegExp")
' Below Pattern w/o double-quotes encoded: (?:input name="authenticity_token" type="hidden" value=")(.*)(?:")
RegX_AuthToken.Pattern = "(?:input name=" & Chr(34) & "authenticity_token" & Chr(34) & " type=" & Chr(34) & "hidden" & Chr(34) & " value=" & Chr(34) & ")(.*)(?:" & Chr(34) & ")"
RegX_AuthToken.IgnoreCase = True
RegX_AuthToken.Global = True

TargetURL = "https://redmine.itransition.com/login"

Set HTTPReq = CreateObject("WinHttp.WinHttpRequest.5.1")
HTTPReq.Open "GET", TargetURL, False
HTTPReq.Send

Set Token_Match = RegX_AuthToken.Execute(HTTPReq.ResponseText)
AuthToken = Token_Match.Item(0).SubMatches.Item(0)

PostData = "authenticity_token=" & AuthToken & "&back_url=https://redmine.itransition.com/" & "&username=" & UN & "&password=" & PW & "&login=Login »"

HTTPReq.Open "POST", TargetURL, False
HTTPReq.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
HTTPReq.Send (PostData)

SetCookieString = HTTPReq.GetResponseHeader("Set-Cookie")

TargetURL = "https://redmine.itransition.com/projects/pmct/time_entries.csv"
HTTPReq.Open "GET", TargetURL, False
HTTPReq.setRequestHeader "Cookie", SetCookieString
HTTPReq.Send

The following URL was helpful in building POST request: http://tkang.blogspot.com/2010/09/sending-http-post-request-with-vba.html

You need to load that page with no credentials, grab what looks like the volatile field authenticity_token from the generated form & post that along with username & password to /login.

Alex K. - thanks again for the brilliant suggestion! (: