Header values overwritten on redirect in HTTPClient

Question

I'm using httpclient 4.2.5 to make http requests which have to handle redirects as well. Here is a little example to understand the context:

1. A sends http request (using httpclient 4.2.5) to B
2. B sends 302 redirect (containing url to C) back to A
3. A follows redirect to C
4. C retrieves request URL and do some work with it

If C parses the request URL by request.getRequestURL() (HttpServlet API) it contains e.g. host and port of the original request from step 1, which is wrong.

The problem exists in step 2, where httpclient handles the redirect. It just copies all headers from the original request (step 1) to the current request (step 3). I already had a look at the responsible code, via grepcode:
DefaultRequestDirector

HttpUriRequest redirect = redirectStrategy.getRedirect(request, response, context);
HttpRequest orig = request.getOriginal();
redirect.setHeaders(orig.getAllHeaders());

I don't really understand why all headers of the original request are copied to the current request.
E.g. using cURL for a simple test is doing it as expected, C would receive the correct host and port.

Implementing my own redirect strategy does not help because the original headers are copied after it.

Answer 1

I had the same problem when trying to download files from bitbucket's download section using HttpClient. After the first request bitbucket sends a redirect to CDN which then complains if the Authorization header is set.

I worked around it by changing DefaultRedirectStrategy.getRedirect() method to return redirect object which does not allow Authorization headers to be set.

I work with Scala so here is the code:

val http = new DefaultHttpClient()
http.setRedirectStrategy(new DefaultRedirectStrategy() {
  override def getRedirect(
    request: HttpRequest, response: HttpResponse, context: HttpContext
  ): HttpRequestBase = {
    val uri: URI = getLocationURI(request, response, context)
    val method: String = request.getRequestLine.getMethod
    if (method.equalsIgnoreCase(HttpHead.METHOD_NAME)) {
      new HttpHead(uri) {
        override def setHeaders(headers: Array[Header]) {
          super.setHeaders(headers.filterNot(_.getName == "Authorization"))
        }
      }
    }
    else {
      new HttpGet(uri) {
        override def setHeaders(headers: Array[Header]) {
          super.setHeaders(headers.filterNot(_.getName == "Authorization"))
        }
      }
    }
  }
})

Answer 2

Please note orig.getAllHeaders() returns an array of headers explicitly added to the message by the caller. The code from DefaultRequestDirector posted above does not copy request headers automatically generated by HttpClient such as Host, Content-Length, Transfer-Encoding and so on.

You post a wire log of the session exhibiting the problem I may be able to tell why redirects do not work as expected.