Does System.Net.Http.HttpClient suffer from HttpWebRequest.AllowWriteStreamBuffering?

问题

I've been trying to use System.Net.Http.HttpClient to POST a larger file (+1GB) but it throws a SystemOutOfMemory exception:

at System.Net.ScatterGatherBuffers.AllocateMemoryChunk(Int32 newSize)
at System.Net.ScatterGatherBuffers..ctor(Int64 totalSize)
at System.Net.ConnectStream.EnableWriteBuffering()
at System.Net.HttpWebRequest.SetRequestSubmitDone(ConnectStream submitStream)
at System.Net.Connection.CompleteStartRequest(Boolean onSubmitThread, HttpWebRequest request, TriState needReConnect)
at System.Net.Connection.SubmitRequest(HttpWebRequest request, Boolean forcedsubmit)
at System.Net.ServicePoint.SubmitRequest(HttpWebRequest request, String connName)
at System.Net.HttpWebRequest.SubmitRequest(ServicePoint servicePoint)
at System.Net.HttpWebRequest.BeginGetRequestStream(AsyncCallback callback, Object state)
at System.Net.Http.HttpClientHandler.StartGettingRequestStream(RequestState state)
at System.Net.Http.HttpClientHandler.PrepareAndStartContentUpload(RequestState state)

Apparently, a similar problem occurs for HttpWebRequest as discussed here: http://support.microsoft.com/kb/908573.

Is there any way to set AllowWriteStreamBuffering of the underlying web request to false? I can't find any.

Cheers,

回答1:

Just to save time of others interested, I'm answering my own question.

After a few tests the exception seems to be down to the same issue with HttpWebRequest as discussed in the question. I use Microsoft.AspNet.WebApi version 4.0.20710.0.

Below are two equivalent pieces of code; the former fails on large files, whereas the latter works fine.

BTW, despite the issue overall benefits of the HttpClient become really apparent :-)

using `HttpClient`

var clientRef = new System.Net.Http.HttpClient(
    new HttpClientHandler()
    {
        Credentials = new System.Net.NetworkCredential(MyUsername, MyPassword)
    });
clientRef.BaseAddress = new Uri(serverAddress);
clientRef.DefaultRequestHeaders.ExpectContinue = false;
clientRef.PostAsync(
    MyFavoriteURL,
    new System.Net.Http.StreamContent(inputStream)).ContinueWith(
        requestTask =>
        {
            HttpResponseMessage response = requestTask.Result;
            response.EnsureSuccessStatusCode();
        }, TaskContinuationOptions.LongRunning).Wait();

using `HttpWebRequest`

// Preauthenticate
var req  = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(MyFavoriteURL);
req.Credentials = new System.Net.NetworkCredential(MyUsername, MyPassword);
req.Method = "POST";
req.PreAuthenticate = true;
req.Timeout = 10000;
using (var resp = (System.Net.HttpWebResponse)req.GetResponse())
{
     if (resp.StatusCode != System.Net.HttpStatusCode.Accepted && resp.StatusCode != System.Net.HttpStatusCode.OK)
     {
         throw new Exception("Authentication error");
     }
}

// Upload
req = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(MyFavoriteURL);
req.Credentials = new System.Net.NetworkCredential(MyUsername, MyPassword);
req.Method = "POST";
req.PreAuthenticate = true;
req.Timeout = 1200000;
req.ContentLength = inputStream.Length;
req.ContentType = "application/binary";
req.AllowWriteStreamBuffering = false;
req.Headers.ExpectContinue = false;
using (var reqStream = req.GetRequestStream())
{
    inputStream.CopyTo(reqStream);
}

using (var resp = (System.Net.HttpWebResponse)req.GetResponse())
{
    if (resp.StatusCode != System.Net.HttpStatusCode.Accepted && resp.StatusCode != System.Net.HttpStatusCode.OK)
    {
        throw new Exception("Error uploading document");
    }
}

回答2:

I also ran into the same issue for large file upload. Adding my findings so that it will help someone.

HttpClient determine whether to do buffering based on the following conditions,

if( HttpRequestMessage.Headers.TransferEncodingChunked == true ||     HttpRequestMessage.Content.Headers.ContentLength != null )
{
  //do streamed upload
}else
{
  //do buffered upload.
}

I have used the PushedStreamContent as HttpRequestMessage.Content because my server accepts at max 150 MB per request. Once I set the TransferEncodingChunked to True the memory spike is reduced.

回答3:

I think you have a duplicate header, you can remove the former

req.Headers.Add("ExpectContinue", "false");
...
req.Headers.ExpectContinue = false;

回答4:

Helped me send a large file without buffering through a 'relay controller'.
The scenario that I got to work was:
Client web app(spa, etc) -> RelayController -> Final content controller storing the http data stream

        var request = new HttpRequestMessage()
        {

            RequestUri = new Uri("https://someurl/api/upload"),
            Method = HttpMethod.Post,
        };
        request.Headers.TransferEncodingChunked = true;
        request.Content = new StreamContent(Request.Body);

        await httpClient.SendAsync(request);`

Had spent a lot of time on getting it to work, got the lead from Prakash P, who had the httpclient streaming vs buffering logic posted.

来源：https://stackoverflow.com/questions/16037249/does-system-net-http-httpclient-suffer-from-httpwebrequest-allowwritestreambuffe

标签

.net

httpclient

out-of-memory