WebException.Response.GetResponseStream() limited to 65536 characters

Question

I am trying to retrieve HTML code from a webpage using HttpWebRequest and HttpWebResponse.

response = (HttpWebResponse)request.GetResponse();
...
Stream stream = response.GetResponseStream();

The response object has a ContentLength value of 106142. When I look at the stream object, it has a length of 65536. When reading the stream with a StreamReader using ReadToEnd(), only the first 65536 characters are returned.

How can I get the whole code?

Edit:

Using the following code segment:

catch (WebException ex)
{
    errorMessage = errorMessage + ex.Message;
    if (ex.Response != null) {
        if (ex.Response.ContentLength > 0) 
        {
            using (Stream stream = ex.Response.GetResponseStream())
            {
                using (StreamReader reader = new StreamReader(stream))
                {
                    string pageOutput = reader.ReadToEnd().Trim();

ex.Response.ContentLength = 106142

ex.Response.GetResponseStream().Length = 65536

stream.Length = 65536

pageOutput.Length = 65534 (because of the trim)

And yes, the code is actually truncated.

Answer 1

You can find an answer in this topic in System.Net.HttpWebResponse.GetResponseStream() returns truncated body in WebException

You have to manage the HttpWebRequest object and change DefaultMaximumErrorResponseLength property. For example :

HttpWebRequest.DefaultMaximumErrorResponseLength = 1048576;

Answer 2

ReadToEnd does specifically just that, it reads to the end of the stream. I would check to make sure that you were actually being sent the entire expected response.

Answer 3

There seems to be a problem when calling the GetResponseStream() method on the HttpWebResponse returned by the exception. Everything works as expected when there is no exception.

I wanted to get the HTML code from the error returned by the server.

I guess I'll have to hope the error doesn't exceed 65536 characters...