TcpClient.GetStream().DataAvailable returns false, but stream has more data

Question

So, it would seem that a blocking Read() can return before it is done receiving all of the data being sent to it. In turn we wrap the Read() with a loop that is controlled b

Answer 1

Using TcpClient.Available will allow this code to read exactly what is available each time. TcpClient.Available is automatically set to TcpClient.ReceiveBufferSize when the amount of data remaining to be read is greater than or equal to TcpClient.ReceiveBufferSize. Otherwise it is set to the size of the remaining data. Hence, you can indicate the maximum amount of data that is available for each read by setting TcpClient.ReceiveBufferSize (e.g., oClient.ReceiveBufferSize = 4096;).

        protected void OnClientCommunication(TcpClient oClient)
        {
            NetworkStream stream = oClient.GetStream();
            MemoryStream msIn = new MemoryStream();

            byte[] aMessage;
            oClient.ReceiveBufferSize = 4096;
            int iBytesRead = 0;

            while (stream.DataAvailable)
            {
                int myBufferSize = (oClient.Available < 1) ? 1 : oClient.Available;
                aMessage = new byte[oClient.Available];

                int iRead = stream.Read(aMessage, 0, aMessage.Length);
                iBytesRead += iRead;
                msIn.Write(aMessage, 0, iRead);
            }
            MemoryStream msOut = new MemoryStream();

            // .. Do some processing adding data to the msOut stream

            msOut.WriteTo(stream);
            stream.Flush();

            oClient.Close();
        }

Answer 2

I'm seeing a problem with this.
You're expecting that the communication will be faster than the while() loop, which is very unlikely.
The while() loop will finish as soon as there is no more data, which may not be the case a few milliseconds just after it exits.

Are you expecting a certain amount of bytes?
How often is OnClientCommunication() fired? Who triggers it?

What do you do with the data after the while() loop? Do you keep appending to previous data?

DataAvailable WILL return false because you're reading faster than the communication, so that's fine only if you keep coming back to this code block to process more data coming in.

Answer 3

I was trying to check DataAvailable before reading data from a network stream and it would return false, although after reading a single byte it would return true. So I checked the MSDN documentation and they also read before checking. I would re-arrange the while loop to a do while loop to follow this pattern.

http://msdn.microsoft.com/en-us/library/system.net.sockets.networkstream.dataavailable.aspx

        // Check to see if this NetworkStream is readable. 
        if(myNetworkStream.CanRead){
            byte[] myReadBuffer = new byte[1024];
            StringBuilder myCompleteMessage = new StringBuilder();
            int numberOfBytesRead = 0;

            // Incoming message may be larger than the buffer size. 
            do{
                 numberOfBytesRead = myNetworkStream.Read(myReadBuffer, 0, myReadBuffer.Length);

                 myCompleteMessage.AppendFormat("{0}", Encoding.ASCII.GetString(myReadBuffer, 0, numberOfBytesRead));

            }
            while(myNetworkStream.DataAvailable);

            // Print out the received message to the console.
            Console.WriteLine("You received the following message : " +
                                         myCompleteMessage);
        }
        else{
             Console.WriteLine("Sorry.  You cannot read from this NetworkStream.");
        }

Answer 4

When I have this code:

    var readBuffer = new byte[1024];
    using (var memoryStream = new MemoryStream())
    {
        do
        {
            int numberOfBytesRead = networkStream.Read(readBuffer, 0, readBuffer.Length);
            memoryStream.Write(readBuffer, 0, numberOfBytesRead);
        }
        while (networkStream.DataAvailable);
    }

From what I can observe:

• When sender sends 1000 bytes and reader wants to read them. Then I suspect that NetworkStream somehow "knows" that it should receive 1000 bytes.
• When I call .Read before any data arrives from NetworkStream then .Read should be blocking until it gets more than 0 bytes (or more if .NoDelay is false on networkStream)
• Then when I read first batch of data I suspect that .Read is somehow updating from its result the counter of those 1000 bytes at NetworkStream and before this happens I suspect, that in this time the .DataAvailable is set to false and after the counter is updated then the .DataAvailable is then set to correct value if the counter data is less than 1000 bytes. It makes sense when you think about it. Because otherwise it would go to the next cycle before checking that 1000 bytes arrived and the .Read method would be blocking indefinitely, because reader could have already read 1000 bytes and no more data would arrive.
• This I think is the point of failure here as already James said:

Yes, this is just the way these libraries work. They need to be given time to run to fully validate the data incoming. – James Apr 20 '16 at 5:24

• I suspect that the update of internal counter between end of .Read and before accessing .DataAvailable is not as atomic operation (transaction) so the TcpClient needs more time to properly set the DataAvailable.

When I have this code:

    var readBuffer = new byte[1024];
    using (var memoryStream = new MemoryStream())
    {
        do
        {
            int numberOfBytesRead = networkStream.Read(readBuffer, 0, readBuffer.Length);
            memoryStream.Write(readBuffer, 0, numberOfBytesRead);

            if (!networkStream.DataAvailable)
                System.Threading.Thread.Sleep(1); //Or 50 for non-believers ;)
        }
        while (networkStream.DataAvailable);
    }

Then the NetworkStream have enough time to properly set .DataAvailable and this method should function correctly.

Fun fact... This seems to be somehow OS Version dependent. Because the first function without sleep worked for me on Win XP and Win 10, but was failing to receive whole 1000 bytes on Win 7. Don't ask me why, but I tested it quite thoroughly and it was easily reproducible.

Answer 5

You have to know how much data you need to read; you cannot simply loop reading data until there is no more data, because you can never be sure that no more is going to come.

This is why HTTP GET results have a byte count in the HTTP headers: so the client side will know when it has received all the data.

Here are two solutions for you depending on whether you have control over what the other side is sending:

1. Use "framing" characters: (SB)data(EB), where SB and EB are start-block and end-block characters (of your choosing) but which CANNOT occur inside the data. When you "see" EB, you know you are done.

2. Implement a length field in front of each message to indicate how much data follows: (len)data. Read (len), then read (len) bytes; repeat as necessary.

This isn't like reading from a file where a zero-length read means end-of-data (that DOES mean the other side has disconnected, but that's another story).

A third (not recommended) solution is that you can implement a timer. Once you start getting data, set the timer. If the receive loop is idle for some period of time (say a few seconds, if data doesn't come often), you can probably assume no more data is coming. This last method is a last resort... it's not very reliable, hard to tune, and it's fragile.