How to “concatenate” or “combine” or “join” a series of 'binarily' serialized byte arrays? [duplicate]

问题

Possible Duplicate:
Is the received stream from a socket limited to a single send command?

Note: I see this question very complicated (hopefully not for you guys, that's why Im asking here lol) and I tried my best to explain it as simple and clear as possible.

In my application, I'm continually receiving byte arrays in a fix sized buffer.

These series of byte arrays that I'm receiving has been serialized 'binarily'.

However, sometimes the byte array received will be bigger than the fix sized buffer so I would need to store the current received byte array into a container and loop again to receive the remaining byte arrays coming in.

My question now is how to "concatenate" or "combine" or "join" all the "batches" of byte arrays I received ( and is stored in a container, possibly a queue of byte arrays) to form a single byte array and then de-serialize them?

int bytesRead = client.EndReceive(ar);
if (bytesRead > 0)
    {
        // There might be more data, so store the data received so far.
        // If the buffer was not filled, I have to get the number of bytes received as Thorsten Dittmar was saying, before queuing it
        dataReceivedQueue.Enqueue(state.buffer);

        // Get the rest of the data.
        client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
        new AsyncCallback(ReceiveCallback_onQuery), state);
    }
else
{
    // All the data has arrived; put it in response.
    response_onQueryHistory = ByteArrayToObject(functionThatCombinesBytes(dataReceivedQueue));

    // Signal that all bytes have been received.
    receiveDoneQuery.Set();
}

state.buffer is buffer where data are received. buffer is a byte array of size 4096. state is of type StateObject.

ByteArrayToObject(byte []) takes care of deserializing the data received and converting it back to its object form

functionThatCombinesBytes(Queue) this function will receive a Queue of bytes and will "combine" all the bytes into one byte array

回答1:

Just because you are calling BeginReceive with a buffer of a particular size, doesn't mean that it will necessarily entirely fill the buffer, so it's very likely that some of your queued buffers will actually only be partially filled with received data, and the remainder being zeros, this will almost certainly corrupt your combined stream if you simply concatenate them together since you're not also storing the number of bytes actually read into the buffer. You also appear to be reusing the same buffer each time, so you'll just be overwriting already-read data with new data.

I would therefore suggest replacing your dataReceivedQueue with a MemoryStream, and using something like:

if (bytesRead > 0)
    {
        // There might be more data, so store the data received so far.
        memoryStream.Write(state.buffer, 0, bytesRead);

        // Get the rest of the data.
        client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
        new AsyncCallback(ReceiveCallback_onQuery), state);
    }
else
{
    // All the data has arrived; put it in response.
    response_onQueryHistory = ByteArrayToObject(memoryStream.ToArray());

    // Signal that all bytes have been received.
    receiveDoneQuery.Set();
}

回答2:

First of all, unless your dataReceivedQueue's type implements its own (or overrides Queue's) Enqueue method, your state.buffer would be rewritten with each client.BeginReceive call.

You can simply add a MemoryStream member to your StateObject and append bytes to it as they come:

state.rawData.Seek(0, SeekOrigin.End);
state.rawData.Write(state.buffer, 0, bytesRead);

回答3:

First of all, you need to not only store the byte array, but also the number of bytes in the arrays that are actually valid. For example, each receive may not fully fill the buffer, thus the number of bytes is returned (bytesRead in your code).

If you had this, you could calculate the size of the final buffer by summing up the number of received bytes for each "batch".

After that you can - in a loop - use Array.Copy to copy a "batch" to a specified position with a specified length into the target array.

For example, this could look like this:

// Batch is a class that contains the batch byte buffer and the number of bytes valid
int destinationPos = 0;
byte[] destination = new byte[<number of bytes in total>];
foreach (Batch b in batches)
{
    Array.Copy(b.Bytes, 0, destination, destinationPos, b.ValidLength);
}

来源：https://stackoverflow.com/questions/13763456/how-to-concatenate-or-combine-or-join-a-series-of-binarily-serialized-by

标签

sockets

bytearray

byte