boost::asio cleanly disconnecting

Question

Sometimes boost::asio seems to disconnect before I want it to, i.e. before the server properly handles the disconnect. I'm not sure how this is possible because the client seems to think its fully sent the message, yet when the server emits the error its not even read the message header... During testing this only happens maybe 1 in 5 times, the server receives the client shut down message, and disconnects the client cleanly.

The error: "An existing connection was forcibly closed by the remote host"

The client disconnecting:

void disconnect()
{
    boost::system::error_code error;
    //just creates a simple buffer with a shutdown header
    boost::uint8_t *packet = createPacket(PC_SHUTDOWN,0);
    //sends it
    if(!sendBlocking(socket,packet,&error))
    {
        //didnt get here in my tests, so its not that the write failed...
        logWrite(LOG_ERROR,"server",
            std::string("Error sending shutdown message.\n")
            + boost::system::system_error(error).what());
    }

    //actaully disconnect
    socket.close();
    ioService.stop();
}
bool sendBlocking(boost::asio::ip::tcp::socket &socket,
    boost::uint8_t *data, boost::system::error_code* error)
{
    //get the length section from the message
    boost::uint16_t len = *(boost::uint16_t*)(data - 3);
    //send it
    asio::write(socket, asio::buffer(data-3,len+3),
        asio::transfer_all(), *error);
    deletePacket(data);
    return !(*error);
}

The server:

void Client::clientShutdown()
{
    //not getting here in problem cases
    disconnect();
}
void Client::packetHandler(boost::uint8_t type, boost::uint8_t *data,
    boost::uint16_t len, const boost::system::error_code& error)
{
    if(error)
    {
        //error handled here
        delete[] data;
        std::stringstream ss;
        ss << "Error recieving packet.\n";
        ss << logInfo() << "\n";
        ss << "Error: " << boost::system::system_error(error).what();
        logWrite(LOG_ERROR,"Client",ss.str());

        disconnect();
    }
    else
    {
        //call handlers based on type, most will then call startRead when
        //done to get the next packet. Note however, that clientShutdown
        //does not
        ...
    }
}



void startRead(boost::asio::ip::tcp::socket &socket, PacketHandler handler)
{
    boost::uint8_t *header = new boost::uint8_t[3];
    boost::asio::async_read(socket,boost::asio::buffer(header,3),
        boost::bind(&handleReadHeader,&socket,handler,header, 
        boost::asio::placeholders::bytes_transferred,boost::asio::placeholders::error));
}
void handleReadHeader(boost::asio::ip::tcp::socket *socket, PacketHandler handler,
    boost::uint8_t *header, size_t len, const boost::system::error_code& error)
{
    if(error)
    {
        //error "thrown" here, len always = 0 in problem cases...
        delete[] header;
        handler(0,0,0,error);
    }
    else
    {
        assert(len == 3);
        boost::uint16_t payLoadLen  = *((boost::uint16_t*)(header + 0));
        boost::uint8_t  type        = *((boost::uint8_t*) (header + 2));
        delete[] header;
        boost::uint8_t *payLoad = new boost::uint8_t[payLoadLen];

        boost::asio::async_read(*socket,boost::asio::buffer(payLoad,payLoadLen),
            boost::bind(&handleReadBody,socket,handler,
            type,payLoad,payLoadLen,
            boost::asio::placeholders::bytes_transferred,boost::asio::placeholders::error));
    }
}
void handleReadBody(ip::tcp::socket *socket, PacketHandler handler,
    boost::uint8_t type, boost::uint8_t *payLoad, boost::uint16_t len,
    size_t readLen, const boost::system::error_code& error)
{
    if(error)
    {
        delete[] payLoad;
        handler(0,0,0,error);
    }
    else
    {
        assert(len == readLen);
        handler(type,payLoad,len,error);
        //delete[] payLoad;
    }
}

Answer 1

I think you should probably have a call to socket.shutdown(boost::asio::ip::tcp::socket::shutdown_both, ec) in there before the call to socket.close().

The boost::asio documentation for basic_stream_socket::close states:

For portable behaviour with respect to graceful closure of a connected socket, call shutdown() before closing the socket.

This should ensure that any pending operations on the socket are properly cancelled and any buffers are flushed prior to the call to socket.close.

Answer 2

I have tried to do this with both the close() method and the shutdown() method

socket.shutdown(boost::asio::ip::tcp::socket::shutdown_both, ec)

The shutdown method is the best of the two. However, I find that using the destructor of the ASIO socket is the clean way to do it as ASIO takes care of it all for you. So your goal is to just let the socket fall out of scope. Now, you can do this easily using a shared_ptr and resetting the shared_ptr to a fresh socket or null. this will call the destructor of the ASIO socket and life is good.

Answer 3

Maybe this is what is happening:

• Client send disconnect packet
• Client shuts socket down
• Server read handler gets called, but there is an error associated with the shutdown packet because the socket is already closed.

I see in your read handlers, if there is an error, you never check to see if your shutdown packet is there. Maybe it is. Basically what I'm saying is maybe your client sometimes is able to send both the close and the shutdown packet before the server has a chance to process them separately.

Answer 4

Use async_write() and put socket.close() inside of write handler. This will make sure packet is processed by boost asio and not neglected in the middle of processing (because of close() calls).

Answer 5

I have a very similar issue. I believe it's related to Windows recycling connections. Is the following familiar?

• you get this error immediately upon starting the program but not after a connection is established?
• The error never happens if you wait more than 4 minutes before restarting your application?

The tcp specs specify that by default it should wait four minutes for the final acknowledgment when a tcp connection is closed. You can see these connections in FIN_WAIT state using netstat. The Windows OS detects when you try to connect to the exact same system and takes these partially closed connections and recycles them. Your second invocation of the program gets the 'closed' connection left behind by the first run. It gets the next acknowledge and then really closes.