A time-out occurs during the first RPC call to a server yet subsequest requests succeed. The server times-out on the response because upon first call it loads the libraries
I suggest faking a connection just before opening up the server. That will ensure that (most) relevant lazy initialisation will have been performed.