I have built a WebSockets server that acts as a chat message router (i.e. receiving messages from clients and pushing them to other clients according to a client ID).
It is a requirement that the service be able to scale to handle many millions of concurrent open socket connections, and I wish to be able to horizontally scale the server.
The architecture I have had in mind is to put the websocket server nodes behind a load balancer, which will create a problem because clients connected to different nodes won't know about each other. While both clients A and B enter via the LoadBalancer, client A might have an open connection with node 1 while client B is connected to node 2 - each node holds it's own dictionary of open socket connections.
To solve this problem, I was thinking of using some MQ system like ZeroMQ or RabbitMQ. All of the websocket server nodes will be subscribers of the MQ server, and when a node gets a request to route a message to a client which is not in the local connections dictionary, it will pub-lish a message to the MQ server, which will tell all the sub-scriber nodes to look for this client and issue the message if it's connected to that node.
Q1: Does this architecture make sense?
Q2: Is the pub-sub pattern described here really what I am looking for?
ZeroMQ would be my option - both architecture-wise & performance-wise
-- fast & low latency ( can measure your implementation performance & overheads, down to sub [usec] scale )
-- broker-less ( does not introduce another point-of-failure, while itself can have { N+1 | N+M } self-healing architecture )
-- smart Formal Communication Pattern primitives ready to be used ( PUB / SUB is the least cardinal one )
-- fair-queue & load balancing architectures built-in ( invisible for external observer )
-- many transport Classes for server-side internal multi-process / multi-threading distributed / parallel processing
-- ready to almost linear scaleability
Adaptive node re-discovery
This is a bit more complex subject. Your intention to create a feasible architecture will have to drill down into more details to solve.
- Node authentication vs. peer-to-peer messaging
- Node (re)-discovery vs. legal & privacy issues
- Node based autonomous self-organising Agents vs. needs for central policy enforcement
来源：https://stackoverflow.com/questions/25701094/scaling-websockets-with-a-message-queue