I'm trying to use Twisted in a sort of spidering program that manages multiple client connections. I'd like to maintain of a pool of about 5 clients working at one time. The functionality of each client is to connect to a specified IRC server that it gets from a list, enter a specific channel, and then save the list of the users in that channel to a database.
The problem I'm having is more architectural than anything. I'm fairly new to Twisted and I don't know what options are available for managing multiple clients. I'm assuming the easiest way is to simply have each ClientCreator instance die off once it's completed its work and have a central loop that can check to see if there's room to add a new client. I would think this isn't a particularly unusual problem so I'm hoping to glean some information from other peoples' experiences.
The best option is really just to do the obvious thing here. Don't have a loop, or a repeating timed call; just have handlers that do the right thing.
Keep a central connection-management object around, and make event-handling methods feed it the information it needs to keep going. When it starts, make 5 outgoing connections. Keep track of how many are in progress, maintain a list with them in it. When a connection succeeds (in connectionMade) update the list to remember the connection's new state. When a connection completes (in connectionLost) tell the connection manager; its response should be to remove that connection and make a new connection somewhere else. In the middle, it should be fairly obvious how to fire off a request for the names you need and stuff them into a database (waiting for the database insert to complete before dropping your IRC connection, most likely, by waiting for the Deferred to come back from adbapi).
Since each of your clients needs to update a database, instinctively I think I'd piggyback off the connection pool -- see here for more (the whole doc is recommended for some important design patterns that often emerge when using twisted).
I don't know if you are forced to use Twisted, otherwise you might want to give Gevent a try.
来源：https://stackoverflow.com/questions/1365737/managing-multiple-twisted-client-connections