Multiple urllib2 connections

问题

I want to download multiple images at the same time. For that I'm using threads, each one downloading an image, using urllib2 module. My problem is that even if threads starts (almost) simultaneously, the images are downloaded one by one, like in a single-threaded environment.

Here is the threaded function:

def updateIcon(self, iter, imageurl):
    req = urllib2.Request('http://site.com/' + imageurl)
    response = urllib2.urlopen(req)
    imgdata = response.read()
    gobject.idle_add(self.setIcon, iter, imgdata)

Debugging my code I found that downloads seems to get stuck at "response = urllib2.urlopen(req)" line. What's the problem? It's because the threading module or urllib2? How I can fix that?

Thank you in advance

回答1:

Consider using urllib3. It supports connection pooling and multiple concurrent requests via processes (not threads). It should solve this problem. Be careful to garbage collect connection pools if you contact many different sites, since each site gets its own pool.

回答2:

In my experience, multithreads of CPython seems to make better performance than those of sigle thread. Because CPython has thread implementation based on kernel thread. But the difference is little, because of GIL(Global Interpreter Lock). Substitute multiprocessing for multithreading. It's easy. Both have similar interface.

来源：https://stackoverflow.com/questions/4139988/multiple-urllib2-connections