Python urllib2: Cannot assign requested address

Question

I am sending thousands of requests using urllib2 with proxies. I have received many of the following error on execution:

urlopen error [Errno 9         


        

Answer 1

As mhawke suggested, the issue of TIME_WAIT seems most likely. The system wide fix for your situation can be to adjust kernel parameters so such connections are cleaned up more often. Two options:

$ sysctl net.ipv4.tcp_tw_recycle=1

This will let the kernel reuse connections in TIME_WAIT state. This may cause issues with NAT setups. Another one is:

$ sysctl net.ipv4.tcp_max_orphans=8192
$ sysctl net.ipv4.tcp_orphan_retries=1

This tells the kernel to keep at most 8192 connections not attached to any user process and only retry once before killing TCP connections.

Note that these are not permanent changes. Add the setting to /etc/sysctl.conf to make them permanent.

http://code.google.com/p/lusca-cache/issues/detail?id=89#c4
http://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.kernel.obscure.html