Python requests module is very slow on specific machine

Question

I\'ve experienced too slow execution of Python requests on some machines and with specific user while other tools (for instance curl) are quite fast. Strange thing is that i

Answer 1

There could be many things slowing the request down. Anything from DNS lookup, throttling etc.

Try getting some more information by turning requests debug logging on

logging.basicConfig() 
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

Answer 2

For anyone reading this and using localhost in their URL, I resolved this issue by changing it to 127.0.0.1.

If this resolves the problem, it's a DNS problem, and it's not a requests problem.

Answer 3

Had similar problem. After investigation determined that python's internal call to socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): was causing the delay.

In my configuration, I was on a Mac, OS/X 10.11. host was dev.local, and I had the relevant entry in /etc/hosts, pointing to a virtual box machine setup on host-only network.

192.168.56.101 dev.local

After banging my head for an hour or two, realized that hostnames ending with .local was the actual issue (despite the /etc/hosts entry).

This eventually lead me to https://superuser.com/questions/539849/long-lookup-times-for-local-in-hosts-file. And voilà, avoiding made up host names with .local was all I needed.

I do realize this isn't the problem here -- 10.50.30.216 obviously doesn't end with .local! But since this question was one of the most relevant ones I ran into looking for an answer to my problem, I thought I'd post an answer here.

Answer 4

I had a similar problem... After a lot of debugging with authors, we concluded that the cause lies in proxy detection code...

So, once I implemented this, it resolved my similar problem: requests: how to disable / bypass proxy

Authors are aware of the problem, and they have fix in works, and it should be released soon: https://github.com/kennethreitz/requests/pull/2992 ...

Hope it helps...

cheers Jaka