python requests is slow

Question

I am developing a download manager. Using the requests module in python to check for a valid link (and hopefully broken links). My code for checking link below:

Answer 1

Not all hosts support head requests. You can use this instead:

r = requests.get(url, stream=True)

This actually only download the headers, not the response content. Moreover, if the idea is to get the file afterwards, you don't have to make another request.

See here for more infos.

Answer 2

Don't use get that actually retrieves the file, use:

r = requests.head(url,allow_redirects=False)

Which goes from 6.9secs on my machine to 0.4secs