python: [Errno 10054] An existing connection was forcibly closed by the remote host

Question

I am writing python to crawl Twitter space using Twitter-py. I have set the crawler to sleep for a while (2 seconds) between each request to api.twitter.com. However, after

Answer 1

I know this is a very old question but it may be that you need to set the request headers. This solved it for me.

For example 'user-agent', 'accept' etc. here is an example with user-agent:

url = 'your-url-here'
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36'}
r = requests.get(url, headers=headers)