Go http.Get, concurrency, and “Connection reset by peer”

Question

I have between 1000-2000 webpages to download from one server, and I am using go routines and channels to achieve a high efficiency. The problem is that every time I run my

Answer 1

It might be possible that the server from which you are downloading the webpages has some type of throttling mechanism which prevents more than a certain number of requests per second/(or similar) from a certain ip?. Try limiting to maybe 100 requests per second or adding sleep between requests. Connection reset by peer is basically server denying you service. (What does "connection reset by peer" mean?)