Go http.Get, concurrency, and “Connection reset by peer”

Question

I have between 1000-2000 webpages to download from one server, and I am using go routines and channels to achieve a high efficiency. The problem is that every time I run my

Answer 1

I had good results by setting the MaxConnsPerHost option on transport...

cl := &http.Client{
    Transport: &http.Transport{MaxConnsPerHost: 50}
}

MaxConnsPerHost optionally limits the total number of connections per host, including connections in the dialing, active, and idle states. On limit violation, dials will block.

https://golang.org/pkg/net/http/#Transport.MaxConnsPerHost

EDIT: To clarify, this option was released in Go 1.11 which was not available at the time of @AG1's or @JimB's answers above, hence me posting this.