Limiting amount of data read in the response to a HTTP GET request

Question

I\'m scraping HTML pages and have set up a HTTP client like so:

client := *http.Client{
        Transport: &http.Transport{
            Dial: (&net.D         


        

Answer 1

Use an io.LimitedReader

A LimitedReader reads from R but limits the amount of data returned to just N bytes.

limitedReader := &io.LimitedReader{R: response.Body, N: limit}
body, err := ioutil.ReadAll(limitedReader)

or

body, err := ioutil.ReadAll(io.LimitReader(response.Body, limit))