HTTP error 400 in R, error handling, How to retry instead of forcing to stop?

Question

I am calling API to download files using curl package in R. Due to unknown reason, sometimes the connection breaks with error code: Error in curl_download

Answer 1

There's a Hadley package designed exactly for this

require(httr)
maxTimes <- 10
testFilename <- "testfile.txt"

for (url in allUrl) {
  RETRY(verb = "GET", url = url, times = maxTimes,
    quiet = FALSE, terminate_on = NULL)
}

Specifially for file downloads with authentication, you can replace the RETRY command with:

GET(url, write_disk(path=testFilename, overwrite=TRUE), authenticate("user", "passwd"))