Curl retry mechanism

Question

I have a script I run to deploy 2 web services and a front-end application. The script calls a method that checks to see if the 2 back-end services are up and running. If so

Answer 1

If you use --max-time 10 this will kill your connection before 10 seconds so its not useful when downloading.

Following will retry 300 times with 5sec delay and 30sec connection timeout

curl \
    --connect-timeout 30 \
    --retry 300 \
    --retry-delay 5 \
    URL

Answer 2

Just to clear up any possible confusion for anyone new...

Please see: https://curl.haxx.se/docs/manpage.html

--retry

If a transient error is returned when curl tries to perform a transfer, it will retry this number of times before giving up. Setting the number to 0 makes curl do no retries (which is the default). Transient error means either: a timeout, an FTP 4xx response code or an HTTP 5xx response code.

When curl is about to retry a transfer, it will first wait one second and then for all forthcoming retries it will double the waiting time until it reaches 10 minutes which then will be the delay between the rest of the retries. By using --retry-delay you disable this exponential backoff algorithm. See also --retry-max-time to limit the total time allowed for retries.

If this option is used several times, the last one will be used.

Added in 7.12.3.

Answer 3

The following statement will retry 5 times or a maximum of 40 seconds with a connection timeout of 5 seconds, and no exponential backoff policy

curl --connect-timeout 5 \
    --max-time 10 \
    --retry 5 \
    --retry-delay 0 \
    --retry-max-time 40 \
    'http://your_url'


--max-time 10     (how long each retry will wait)
--retry 5         (it will retry 5 times)
--retry-delay 0   (an exponential backoff algorithm)
--retry-max-time  (total time before it's considered failed)

Note that there is also a --retry-connrefused (since curl 7.52.0) that retries even when the connection is refused and --retry-all-errors (since curl 7.71.0) which "is the sledgehammer of retrying".