Parallel download using Curl command line utility

Question

I want to download some pages from a website and I did it successfully using curl but I was wondering if somehow curl downloads multiple pages at a

Answer 1

As of 7.66.0, the curl utility finally has built-in support for parallel downloads of multiple URLs within a single non-blocking process, which should be much faster and more resource-efficient compared to xargs and background spawning, in most cases:

curl -Z 'http://httpbin.org/anything/[1-9].{txt,html}' -o '#1.#2'

This will download 18 links in parallel and write them out to 18 different files, also in parallel. The official announcement of this feature from Daniel Stenberg is here: https://daniel.haxx.se/blog/2019/07/22/curl-goez-parallel/