Make parallel libcurl HTTP requests

Question

I have a question regarding the safety of performing parallel HTTP-requests using libcurl (C++). When reading this question, please bear in mind I have limited knowledge abo

Answer 1

First of all libcurl is thread safe:

libcurl is designed and implemented entirely thread safe

As pointed out by this official documentation all you need to do is:

Never share libcurl handles between multiple threads. You should only use one handle in one single thread at any given time.

In addition there is also this official FAQ entry that provides some more precisions, e.g if you plan to use SSL:

If you use a OpenSSL-powered libcurl in a multi-threaded environment, you need to provide one or two locking functions

As you can see there is an official example which illustrates a multi-threaded use of the easy handles: see multithread.c:

/* This is the callback executed when a new threads starts */
static void *pull_one_url(void *url)
{
  CURL *curl;

  curl = curl_easy_init();
  curl_easy_setopt(curl, CURLOPT_URL, url);
  curl_easy_perform(curl); /* ignores error */ 
  curl_easy_cleanup(curl);

  return NULL;
}

/* ... */

/* This is the main loop that creates `NUMT` threads for parallel fetching */
for(i=0; i< NUMT; i++) {
    error = pthread_create(&tid[i],
                           NULL, /* default attributes please */ 
                           pull_one_url,
                           (void *)urls[i]);
  /* ... */
}

So feel free to give this example a try.

At last keep in mind that libcurl also provides a so-called multi interface which offers multiple transfers using a single thread. You may find it convenient too according to your use case.

EDIT

Regarding OpenSSL + multi-threads there are specific official examples that might help:

• opensslthreadlock.c
• threaded-ssl.c