Make parallel libcurl HTTP requests

Question

I have a question regarding the safety of performing parallel HTTP-requests using libcurl (C++). When reading this question, please bear in mind I have limited knowledge about HTTP-requests in general.

Basically, let's say I have two (or more) threads, each thread makes a HTTP-request once per second. (All the requests made are to the same server). How does my program (or something else?) keep track of what HTTP-response belongs to which tread? I mean, can I be sure that if request A was sent from thread 1, and request B from thread 2 at the same time, and the responses are retrived at the same time, the correct reponse (response A) goes to thread 1 and response B to thread 2?

Please excuse my ignorance in this matter.

Thanks.

Answer 1

First of all libcurl is thread safe:

libcurl is designed and implemented entirely thread safe

As pointed out by this official documentation all you need to do is:

Never share libcurl handles between multiple threads. You should only use one handle in one single thread at any given time.

In addition there is also this official FAQ entry that provides some more precisions, e.g if you plan to use SSL:

If you use a OpenSSL-powered libcurl in a multi-threaded environment, you need to provide one or two locking functions

As you can see there is an official example which illustrates a multi-threaded use of the easy handles: see multithread.c:

/* This is the callback executed when a new threads starts */
static void *pull_one_url(void *url)
{
  CURL *curl;

  curl = curl_easy_init();
  curl_easy_setopt(curl, CURLOPT_URL, url);
  curl_easy_perform(curl); /* ignores error */ 
  curl_easy_cleanup(curl);

  return NULL;
}

/* ... */

/* This is the main loop that creates `NUMT` threads for parallel fetching */
for(i=0; i< NUMT; i++) {
    error = pthread_create(&tid[i],
                           NULL, /* default attributes please */ 
                           pull_one_url,
                           (void *)urls[i]);
  /* ... */
}

So feel free to give this example a try.

At last keep in mind that libcurl also provides a so-called multi interface which offers multiple transfers using a single thread. You may find it convenient too according to your use case.

EDIT

Regarding OpenSSL + multi-threads there are specific official examples that might help:

• opensslthreadlock.c
• threaded-ssl.c