Can servers block curl requests?

Question

I am working on ZOHO API and trying to update the record using cURL. I tried different cURL variations, but it always returns "false". But when I call the same URL using a browser, it works.

Is there any way they can block cURL requests? Is there any other way I can call that URL using a POST or maybe a GET request?

The cURL code I have tried is as below:

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, 0);
$data = curl_exec($ch);
curl_close($ch);

Answer 1

Servers cannot block cURL requests per se, but they can block any request that they do not like. If the server checks for some parameters that your cURL request does not satisfy, it could decide to respond differently.

In the vast majority of cases, this difference in behavior is triggered by the presence (or absence) and values of the HTTP request headers. For example, the server might check that the User-Agent header is present and has a valid value (it could also check lots of other things).

To find out what the HTTP request coming from the browser looks like, use an HTTP debugging proxy like Fiddler or your browser's developer tools.

To add your own headers to your cURL request, use

curl_setopt($ch, CURLOPT_HTTPHEADER, array('HeaderName: HeaderValue'));

Answer 2

Many web servers want to block HTTP requests forged by something else than a browser, to prevent bots abuses. If you want to simulate/pretend your request from a browser, you at least have to:

1. Pass the exact same headers than your browsers (use ie Firebug to get them)

   curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
   

2. Change the user agent (name of the browser)

   curl_setopt($ch, CURLOPT_USERAGENT, $agent);
   

3. Enable cookies (for eg redirection and session handling)

   curl_setopt ($ch, CURLOPT_COOKIEJAR, $file);
   curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
   

4. Add referers

   curl_setopt($curl, CURLOPT_REFERER, 'http://www.google.com');
   curl_setopt($curl, CURLOPT_AUTOREFERER, true);
   

And pray you haven't missed anything!

Answer 3

To answer your question "Is there any way they can block CURL requests?": Yes, in fact one may detect a cURL request by reading the User-Agent header.

You can change the user agent by calling curl_setopt($ch, CURLOPT_USERAGENT, 'My user agent string!');.

Answer 4

Just to elaborate a little more on this, you can use the curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0'); or something like that to fake the user agent. In this case, the server would think a Firefox browser was making the request.