Why does cURL return an empty string?

Question

I\'m having a problem with PHP\'s cURL returning an empty string with some URL\'s. I\'m trying to parse the OG metadata of different webpages and it works with all website

Answer 1

(That other answer is me also)

This is what did it for me. It was looking for SSL verificaiton, which I happened to not need in this specific case.

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);

Answer 2

These 5 lines did the magic for me.

   curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.52 Safari/537.17');
   curl_setopt($ch, CURLOPT_AUTOREFERER, true); 
   curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
   curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
   curl_setopt($ch, CURLOPT_VERBOSE, 1);

Answer 3

My guess is that a site like the New York times has protection against such behavior. Most likely this is based on the user agent, which you can fake as so:

curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.52 Safari/537.17');

This is the most common agent btw.

Answer 4

This is what did it for me. It was looking for SSL verificaiton, which I happened to not need in this specific case.

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);