Curl - Request with wp_remote_get() responds 500, curl_exec responds 200

Question

I am getting an odd server php curl error in both my local and production servers (Ubuntu 14.04.2 LTS, PHP 5.5.9-1ubuntu4.11, Apache 2.4.7).

Basically, a curl request to a remote API returns a status code 500 response, ONLY in wp_remote_get(), where it returns status 200 in both curl_exec() and a browser request.

My debug code:

<?php
$url = 'https://yoast.com?edd_action=activate_license&license=my-license-key-here&item_name=WooCommerce+Yoast+SEO&url=https://google.com';

// this return status 200:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);
curl_close($ch);
echo '<pre>' . print_r($result, true) . '</pre>';

// this return status 500:
$testResp = wp_remote_get($url);
echo '<pre>' . print_r($testResp, true) . '</pre>';

I cannot figure out why it responds 500 for wp_remote_get(). I've tried adjusting args passed to wp_remote_get(), but still a 500 with it.

I've also disabled all plugins in debugging.

Any Ideas?

Answer 1

OK, after a bit of debugging, I believe the issue is the default User-Agent string Wordpress sets in wp-includes/class-http.php, set when creating an http request for wp_remote_get().

The option has a filter, but the default is created like so:

'user-agent' => apply_filters( 'http_headers_useragent', 'WordPress/' . $wp_version . '; ' . get_bloginfo( 'url' ) ),

So in my case, the 'user-agent' header value was: "Wordpress/4.3.1; http://myurl.com"

When I hook into the filter http_headers_useragent and return an empty string, or even a different user-agent string such as: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8) AppleWebKit/535.6.2 (KHTML, like Gecko) Version/5.2 Safari/535.6.2', the request will return a successful 200 response.

Not sure if the semicolon is the true culprit, but if I remove it and set the user-agent string to just "Wordpress/4.3.1", the request is successful as well.

Answer 2

I had the same problems - wp_remote_get was not working while the classic Curl calls were making the calls. Indeed the problem is on 'user agent' . This is my solution based on "chuuke" findings

 $args = array(
            'user-agent'  =>  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8) AppleWebKit/535.6.2 (KHTML, like Gecko) Version/5.2 Safari/535.6.2',
        ); 
 $data = wp_remote_get($new_url_signed,$args);

Thanks