500 error with LWP:UserAgent

Question

Solved ! thanks for every help i had :)

Found http://stackoverflow.com/questions/8026524/how-do-i-force-lwp-to-use-cryptssleay-for-https-requests and solved my issue with

use Net::SSL (); # From Crypt-SSLeay
BEGIN {
  $Net::HTTPS::SSL_SOCKET_CLASS = "Net::SSL"; # Force use of Net::SSL
  $ENV{HTTPS_PROXY} = 'http://10.0.3.1:3128';

}

I tried a thousand different ways to connect to the URL

https://sis-t.redsys.es:25443/sis/entradaXMLEntidad/

and i seem unable to get something else than a 500 error.

Latest code i tried was

require LWP::UserAgent;

## Code added according to Sinan Ünür s comment
use Net::SSLeay;
$Net::SSLeay::trace = 2;
## End of code added

my $ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 0});
$ua->agent("Mozilla/8.0");
$ua->timeout(10);
my $url = 'https://sis-t.redsys.es:25443/sis/entradaXMLEntidad/'

my $req = HTTP::Request->new( GET => $url);
$req->header( 'Accept' => 'text/html' );

# send request
my $res = $ua->request($req);

# check the outcome
if ( $res->is_success ) {
    print $res->decoded_content;
} else {
    print "Error: " . $res->status_line . "\n";
}

I tried with and without

$ua->proxy('https', 'http://myproxy');

I tried POST and GET (since in the end, I am going to have to access that URL with POST).

Of course I made sure I was able to get this page from a regular browser. I made sure my code always was able to connect to other pages with the form of

https + example.com:example-port + /path/to/another/page.

If anyone can help me to find what I am doing wrong, I'd be thankful.

EDIT the added code for Sinan Ünür's comment produced the following error

DEBUG: .../IO/Socket/SSL.pm:449: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:451: socket connected
DEBUG: .../IO/Socket/SSL.pm:469: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:504: using SNI with hostname sis-t.redsys.es
DEBUG: .../IO/Socket/SSL.pm:527: set socket to non-blocking to enforce timeout=10
DEBUG: .../IO/Socket/SSL.pm:550: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:560: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:570: handshake failed because socket did not became ready
Error: 500 Can't connect to sis-t.redsys.es:25443

the output was the same combining different mixes I thought of for the ua creation (with/out verification/SSL_version/...

my $ua = LWP::UserAgent->new; #(ssl_opts => { verify_hostname => 1, SSL_version => 'TLSv1' });

Since this was also spotted here : http://www.perlmonks.org/?node_id=1106004

---

EDIT2 : the code

#!/usr/bin/perl -w
use strict;

use Net::SSLeay;
$Net::SSLeay::trace = 2;

require LWP::UserAgent;

my $ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 0, SSL_version => 'TLSv1:!TLSv11:!TLSv12:!SSLv2:!SSLv3'});
$ua->agent("Mozilla/8.0");
$ua->timeout(10);

my $req = HTTP::Request->new( GET => "https://sis-t.redsys.es:25443/sis/entradaXMLEntidad/");
$req->header( 'Accept' => 'text/html' );

# send request
my $res = $ua->request($req);

# check the outcome
if ( $res->is_success ) {
    print $res->decoded_content;
} else {
    print "Error: " . $res->status_line . "\n";
}

works fine on a machine with perl 5.14.2 LWP:User::Agent 6.04 IO::Socket::SSL 1.76 Net::SSLeay 1.48

but not on mine with perl 5.14.2 LWP:User::Agent 6.06 IO::Socket::SSL 1.955 Net::SSLeay 1.55 and I seem unable to find more differences...

Answer 1

(Based on my answer from http://www.perlmonks.org/?node_id=1106240)

... sis-t.redsys.es:25443

This server has some serious issues. The default cipher set for the version of IO::Socket::SSL you use (1.76) is "ALL:!LOW". If this cipher set is used together with TLS1.0 or higher the connection simply hangs. You can verify this with s_client:

$ openssl s_client -cipher 'ALL:!LOW' -connect  sis-t.redsys.es:25443 -servername sis-t.redsys.es
CONNECTED(00000004)
... and there it hangs ...

This kind of problem is often seen in connection with older F5 load balancers in front of the server, which simply drop ClientHello packets greater than 255 bytes. If you would not add the servername option in the s_client call the request would magically succeed, because the packet is now smaller than 255 byte. Since version 1.962 (about a year ago) IO::Socket::SSL will use a smaller (and more secure) cipher set to work around such problems.

Crypt::SSLeay instead uses the default cipher set of OpenSSL, which makes the packet in this case just 248 bytes and thus small enough. That's why it works with Crypt::SSLeay. But please note, that Crypt::SSLeay does not make any verification of the hostname against the servers certificate and is thus open to man-in-the-middle attacks.

With current versions of IO::Socket::SSL the problem should be fixed, that is you should not need any special settings and you can leave verification enabled etc. Your latest code works thus without problems on my machine with IO::Socket::SSL 2.002.

But, since you are using a proxy you should use the latest versions of LWP::UserAgent and LWP::Protocol::https too, because proper proxy support with IO::Socket::SSL backend was only added to version 6.06 (it might be in 6.04 already if you use Debian or a spinoff like Ubuntu). While it looks like you are using LWP::UserAgent 6.06 you do not show the version of LWP::Protocol::https. This separate module must be version 6.06 too to have working proxy support.

Answer 2

First off, given that the site's certificate is not self-signed, there is no good reason for:

LWP::UserAgent->new(ssl_opts => { verify_hostname => 0});

This made me suspect that your problem may be related to Crypt::SSLeay, but I do not have a proxy set up to test this.

In any case, unless you find out the problem is either due to your crawler being blocked, or soemthing related to your proxy set up, I would ask that you run the following code:

#!/usr/bin/env perl

use strict;
use warnings;

use LWP::UserAgent;
use Net::HTTPS;
my $sc = $Net::HTTPS::SSL_SOCKET_CLASS;
my $v = eval "require $sc; \${${sc}::VERSION}";
print "$_\n" for $sc, $v;

and report the information here, along with your perl and LWP versions.

According to your comment, you are using:

• IO::Socket::SSL 1.955
• perl 5.14.2
• LWP::UserAgent 6.06

Please add

use Net::SSLeay;
$Net::SSLeay::trace = 2;

to the top of your original script, and update your question with the debugging output.