SSLHandshakeException: Received fatal alert: handshake_failure after Java 6 -> 8 upgrade

Question

We\'ve recently updated a project from Java 6 to Java 8 and now we\'ve hit a brick wall regarding SSL handshake.

The service layer uses a client to request and recei

Answer 1

Alright, so we've got it working now. I'll post the answer here in case someone might need it some day.

We have tried quite a few things, so I'm not exactly sure what actually needed to be done in order for it to work, but here are some of the things we changed in the process.

 -Dhttps.protocols=SSLv3,TLSv1,SSLv2Hello

Adding this flag led to the certificate being presented in the javax.net.debug logs, but we were still getting SSLHandshakeException. It seems like the only cipher the server would accept was SSL_RSA_WITH_RC4_128_MD5. This was not picked automatically by our client.

-Dhttps.cipherSuites=SSL_RSA_WITH_RC4_128_MD5

We added this flag to restrict the cipher suits for the client. Together with setting the same restriction programmatically (not sure if both are needed):

socket.setEnabledCipherSuites(new String[] {"SSL_RSA_WITH_RC4_128_MD5"});

Restricting the available cipher suites to the only one that the client could use, made the client pick that cipher suite.

We also did the following changes the the jre/lib/security/java.security file to enable SSLv3 and the SSL_RSA_WITH_RC4_128_MD5 cipher:

• remove SSLv3 from jdk.tls.disabledAlgorithms
• add SSL_RSA_WITH_RC4_128_MD5 to jdk.tls.legacyAlgorithms

This is probably not recommended for production servers, since SSLv3 is obsolete, and the cipher is very old and outdated, but in this case security is not a huge concern (internal application use).

These posts were also helpful to me:

• SSL connection failing for Java 7
• Java 7 (acting as client) SSL handshake failure with keystore and truststore that worked in Java 6
• Received fatal alert: handshake_failure through SSLHandshakeException
• Java cipher suites

Answer 2

String testURL = "https://api.chargeio.com/status";
SSLContext sslcontext = SSLContext.getInstance("TLSv1.2");
sslcontext.init(null, null, null);
try {
    SSLConnectionSocketFactory socketFactory = new
    SSLConnectionSocketFactory(sslcontext,
        SSLConnectionSocketFactory.BROWSER_COMPATIBLE_HOSTNAME_VERIFIER); // Socket
    HttpClient client =
        HttpClients.custom().setSSLSocketFactory(socketFactory).build();
    HttpGet httpget = new HttpGet(testURL);
    HttpResponse response = client.execute(httpget);
    System.out.println(EntityUtils.toString(response.getEntity()));
    System.out.println("Response Code (Apache): " + response.getStatusLine().getStatusCode());
} catch (Exception e) {
    System.err.println("HttpsURLConnection Failed");
    e.printStackTrace();
}

Answer 3

My guess is that your server is too broken to deal properly with TLS 1.2 handshakes. Usually a server which does not understand TLS 1.2 should reply with the best version it can but your server does not.

Broken servers like this exists and browsers try to work around these by retrying with a lower TLS version. Outside of browsers these retries are not common so these clients simply fail.

While I cannot say for sure that the server is broken the certificate which expired 15 years ago and was signed with the long broken MD5 algorithm suggest that you have to do with a very old and neglected installation. So chances are high that it never occurred to the developers of the original server that something like TLS 1.2 might ever exist or that the it croaks on one of the TLS extensions used in the TLS 1.2 handshake.

Since this issue is not related to the validation of the certificate all attempts to fix the issue by fiddling in the area of validation are useless. You might have more success if you enforce the use of TLS 1.1 or TLS 1.1 instead of TLS 1.2. You might try to do this with the -Dhttps.protocols=TLSv1,TLSv1.1 or -Dhttps.protocols=TLSv1 settings.