SSLError: bad handshake, Python requests

Question

I am consuming Ebay Trading APIs using Ebay python sdk which is eventually sing python-requests for making API calls.

All was working fine, but since last few days I

Answer 1

My guess is this is related to Python Urllib2 SSL error, i.e. a problem of multiple trust path in the underlying implementation of OpenSSL. See there for the details of the problem.

To fix this without making changes to your trusted CA's you would need to have a fixed OpenSSL, i.e. OpenSSL 1.0.2. Or you could add some of the older CA certificates back to your trust store.

1. pass argument verify=False
2. pass CA certificate
3. append you CA certificate in cacert.pem file (I tried this, didn't work)

... This is bad in security point of view.

While verify=False is bad for security because it disables validation the other options are not bad because they only add additional trust anchors but keep validation enabled.

Why SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed' is in traceback, when I have disabled SSL3.

Even if it talks about SSLv3 there it does not mean it. TLS and SSLv3 share a lot of functionality, i.e. TLS 1.0 is actually SSL 3.1. Thus lots of the SSL3_* functions in the OpenSSL code are used with TLS 1.x too which causes these irritating messages.

Answer 2

I think it's related to this section of https://pypi.python.org/pypi/certifi/

1024-bit Root Certificates

Browsers and certificate authorities have concluded that 1024-bit keys are unacceptably weak for certificates, particularly root certificates. For this reason, Mozilla has removed any weak (i.e. 1024-bit key) certificate from its bundle, replacing it with an equivalent strong (i.e. 2048-bit or greater key) certifiate from the same CA. Because Mozilla removed these certificates from its bundle, certifi removed them as well.

Unfortunately, old versions of OpenSSL (less than 1.0.2) sometimes fail to validate certificate chains that use the strong roots. For this reason, if you fail to validate a certificate using the certifi.where() mechanism, you can intentionally re-add the 1024-bit roots back into your bundle by calling certifi.old_where() instead. This is not recommended in production: if at all possible you should upgrade to a newer OpenSSL. However, if you have no other option, this may work for you.

My test goes like this:

root@43b7ec35c240:/usr/src/app# cat /etc/debian_version
8.2
root@43b7ec35c240:/usr/src/app# cat test.py
import logging

import requests

logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

requests.get('https://www.google.com/')
root@43b7ec35c240:/usr/src/app# python test.py
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): www.google.com
DEBUG:requests.packages.urllib3.connectionpool:"GET / HTTP/1.1" 302 263
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): www.google.com.hk
DEBUG:requests.packages.urllib3.connectionpool:"GET /?gfe_rd=cr&ei=XXKbVqqBD8WM8Qe3v7-YAQ HTTP/1.1" 200 None
root@43b7ec35c240:/usr/src/app# openssl version
OpenSSL 1.0.1k 8 Jan 2015
root@43b7ec35c240:/usr/src/app# pip install certifi
Collecting certifi
  Using cached certifi-2015.11.20.1-py2.py3-none-any.whl
Installing collected packages: certifi
Successfully installed certifi-2015.11.20.1
root@43b7ec35c240:/usr/src/app# python test.py
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): www.google.com
Traceback (most recent call last):
  File "test.py", line 16, in <module>
    requests.get('https://www.google.com/')
  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 69, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 433, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
root@43b7ec35c240:/usr/src/app# pip uninstall certifi
Uninstalling certifi-2015.11.20.1:
  /usr/local/lib/python2.7/site-packages/certifi-2015.11.20.1.dist-info/DESCRIPTION.rst
  /usr/local/lib/python2.7/site-packages/certifi-2015.11.20.1.dist-info/METADATA
  /usr/local/lib/python2.7/site-packages/certifi-2015.11.20.1.dist-info/RECORD
  /usr/local/lib/python2.7/site-packages/certifi-2015.11.20.1.dist-info/WHEEL
  /usr/local/lib/python2.7/site-packages/certifi-2015.11.20.1.dist-info/metadata.json
  /usr/local/lib/python2.7/site-packages/certifi-2015.11.20.1.dist-info/top_level.txt
  /usr/local/lib/python2.7/site-packages/certifi/__init__.py
  /usr/local/lib/python2.7/site-packages/certifi/__init__.pyc
  /usr/local/lib/python2.7/site-packages/certifi/__main__.py
  /usr/local/lib/python2.7/site-packages/certifi/__main__.pyc
  /usr/local/lib/python2.7/site-packages/certifi/cacert.pem
  /usr/local/lib/python2.7/site-packages/certifi/core.py
  /usr/local/lib/python2.7/site-packages/certifi/core.pyc
  /usr/local/lib/python2.7/site-packages/certifi/old_root.pem
  /usr/local/lib/python2.7/site-packages/certifi/weak.pem
Proceed (y/n)? y
  Successfully uninstalled certifi-2015.11.20.1
root@43b7ec35c240:/usr/src/app# python test.py
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): www.google.com
DEBUG:requests.packages.urllib3.connectionpool:"GET / HTTP/1.1" 302 263
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): www.google.com.hk
DEBUG:requests.packages.urllib3.connectionpool:"GET /?gfe_rd=cr&ei=AnObVtG4BcGM8QeC6r-YCw HTTP/1.1" 200 None

Then I try to update my libssl package by doing:

# curl http://apt.wikimedia.org/autoinstall/keyring/wikimedia-archive-keyring.gpg | apt-key add -

# echo "deb http://apt.wikimedia.org/wikimedia jessie-wikimedia backports" >> /etc/apt/sources.list

# apt-get update
# apt-get install libssl1.0.0

# openssl version
OpenSSL 1.0.1k 8 Jan 2015 (Library: OpenSSL 1.0.2e 3 Dec 2015)

After that, even with certifi installed, I am all good.

# pip freeze |grep certifi
certifi==2015.11.20.1

# python test.py
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): www.google.com
DEBUG:requests.packages.urllib3.connectionpool:"GET / HTTP/1.1" 302 263
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): www.google.com.hk
DEBUG:requests.packages.urllib3.connectionpool:"GET /?gfe_rd=cr&ei=4XSbVt2gK8OM8Qe1tICIBA HTTP/1.1" 200 None