SSL certificate verification failure in python on Mavericks

Question

I'm stuck on a persistent SSL verification issue.

SSL: CERTIFICATE_VERIFY_FAILED

I discovered the error while building a Django app that had users authenticate using Mozilla Persona.

(python3.4)> import requests
(python3.4)> requests.get('https://verifier.login.persona.org')

I get a SSL: CERTIFICATE_VERIFY_FAILED tracing back from requests to urllib3 to ssl:

...
"/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/ssl.py", line 805, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598)

...
"/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests-2.4.1-py3.4.egg/requests/packages/urllib3/connectionpool.py", line 543, in urlopen
    raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598)

...
"/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests-2.4.1-py3.4.egg/requests/adapters.py", line 420, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598)

Difference between python3 and python2

Here's where it starts to get interesting: I don't get the same issue when using python2.7:

(python2.7)> import requests
(python2.7)> requests.get('https://verifier.login.persona.org')
<Response [200]>

My first thought was that the two versions of requests might be using different certs[1], so I was pretty surprised to find the two files were exactly the same:

(bash)$ diff `python3.4 -c "import requests; print(requests.certs.where())"` \
             `python2.7 -c "import requests; print requests.certs.where()"`
# no diff

Error recreated in openssl and solved using -CAFile

Interestingly, the issue is not limited to python3.4[2].

(bash)$ openssl s_client -connect github.com:443
...
Verify return code: 20 (unable to get local issuer certificate)

Edit A comment from Steffen informed me that this "debugging" method isn't actually informative, as s_client expects a -CApath in order to verify. However, the fact that I can specify the same certificate that the requests package is using and I don't get the same error is still interesting:

(bash)$ openssl s_client -connect github.com:443 \
        -CAfile `python3 -c 'import requests; print(requests.certs.where())'`
...
Verify return code: 0 (ok)

At this point, I'm completely out of my element. I don't know is if this is really an openssl issue, or something about OSX Mavericks[3]. Here's the version of openssl I'm using:

(bash)$ openssl version
OpenSSL 1.0.1f 6 Jan 2014

Mavericks KeyChain.app

For OS-specific solutions, I've tried clearing my login KeyChain[4], to no avail.

Issues with pip

There's one last bit of evidence that may or may not be relevant. python3.4 ships with pip intact. However, the pip3 command is useless to me. No matter what I try to install:

(bash)$ pip3 install [new-lib] # pip 1.5.6

I get:

Downloading/unpacking [new-lib]
    Cannot fetch index base URL https://pypi.python.org/simple/
    Could not find any downloads that satisfy the requirement [new-lib]
Cleaning up...
    No distributions at all found for [new-lib]
    Storing debug log for failure in ~/.pip/pip.log

Although this isn't (explicitly) an SSL error, it seems similar[5] and a successful workaround has been to install an older version of pip using easy_install in my virtualenvs[5]. I'm crossing my fingers that the two issues are related.

Recap:

• Seeking possible solutions for SSL certificate failure error (without using verify = False in the requests calls).
• I get the error in python3.4 but not python2.7 even though the cert.pem used in both cases is exactly the same.
• Though I can recreate an SSL error using openssl s_client -connect I can avoid it by specifying -CAFile to the cert.pem used by the requests library.
• My best guess is that this is an issue particular to Mavericks, but I have no idea how to proceed.
• I'm hoping to find a solution that also allows me to use pip3 to install python3.4 packages as expected.

Thanks for your help!

[1]: python2.7 on my machine was installed using Enthought. But installing a system version of python2.7 and the requests library works too.

[2]: See openssl, python requests error: "certificate verify failed" for a similar problem using python 2.7

[3]: It seems Mavericks introduced a change in openssl? http://curl.haxx.se/mail/archive-2013-10/0036.html

[4]: Clearning KeyChain.app from here: https://superuser.com/a/721629/261875

[5]: SSL error with pip3: https://stackoverflow.com/a/22051466/2506078

Answer 1

From the additional information that you have supplied, it appears you have installed the 32-bit-only version of Python 3.4.1 from python.org. This version is intended primarily for use on OS X 10.5 systems; as such, it is linked with the version of OpenSSL supplied by Apple with 10.5. You can avoid this problem by using the 64-bit/32-bit 3.4.1 installer from python.org; this version is recommended for OS X 10.6+ and is linked with the newer version of Apple's OpenSSL. Otherwise, you could manually download distributions from PyPI using curl or a browser and have install pip install them from the downloaded file(s).

Answer 2

Just a guess: the OpenSSL as shipped with Mac OS X (which is still 0.9.8) has special hooks in it so that it falls back to OS X keyring if verification fails against the CAs given to OpenSSL itself. But, if you use your own OpenSSL it does not have this fallback.

This means, that if you use the built-in OpenSSL with python2 it will successfully verify the site if it finds a CA inside the OS X keyring, even if it is not in the cert store provided by requests itself. But if you have compiled python3 against your own OpenSSL it will only use the CAs which are provided by requests itself and not fall back to OS X keyring and thus will fail to verify if the CA is not in requests keyring.

For details about this "feature" of Mac OS X and the problems it introduces see https://hynek.me/articles/apple-openssl-verification-surprises/.

Unfortunately this does not explain why openssl successfully verifies against the default certificates of the requests library, unless there is yet another OpenSSL version involved, i.e. a version used by python3 without the keyring fallback and a recent version on the command line which has the fallback.