How to make Python use CA certificates from Mac OS TrustStore?

Question

I need to use curtom root certificates on the company intranet and loading them in the Mac OS TrustStore (KeyChain) does solve the problem for all browsers and GUI apps.

It seems that it works even with the version of curl that ships with Mac OS X but it doesn't work with python, even the version that ships with Mac OS 10.12 Sierra (Python 2.7.10)

Still, it seems that I would be hit by:

urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>

How can I solve this?

Because I encounter this issue in lots and lots of Python tools I would really appreciate if I find a way to avoid it without having to patch them.

Providing the custom CA certificate myself is not an option because I cannot patch tens of Python tools that I use.

Most of the tools are using the requests library but there are a few that are using the native ssl support in Python directly.

Answer 1

This is also a problem in Python 3.6 with MacOS Sierrra. I know your use case is different. But I stumbled upon this thread while investigating this problem. So if anyone is also having this article is worth checking out:

http://www.cdotson.com/2017/01/sslerror-with-python-3-6-x-on-macos-sierra/

In a nutshell: Python 3.6 does not rely on MacOS' openSSL anymore. It comes with its own openSSL bundled and doesn't have access on MacOS' root certificates.

You have two options:

Run an install command shipped with Python 3.6

cd /Applications/Python\ 3.6/
./Install\ Certificates.command

or

Install the certifi package with

pip install certifi

I chose the first option and it worked.

Answer 2

If you put the additional certificates in a PEM bundle file you can use these two environment variables to overwrite the default cert stores used by Python openssl and requests.

SSL_CERT_FILE=/System/Library/OpenSSL/cert.pem
REQUESTS_CA_BUNDLE=/System/Library/OpenSSL/cert.pem

Please note that this file does not exist, you need to build it yourself.

Answer 3

As an update and datapoint, I ran into this issue running Python 3.7.0 on macOS 10.13.4:

$ ipython
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.0.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import bokeh.sampledata

In [2]: bokeh.sampledata.download()
Using data directory: /Users/me/.bokeh/data

...
SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)

Instructions for solving the problem are in /Applications/Python\ 3.7/ReadMe.rtf

Following the suggestion there and running /Applications/Python\ 3.7/Install\ Certificates.command solved the problem:

From the terminal:

$ /Applications/Python\ 3.7/Install\ Certificates.command

Re-starting IPython...

$ ipython
>>> import bokeh.sampledata

>>> bokeh.sampledata.download()
Using data directory: /Users/me/.bokeh/data
Downloading: CGM.csv (1589982 bytes)
   1589982 [100.00%]
...

Answer 4

Mac brew install python env.

$ python3
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 16:52:21) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import certifi
>>> certifi.where()
'/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/certifi/cacert.pem'
>>> 

Or from the command line:

$ python -m certifi

then need link cacert.pem as cert.pem

$ ln -s /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/certifi/cacert.pem cert.pem
$ pwd
/Library/Frameworks/Python.framework/Versions/3.7/etc/openssl

rehash

then work fine.

Answer 5

For me /Applications/Python\ 3.6/./Install\ Certificates command fails on pip certifi install. I am on mac High Sierra and use python3 so pip somewhat fails and I have to use pip3 instead.

So here what I did:

1. Manually ran pip3 install --update certify in a shell
2. Remove the install certifi line from the command script
3. Reran the script and everything was fine.

Note that you will end up with a cert.pem symbolic link in: /Library/Frameworks/Python.framework/Versions/3.6/etc/openssl/