Coredump when compiling python with a custom openssl version

Question

When compiling python-3.4.0rc3 with a local openssl-1.0.1f shared install, make prints no error but then I get the following core dump on make install or make t

Answer 1

I couldn't find anyway to modify how _hashlib.so is generated, as there is too much Makefile magic involved (it doesn't appear anywhere, nor does '-lssl' yet both magically end up on the same line together

grep is your friend ;)

$ grep -R _hashlib * | grep ssl
Lib/hashlib.py:        f = getattr(_hashlib, 'openssl_' + name)
Lib/hashlib.py:            _hashlib.openssl_md_meth_names)
Lib/test/test_hashlib.py:            self.assertTrue(hasattr(_hashlib, 'openssl_md5'))
Lib/test/test_hashlib.py:            self.assertTrue(hasattr(_hashlib, 'openssl_sha1'))
Lib/test/test_hashlib.py:                constructor = getattr(_hashlib, 'openssl_'+algorithm, None)
Lib/test/ssltests.py:TESTS = ['test_asyncio', 'test_ftplib', 'test_hashlib', 'test_httplib',
Lib/test/time_hashlib.py:    print(" '_hashlib' 'openssl_hName' 'fast' tests the builtin _hashlib")
Modules/_hashopenssl.c:    "_hashlib.HASH",    /*tp_name*/
Modules/_hashopenssl.c:static struct PyModuleDef _hashlibmodule = {
Modules/_hashopenssl.c:    "_hashlib",
Modules/_hashopenssl.c:PyInit__hashlib(void)
Modules/_hashopenssl.c:    m = PyModule_Create(&_hashlibmodule);
PCbuild/build_ssl.py:# Script for building the _ssl and _hashlib modules for Windows.
PCbuild/build_ssl.py:# for the actual _ssl.pyd and _hashlib.pyd DLLs.
PCbuild/build_ssl.py:# it should configure and build SSL, then build the _ssl and _hashlib
setup.py:                exts.append( Extension('_hashlib', ['_hashopenssl.c'],
setup.py:                print("warning: openssl 0x%08x is too old for _hashlib" %
Tools/ssl/test_multiple_versions.py:    "test_asyncio", "test_ftplib", "test_hashlib", "test_httplib",
Tools/ssl/test_multiple_versions.py:MINIMAL_TESTS = ["test_ssl", "test_hashlib"]

Answer 2

I am able to run python 2.7.11 with non-default SSL after patching fix mentioned here https://gist.github.com/eddy-geek/9604982

However, with this it's not building _socket module which is required by many other modules. for example, easy_install / pip started failing with error Importerr: no module named _socket

In Module/Setup.dist, am i suppose to uncomment or comment the line _socket socketmodule.o ?

I see socketmodule.o and timemodule.o getting generated. but not _socket.so Am i missing something ?

Answer 3

How can I fix this, or at least investigate what is going on ?
...
export LDFLAGS='-L/data2/soft/openssl/lib -L/data2/local/lib/'
export LD_LIBRARY_PATH="/data2/soft/openssl/lib:/data2/local/lib/

I run into these problems a lot too because I avoid the crippled versions of OpenSSL shipped by Debian, Ubuntu, Fedora, et al. For example, Ubuntu ships an OpenSSL that disables TLSv1.1 and TLS v1.2 (cf., Ubuntu 12.04 LTS: OpenSSL downlevel version and does not support TLS 1.2).

You're probably loading the wrong version of the OpenSSL library. If you can get the misbehaving program under the debugger, issue info shared to see which libcrypto and libssl you are actually loading.

ldd might help too. Run it on the Pyhton executable: ldd /data2/soft/python3/python. I can only say it "may" help because the OpenSSL's are binary compatible, so you might only see a dependency on, for example, libcrypto.so.1.0.0 (use otool -L on Mac OS X). Below I used an rpath to force linking against the libraries in /usr/local/ssl/lib/.

$ ldd my-test.exe 
    linux-vdso.so.1 =>  (0x00007fffd61ff000)
    libssl.so.1.0.0 => /usr/local/ssl/lib/libssl.so.1.0.0 (0x00007f151528e000)
    libcrypto.so.1.0.0 => /usr/local/ssl/lib/libcrypto.so.1.0.0 (0x00007f1514e74000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1514c42000)
    ...

---

As a fix, you might try adding an rpath:

LDFLAGS='-L/data2/soft/openssl/lib -L/data2/local/lib/ -Wl,-rpath,/data2/soft/openssl/lib'

A better fix would be to link to the static version of the OpenSSL library to avoid these problems all together. I believe you can do that with: -Bstatic -lssl -lcrypto -Bdynamic -ldl.

Personally, I don't even use -Bstatic because of different one-off problems. I open the Makefiles, remove all instances of -lssl -lcrypto, and add a full path to the archive to remove all ambiguity. For example, /data2/soft/openssl/lib/libssl.a and /data2/soft/openssl/lib/libcrypto.a.

Note well: an rpath is not honored on Mac OS X. More extreme measures are needed for Mac OS X because the linker does not honor -Bstatic either. You'll have to use the full path trick.

---

openssl was built with ./config shared --openssldir=/data2/soft/openssl

Another thing... On Fedora, its not enough to specify shared. You need to add the following, too:

export CFLAGS="-fPIC"

Otherwise, the shared object is not built. If the 1.0.1f shared object is not built, you're probably getting a downlevel version supplied by the distro. You can check what was built before install with:

./config shared --openssldir=/data2/soft/openssl
make all

# Verify artifacts
find . -iname -libcrypto.*
find . -iname -libssl.*

# Proceed if OK
sudo make install

---

Finally, make sure that all of Python's dependencies are also using your version of OpenSSL and not the system's version of OpenSSL.

I recently suffered that problem when my program used my OpenSSL; but my program also used libevent and libevent used the system's version of OpenSSL. I fixed it by rebuilding libevent and forcing it to statically link to my version of OpenSSL.

Answer 4

We had a similar problem. We are using apache httpd + mod_wsgi + python + django and our c++ module for the apache httpd which also uses openssl. Now everything is loaded within one httpd process, correct version of openssl shared lib was loaded (1.0.0l) with our c++ module. But as soon as we access the web, python loads the hashlib and exactly the same problem appears - segfault in openssl called from python.

Normally python compiles with whatever openssl is available at the default location and there is no way how to specify it without fiddling with setup.py or makefiles. Python developers should add configure setting --with_ssl=path.

We installed new openssl libs and rebuild python and other binaries but with no success. We mapped default libssl.so and libcrypto.so to the new openssl binaries with no success. Finally after reading this thread I realized that probably wrong headers are being used while compiling python. And that was the problem. Follow the step to workaround the problem:

• there must not be incorrect version of openssl headers in /usr/include and default locations /usr/local/ssl, /usr/contrib/ssl (if you installed openssl-devel then uninstall it, or simply erase/rename the directory)

yum remove openssl-devel

• make symbolic link to your openssl installation from /usr/local/ssl

ln -s /opt/openssl-1.0.1l /usr/local/ssl

• ensure that the new openssl libs are accessible from /usr/lib (make symbolic links if it is not installed here)

ln -s /opt/openssl-1.0.1l/lib/libcrypto.so.1.0.0 /usr/lib/libcrypto.so.1.0.0 ...

• now configure and clean build the python

Answer 5

How should I go about changing it ? Will adding a _ssl.c: gcc ... line anywhere override the default behaviour ? My Makefile skills are rusted and this beast is 1600+ lines long !!

Here are the files you want to look at:

$ cd Python-3.4.0rc3

$ grep -R -- '-lcrypto' *
Modules/Setup.dist:#    -L$(SSL)/lib -lssl -lcrypto

$ grep -R -- '-lssl' *
Modules/Setup.dist:#    -L$(SSL)/lib -lssl -lcrypto

Here are the lines of interest in Setup.dist:

# Socket module helper for SSL support; you must comment out the other
# socket line above, and possibly edit the SSL variable:
#SSL=/usr/local/ssl
#_ssl _ssl.c \
#   -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
#   -L$(SSL)/lib -lssl -lcrypto

Uncomment the lines and change Setup.dist to the following. I keep my OpenSSL in /usr/local/ssl, so that's how mine is setup below (you should use /data2/soft/openssl/lib/...):

SSL=/usr/local/ssl
_ssl _ssl.c \
    -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
    /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl

Use the full path to the archive, and don't use -l. Be sure to add -ldl because OpenSSL needs it in this configuration.

Once you change Setup.dist, re-run ./configure to effect the changes.

---

After changing the above line to, here's what I look like after a configure:

$ grep -R "libssl.a" *
Makefile:LOCALMODLIBS=  /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl 
Makefile:Modules/_ssl$(SO):  Modules/_ssl.o; $(BLDSHARED)  Modules/_ssl.o  /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl  -o Modules/_ssl$(SO)
Modules/Setup: /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl 
Modules/Setup.dist: /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a -ldl 

---

You can test the static linking by using ldd or otool -L. You will not see an OpenSSL dependency. After make'ing, here's what I got:

$ find . -iname python
./python
$ ldd ./python
    linux-vdso.so.1 =>  (0x00007fff67709000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3aed8e1000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3aed6dd000)
    libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f3aed4d9000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3aed257000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3aececc000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f3aedb14000)

No libssl or libcrypto dependencies to go wrong :) And make test ran fine (actually, one failed test due to a Python bug: Issue 20896):

======================================================================
ERROR: test_get_server_certificate (test.test_ssl.NetworkedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jwalton/Python-3.4.0rc3/Lib/test/test_ssl.py", line 1373, in test_get_server_certificate
    _test_get_server_certificate('svn.python.org', 443, SVN_PYTHON_ORG_ROOT_CERT)
  File "/home/jwalton/Python-3.4.0rc3/Lib/test/test_ssl.py", line 1354, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port))
  File "/home/jwalton/Python-3.4.0rc3/Lib/ssl.py", line 902, in get_server_certificate
    with context.wrap_socket(sock) as sslsock:
  File "/home/jwalton/Python-3.4.0rc3/Lib/ssl.py", line 344, in wrap_socket
    _context=self)
  File "/home/jwalton/Python-3.4.0rc3/Lib/ssl.py", line 540, in __init__
    self.do_handshake()
  File "/home/jwalton/Python-3.4.0rc3/Lib/ssl.py", line 767, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:598)

----------------------------------------------------------------------
Ran 96 tests in 8.610s

FAILED (errors=1, skipped=3)
test test_ssl failed
make: *** [test] Error 1

Answer 6

Another partial answer...

But I can get it linking dynamically to my own openssl through good old -I/-L:

...

Now the only problem is, gdb info shared still tells me another one is used at core-time ... but how ?

That's your good old friends -l and -L. Don't use them because they do this sort of thing all the time (take it from a guy who has suffered it in the past). Instead, specify the full path to libssl and libcrypto. E.g., use /data2/soft/openssl/lib/libssl.a.