问题
So, being a Linux guy I stumbled into something pretty puzzling on Windows that I just can't explain.
I have a project structure analougus to this example:
D:\PROJECT
|
| tolkien.py
| __init__.py
|
\---MiddleEarth
| gondor.py
| isengrad.c
| __init__.py
|
\---lib
isengrad.so
Problem: I compile isengrad.c into the shared libary isengrad.so, then load it in gondor.py. My aim is to import gondor.py into tolkien.py.
While gondor.py runs flawlessly when it is run directly, when I import it, the code exits at the point when I load the shared library via ctypes.CDLL, without any error messages.
Reproduction: The content of the files (added some "status messages" to follow where the problem happens):
isengrad.c:
int isengrad(int hobbit){
return hobbit/2;
}
This is then compiled to isengrad.so with
D:\project>chdir MiddleEarth
D:\project\MiddleEarth>gcc -fPIC -shared -o lib/isengrad.so isengrad.c
The shared library is then accessed in gondor.py:
print("started gondor")
import os, ctypes
path_to_isengrad = "D:/project/MiddleEarth/lib/isengrad.so"
print("gondor loads isengrad")
gondor = ctypes.CDLL(path_to_isengrad) # <--- crashes here when imported, not when ran directly
print("gondor loaded isengrad")
gondor.isengrad.argtypes = (ctypes.c_int,)
def faramir(hobbit):
catched_hobbits = gondor.isengrad(hobbit)
return catched_hobbits
if __name__ == '__main__':
print(faramir(5))
print("gondor ran")
print("gondor finished")
which is then imported in tolkien.py:
print("started tolkien")
from MiddleEarth import gondor
print("tolkien imported gondor")
got = gondor.faramir(4)
print(got)
print("tolkien worked")
Now check what happens when I use gondor.py directly VS when I import it in tolkien.py:
D:\project>python MiddleEarth/gondor.py
started gondor
gondor loads isengrad
gondor loaded isengrad
2
gondor ran
gondor finished
D:\project>python tolkien.py
started tolkien
started gondor
gondor loads isengrad
D:\project>
Directly running it causes no problem at all. But importing it causes the whole thing to crash without any word and traceback when loading the shared library. How is this even happening? I even hard-coded the path to the shared library, so different working directory shouldn't be a problem... I didn't have any problem with the very same project on Kubuntu, so this is probably some Windows-related stuff.
Environment:
- Python:
Python 3.7.3 (default, Mar 27 2019, 17:13:21) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32 - OS:
Windows 10 10.0.17134 Build 17134(installed on C:) - GCC: Installed via Cygwin, version 7.4.0
- Please ask if any other details needed.
回答1:
From the moment I saw this question, I wanted to say it's Undefined Behavior (UB). Python comes with its C runtime (UCRTLib), while the Cygwin .dll comes with its own. Mixing compilers and C runtimes in a process, is generally a recipe for disaster.
I found an official statement [Cygwin]: 6.15. Can I link with both MSVCRT*.DLL and cygwin1.dll? (emphasis is mine):
No, you must use one or the other, they are mutually exclusive.
Check [SO]: How to circumvent Windows Universal CRT headers dependency on vcruntime.h (@CristiFati's answer) for more details on MSVCRT*.DLL
Now, the beauty of UB is that it describes a seemingly random behavior.
I've prepared a comprehensive example (slightly modifying your code).
isengrad.c:
#if defined(_WIN32)
# define ISENGRAD_EXPORT_API __declspec(dllexport)
#else
# define ISENGRAD_EXPORT_API
#endif
ISENGRAD_EXPORT_API int isengrad(int hobbit) {
return hobbit / 2;
}
script0.py:
#!/usr/bin/env python3
import sys
import ctypes
dll_name = "./lib/isengrad_{0:s}_{1:03d}.dll".format(sys.argv[1][:3] if sys.argv else sys.platform[:3].lower(), ctypes.sizeof(ctypes.c_void_p) * 8)
print("Attempting to load: {0:s}".format(dll_name))
isengrad_dll = ctypes.CDLL(dll_name)
print("DLL Loaded")
def main():
isengrad_func = isengrad_dll.isengrad
isengrad_func.argtypes = [ctypes.c_int]
isengrad_func.restype = ctypes.c_int
res = isengrad_func(46)
print("{0:s} returned {1:}".format(isengrad_func.__name__, res))
if __name__ == "__main__":
print("Python {0:s} {1:d}bit on {2:s}\n".format(" ".join(item.strip() for item in sys.version.split("\n")), 64 if sys.maxsize > 0x100000000 else 32, sys.platform))
main()
print("\nDone.")
script1.py:
#!/usr/bin/env python3
import sys
import script0
def main():
pass
if __name__ == "__main__":
print("Python {0:s} {1:d}bit on {2:s}\n".format(" ".join(item.strip() for item in sys.version.split("\n")), 64 if sys.maxsize > 0x100000000 else 32, sys.platform))
main()
print("\nDone.")
Outputs:
- I'll be using 3 windows:
- cmd - Win (32bit and 64bit)
- Cygwin's Mintty:
- 64bit
- 32bit
- Note that even if I'll paste each's contents in one chunk (to avoid scattering them), I switched between them when running the commands
Cygwin 32bit:
[cfati@cfati-5510-0:/cygdrive/e/Work/Dev/StackOverflow/q056855348]> ~/sopr.sh *** Set shorter prompt to better fit when pasted in StackOverflow (or other) pages *** [032bit prompt]> gcc -shared -fPIC -o lib/isengrad_cyg_032.dll isengrad.c [032bit prompt]> ls lib/*.dll lib/isengrad_cyg_032.dll lib/isengrad_cyg_064.dll lib/isengrad_win_032.dll lib/isengrad_win_064.dll [032bit prompt]> [032bit prompt]> python3 script0.py cyg Attempting to load: ./lib/isengrad_cyg_032.dll DLL Loaded Python 3.6.4 (default, Jan 7 2018, 17:45:56) [GCC 6.4.0] 32bit on cygwin isengrad returned 23 Done. [032bit prompt]> [032bit prompt]> python3 script1.py cyg Attempting to load: ./lib/isengrad_cyg_032.dll DLL Loaded Python 3.6.4 (default, Jan 7 2018, 17:45:56) [GCC 6.4.0] 32bit on cygwin Done. [032bit prompt]> [032bit prompt]> python3 script0.py win Attempting to load: ./lib/isengrad_win_032.dll DLL Loaded Python 3.6.4 (default, Jan 7 2018, 17:45:56) [GCC 6.4.0] 32bit on cygwin isengrad returned 23 Done. [032bit prompt]> [032bit prompt]> python3 script1.py win Attempting to load: ./lib/isengrad_win_032.dll DLL Loaded Python 3.6.4 (default, Jan 7 2018, 17:45:56) [GCC 6.4.0] 32bit on cygwin Done.Cygwin 64bit:
[cfati@cfati-5510-0:/cygdrive/e/Work/Dev/StackOverflow/q056855348]> ~/sopr.sh *** Set shorter prompt to better fit when pasted in StackOverflow (or other) pages *** [064bit prompt]> gcc -shared -fPIC -o lib/isengrad_cyg_064.dll isengrad.c [064bit prompt]> ls lib/*.dll lib/isengrad_cyg_032.dll lib/isengrad_cyg_064.dll lib/isengrad_win_032.dll lib/isengrad_win_064.dll [064bit prompt]> [064bit prompt]> python3 script0.py cyg Attempting to load: ./lib/isengrad_cyg_064.dll DLL Loaded Python 3.6.8 (default, Feb 14 2019, 22:09:48) [GCC 7.4.0] 64bit on cygwin isengrad returned 23 Done. [064bit prompt]> [064bit prompt]> python3 script1.py cyg Attempting to load: ./lib/isengrad_cyg_064.dll DLL Loaded Python 3.6.8 (default, Feb 14 2019, 22:09:48) [GCC 7.4.0] 64bit on cygwin Done. [064bit prompt]> [064bit prompt]> python3 script0.py win Attempting to load: ./lib/isengrad_win_064.dll DLL Loaded Python 3.6.8 (default, Feb 14 2019, 22:09:48) [GCC 7.4.0] 64bit on cygwin isengrad returned 23 Done. [064bit prompt]> [064bit prompt]> python3 script1.py win Attempting to load: ./lib/isengrad_win_064.dll DLL Loaded Python 3.6.8 (default, Feb 14 2019, 22:09:48) [GCC 7.4.0] 64bit on cygwin Done.cmd:
[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q056855348]> sopr.bat *** Set shorter prompt to better fit when pasted in StackOverflow (or other) pages *** [prompt]> dir /b lib [prompt]> "c:\Install\x86\Microsoft\Visual Studio Community\2017\VC\Auxiliary\Build\vcvarsall.bat" x64 ********************************************************************** ** Visual Studio 2017 Developer Command Prompt v15.9.14 ** Copyright (c) 2017 Microsoft Corporation ********************************************************************** [vcvarsall.bat] Environment initialized for: 'x64' [prompt]> cl /nologo /DDLL isengrad.c /link /NOLOGO /DLL /OUT:lib\isengrad_win_064.dll isengrad.c Creating library lib\isengrad_win_064.lib and object lib\isengrad_win_064.exp [prompt]> [prompt]> "c:\Install\x86\Microsoft\Visual Studio Community\2017\VC\Auxiliary\Build\vcvarsall.bat" x86 ********************************************************************** ** Visual Studio 2017 Developer Command Prompt v15.9.14 ** Copyright (c) 2017 Microsoft Corporation ********************************************************************** [vcvarsall.bat] Environment initialized for: 'x86' [prompt]> cl /nologo /DDLL isengrad.c /link /NOLOGO /DLL /OUT:lib\isengrad_win_032.dll isengrad.c Creating library lib\isengrad_win_032.lib and object lib\isengrad_win_032.exp [prompt]> dir /b lib\*.dll isengrad_cyg_032.dll isengrad_cyg_064.dll isengrad_win_032.dll isengrad_win_064.dll [prompt]> set _PATH=%PATH% [prompt]> :: Python 32bit [prompt]> set PATH=%_PATH%;e:\Install\x86\Cygwin\Cygwin\Version\bin [prompt]> "e:\Work\Dev\VEnvs\py_032_03.07.03_test0\Scripts\python.exe" script0.py win Attempting to load: ./lib/isengrad_win_032.dll DLL Loaded Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] 32bit on win32 isengrad returned 23 Done. [prompt]> "e:\Work\Dev\VEnvs\py_032_03.07.03_test0\Scripts\python.exe" script1.py win Attempting to load: ./lib/isengrad_win_032.dll DLL Loaded Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] 32bit on win32 Done. [prompt]> "e:\Work\Dev\VEnvs\py_032_03.07.03_test0\Scripts\python.exe" script0.py cyg Attempting to load: ./lib/isengrad_cyg_032.dll DLL Loaded Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] 32bit on win32 isengrad returned 23 Done. [prompt]> "e:\Work\Dev\VEnvs\py_032_03.07.03_test0\Scripts\python.exe" script1.py cyg Attempting to load: ./lib/isengrad_cyg_032.dll DLL Loaded Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] 32bit on win32 Done. [prompt]> :: Python 64bit [prompt]> set PATH=%_PATH%;c:\Install\x64\Cygwin\Cygwin\AllVers\bin [prompt]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" script0.py win Attempting to load: ./lib/isengrad_win_064.dll DLL Loaded Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] 64bit on win32 isengrad returned 23 Done. [prompt]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" script1.py win Attempting to load: ./lib/isengrad_win_064.dll DLL Loaded Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] 64bit on win32 Done. [prompt]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" script0.py cyg Attempting to load: ./lib/isengrad_cyg_064.dll DLL Loaded Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] 64bit on win32 isengrad returned 23 Done. [prompt]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" script1.py cyg Attempting to load: ./lib/isengrad_cyg_064.dll [prompt]> [prompt]> echo %errorlevel% -1073741819
As seen, cross compiler .exe - .dll worked in 7 (out of 8) cases (crashed on 64bit Win Python with script1.py), while the same compiler worked in all 8 of them.
So, I'd advise that when playing with such environments, try to keep the compilers used to build various parts consistent (or compatible at least).
Update #0
I just thought of a reason why things could go wrong on 64bit: sizeof(long) generally differs (sizes below are in bytes):
- 4 on Win
- 8 on Cygwin (on Nix, in general)
Same thing for sizeof(long double) (which is 2 * sizeof(long)).
So, if the Cygwin .dll exposes some long value greater than 2 ** 64 (1 << 64), it will be truncated in the Win process, and in this case a crash might occur. Theoretically, this situation should affect the reverse scenario as well, but it doesn't.
There are other factors that might lead to this behavior, like default memory alignment, and so on.
来源：https://stackoverflow.com/questions/56855348/paradoxon-silent-crash-on-pythons-ctypes-cdll-when-importing-but-not-when-run