Python C Module - Malloc fails in specific version of Python

问题

I'm writing a Python module to perform IO on a O_DIRECT context. One of the limitations of O_DIRECT is you must read into a buffer aligned on a 4096 byte boundary for 2.4 and 2.5 kernels, and 2.6 and up will accept any multiple of 512.

The obvious memory allocation candidate for this is posix_memalign(void **memptr, size_t alignment, size_t size)

In my code, I allocate an area like so:

char *buffer = NULL;

int mem_ret = posix_memalign((void**)&buffer, alignment, size);

if (!buffer) {
    PyErr_NoMemory();
    return NULL;
}

/* I do some stuff here */

free(buffer);

When I compile and import the module with python3.2, this (and the rest of the unshown module) work fine.

When I attempt the same with python2.7 (I'd like to preserve compatibility) it throws the PyErr_NoMemory exception, and mem_ret == ENOMEM, indicating it was unable to allocate.

Why would the version of Python I compile against affect how posix_memalign operates?

OS: Ubuntu 12.04 LTS

Compiler: Clang + GCC Show same behaviour

UPDATE

I now have a working piece of code, thanks to user694733
However the fact that it works has me even more confused:

#if PY_MAJOR_VERSION >= 3
char *buffer = NULL;

int mem_ret = posix_memalign((void**)&buffer, alignment, count);
#else
void *mem = NULL;

int mem_ret = posix_memalign(&mem, alignment, count);

char *buffer = (char*)mem;
#endif

Can anyone explain why the incorrect first block works under Python3, but not 2.7, and more importantly why the correct second block does not work under Python3?

UPDATE 2

The plot thickens, having settled on the correct form of the code below, I tested on 4 different version of Python.

void *mem = NULL;

int mem_ret = posix_memalign(&mem, alignment, count);

char *buffer = (char*)mem;

if (!buffer) {
    PyErr_NoMemory();
    return NULL;
}

/* Do stuff with buffer */

free(buffer);

Under Python 2.7: This code operates as expected.
Under Python 3.1: This code operates as expected.
Under Python 3.2: This code generates mem_ret == ENOMEM and returns NULL for buffer
Under Python 3.3: This code operates as expected.

The Python versions not included in the Ubuntu repositories were installed from the PPA at https://launchpad.net/~fkrull/+archive/deadsnakes

If the version tagged Python binaries are to be believed, the versions I have installed are:

python2.7 
python3.1
python3.2mu (--with-pymalloc --with-wide-unicode)
python3.3m (--with-pymalloc)

Could the use of the wide-unicode flag in the default Python3 distribution be causing this error? If so, how is this happening?

For clarity, the ENOMEM failure to allocate will occur with any variant of malloc(), even something as simple as malloc(512).

回答1:

For a quick work-around, stick to mmap instead of malloc+memalign

回答2:

posix_memalign may not be the same body of code in one compilation environment as another. You could easily imagine that Python 3 would use different feature test macros to Python 2. That could mean it ends up running different code.

You might have a look at the symbols that are used... often times the output of ldd or nm will have mangled names that indicate what version is actually being used.

Additionally, what does an strace show of the allocation system call? I find that's a good way of seeing if the arguments passed in are incorrect, which can be a reason for getting ENOMEM.

来源：https://stackoverflow.com/questions/21458818/python-c-module-malloc-fails-in-specific-version-of-python

标签

python

memory-management

cpython