Memory leak loading python modules compiled by MinGW

[buergi]

Just tried to build an own python module with the MinGW gcc shipped with anaconda.
When loading the build module python allocates >4GB memory instantly, freezing the whole computer and freeing it not before python is closed.
I use a fresh installation of Anaconda3 2.1.0 for Win7 x64 with python 3.4 and the following useless python module.
foo.c

#include <Python.h>
static PyMethodDef funcs[] = {
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef foomodule = {
    PyModuleDef_HEAD_INIT,"foo","just foo",-1,   funcs };

PyMODINIT_FUNC PyInit_foo(void) {
    return PyModule_Create(&foomodule);
}

setup.py

from distutils.core import setup, Extension
setup (name = 'foo', version = '1.0',description = 'foo',
       author = 'me', ext_modules = [Extension('foo', sources = ['foo.c'])])

Calling python setup.py build and importing the module yields the undesired result.

I guess it has something to do with libpython, is there anything I can do to produce working python modules?

[buergi]

I reproduced the problem on Windows 8.1 and I did some further tests. It seems importlib._bootstrap allocates exactly 4GB for some reason:

Python 3.4.1 |Anaconda 2.1.0 (64-bit)| (default, Sep 24 2014, 18:32:42) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import tracemalloc
>>> tracemalloc.start(25)
>>> import foo
>>> s = tracemalloc.take_snapshot()
>>> t = s.statistics('lineno')
>>> print(t[0])
<frozen importlib._bootstrap>:321: size=4096 MiB, count=5, average=819 MiB
>>> t = s.statistics('traceback')
>>> for i in t[0].traceback.format(): print(i)
...
  File "<frozen importlib._bootstrap>", line 321
  File "<frozen importlib._bootstrap>", line 1715
  File "<frozen importlib._bootstrap>", line 539
  File "<frozen importlib._bootstrap>", line 1161
  File "<frozen importlib._bootstrap>", line 1191
  File "<frozen importlib._bootstrap>", line 2226
  File "<frozen importlib._bootstrap>", line 2237
  File "<stdin>", line 1

[buergi]

The problem is buried deep in the python internal import routines, as it also occurs when using _imp:

import _imp
_imp.load_dynamic("foo","foo.pyd")

So I guess it has something to do with mingw's binary format which python does not like.
I did not manage to get the MS x64 compiler working so, but I guess it would work with it.
Furthermore I can confirm that it is working in the 32-bit Version of anaconda.

[buergi]

So finally I got it working, compiling with Windows SDK 7.1 for x86 and x64 both imports fine, so it really is a pure MinGW x64 problem.

[ccordoba12]

@buergi it's known that MinGW x64 is still unstable. It should be recommended to use Visual Studio in Anaconda documentation.

[ccordoba12]

@ilanschnell what do you think about suggesting in our docs that Visual Studio should be used to compile extensions on Windows?

[ilanschnell]

Sure, but this is general Python knowledge, not specific to Anaconda.

[ccordoba12]

@ilanschnell yes, but since Anaconda comes with MinGW, people assumes is better to use it instead of Visual Studio.

[ccordoba12]

For example, see issue #175

[carlkl]

@buergi 's test module works out of the box with the Winpython distribution https://winpython.github.io. This distro is equiped with the mingwstatic toolchain https://bitbucket.org/carlkl/mingw-w64-for-python/downloads that is in development particularly for compilation of python extensions.
https://bitbucket.org/carlkl/mingw-w64-for-python/downloads/mingwpy-2015-01-readme.html gives an explanation on how to use this toolchain.

[ccordoba12]

Pinging @stonebig to see if he can give us further context.

[carlkl]

the mingwstatic toolchains mentioned have some enhancements for better compatibility to the MSVC compilers used by the Windows Python distributions. It should be noted, that it is possible to compile numpy and scipy for 32 and 64 bit with this toolchains. Several issues with some problematic scipy testcases are not solved yet.
However, compiling cython extensions should work out of the box though. Think of this toolchains as a convinient gcc based build system for python extensions.

[ccordoba12]

@carlkl thanks for the explanation. But it'd be better to have a full fledged compiler as part of Anaconda. What about tdm-gcc? (mentioned in issue #175 )

[carlkl]

the mingwstatic toolchain IS a fully fledged compiler toolchain with its own set of patches. tdm-mingw i.e. doesn't solve the CRT mismatch issues your are faced with development of python extensions on Windows.

[stonebig]

Hi all,

Carl said it all.

I do think the carlkl mingwstatic toolchain is the pertinent cython companion for windows, because it solves all problems:
. it's for whatever version of python 2.7/3.3/3.4, 32bit to 64bit,
. it's more stable than anything else because dll are static,
. openMP is included,
. you can easily include it in your distribution,
....

I don't think I received any complaint since I ship carlkl mingwstatic in Winpython, since 4 months

Hope it helps

[ccordoba12]

@stonebig, thanks a lot for the clarification!!

@carlkl, sorry for the confusion. I thought mingwstatic was not a full complier :-)