Tests broken with out-of-source builds

[mgorny]

The test suite basically converts an out-of-source build back into an ugly in-source build. Therefore, if multiple versions of the package are built from the same sources, the tests will work with only one of them properly.

I'm not sure if I understand the tests correctly but I believe that you rather should adjust sys.path within the tests to respect build-dir rather than linking the built files back into source dir.

[schmir]

greenlet/greenlet.so is being replaced with a version for the currently running interpreter. this should only be a problem if you're running tests for different python version in parallel.

Can you tell us how to reproduce the problem?

[mgorny]

Yes, we are doing exactly that. It saves us a lot of time.

[schmir]

You can start run-tests.py with the -n option, which will skip building greenlet and not use greenlet.so in the current directory.

Does that help?

[mgorny]

Well, that fixed one of the problems. Now I think I'm seeing a race condition with the test extension.

Traceback (most recent call last):
  File "run-tests.py", line 57, in <module>
    suite = test_collector()
  File "/tmp/portage/dev-python/greenlet-0.4.0-r1/work/greenlet-0.4.0/tests/__init__.py", line 44, in test_collector
    return unittest.TestLoader().loadTestsFromNames(test_module_list)
  File "/usr/lib64/python2.5/unittest.py", line 565, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib64/python2.5/unittest.py", line 533, in loadTestsFromName
    module = __import__('.'.join(parts_copy))
  File "/tmp/portage/dev-python/greenlet-0.4.0-r1/work/greenlet-0.4.0/tests/test_extension_interface.py", line 5, in <module>
    import _test_extension
AttributeError: PyCapsule_Import "greenlet._C_API" is not valid
 * ERROR: dev-python/greenlet-0.4.0-r1 failed (test phase):
 *   Tests fail with python2.5

[snaury]

See tests/init.py function build_test_extensions(). The test suite builds additional extensions (that are only used in testing) in a separate directory by specifying its own build_base. Since out of source build is done in esetup.py with a --build-base argument it's obviously not used in tests.

The error you see is because test extensions haven't been built correctly. Capsules are only supposed to be used on Python >= 2.7, yet extension is somehow trying to use capsules on what is supposed to be Python 2.5. Clearly different Pythons got mixed somehow.

[snaury]

In short, could you maybe fix ebuild to set DISTUTILS_IN_SOURCE_BUILD=1? It's obviously not supported (yet?) when testing is needed.

[mgorny]

Yes, I would probably need a way to specify proper --build-lib for that then.

And yes, I have worked this around already. But I'm mostly trying to address the actual issue to get it fixed properly someday.

[snaury]

Ok, I see. To be honest I don't like the way test extensions are built right now, but I'll have to see if it's possible to build them properly and not have them installed as a side-effect.

[mgorny]

Well, I see two possibilities here:

1. Adding additional commands to setup.py to build tests (and requiring them in test command), therefore allowing user to override the install path,
2. or just installing everything test-related in tests sub-package. A few Python packages just do that, allowing users to re-run tests on installed packages.

[schmir]

if we would compile the test extensions in some python version specific path, would that solve your problem?

[schmir]

btw option 2 is not an option IMHO.

[mgorny]

If by 'Python version' you mean sys.version, that would be an ugly hack and it will fail if more than one implementation was identified with the same Python language version. This is a case in PyPy (although it doesn't seem to be supported by greenlet right now).

One tangential possibility would be to create a temporary directory (with tempfile.mkdtemp()), compile the extension there and remove it after the tests finish. The same directory could be reused if tests every need to use temporary files.

[schmir]

pypy already comes with a greenlet module.

would you also be happy with a setup.py command line switch, which adds the test extensions to ext_modules?
they would then also be installed when calling setup.py install?

[mgorny]

In that case, I'd prefer that the switch installed both the extension and the tests. In case we wouldn't strip it, it seems that the extension alone is not really meaningful.

Also, please make sure that the extension is installed in proper namespace.

[snaury]

@mgorny I'm sorry, but given this bug report that I found:

http://bugs.python.org/issue1887

I don't think it would be appropriate to support out-of-source builds in greenlet.

[mgorny]

I'm afraid I don't understand the relevance between the cited bug report and this request. The referenced bug only states that autotools-style out-of-source builds are not supported, that is builds where setup.py is invoked from another directory. This bug is about distutils-style out-of-source build where setup.py is executed from the current directory and an option is used to change build directory.

[snaury]

But I'm not sure how that is supposed to work. For example description for setuptools test command (https://pythonhosted.org/setuptools/setuptools.html#test-build-package-and-run-a-unittest-suite) reads:

The test command runs a project’s unit tests without actually deploying it, by temporarily putting the project’s source on sys.path, after first running build_ext -i and egg_info to ensure that any C extensions and project metadata are up-to-date.

The build_ext -i already implies that for setuptools + unittest based projects built extensions would be copied to the source directory. How do other projects support this?

[snaury]

Ok, after looking closely at what's currently going on I may have a solution:

https://github.com/snaury/greenlet/commit/db2029c92c8f8c13485f84f395194665fe945c66

It no longer copies test extensions to the source directory and run-tests.py allows you to specify build directory for tests, e.g.:

python run-tests.py -n -b /tmp/greenlet-tests

Would that help?

[mgorny]

I've just tested it and it seems to work properly. Thanks.

[snaury]

@mgorny cool, I pushed a different version of that patch to master. It also no longer symlinks greenlet.so, unless building inplace with -i.