2.0.0: pytest is failing in 3 units

[kloczek]

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix> using 'installer` module
• run pytest with $PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>
• build is performed in env which is cut off from access to the public network (pytest is executed with -m "not network")

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-crochet-2.0.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-crochet-2.0.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network'
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.8.17, pytest-7.4.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/crochet-2.0.0
collected 114 items

crochet/tests/test_api.py F...............s...........................................F...                                                                                            [ 56%]
crochet/tests/test_logging.py ..F.                                                                                                                                                    [ 59%]
crochet/tests/test_mypy.py ssssssssssssss                                                                                                                                             [ 71%]
crochet/tests/test_process.py .                                                                                                                                                       [ 72%]
crochet/tests/test_resultstore.py .....                                                                                                                                               [ 77%]
crochet/tests/test_setup.py ............s...s                                                                                                                                         [ 92%]
crochet/tests/test_shutdown.py .....                                                                                                                                                  [ 96%]
crochet/tests/test_util.py ....                                                                                                                                                       [100%]

========================================================================================= FAILURES ==========================================================================================
__________________________________________________________________________ ResultRegistryTests.test_runs_with_lock __________________________________________________________________________
'NoneType' object is not iterable

During handling of the above exception, another exception occurred:
NOTE: Incompatible Exception Representation, displaying natively:

twisted.trial.util.DirtyReactorAggregateError: Reactor was unclean.
DelayedCalls: (set twisted.internet.base.DelayedCall.debug = True to debug)
<DelayedCall 0x7fa5f160b040 [0.09723114967346191s] called=0 cancelled=0 LoopingCall<0.1>(reapAllProcesses, *(), **{})()>
_________________________________________________________________________ PublicAPITests.test_eventloop_api_reactor _________________________________________________________________________

self = <crochet.tests.test_api.PublicAPITests testMethod=test_eventloop_api_reactor>

    def test_eventloop_api_reactor(self):
        """
        The publicly exposed EventLoop will, when setup, use the global
        reactor.
        """
        from twisted.internet import reactor
>       _main.no_setup()

crochet/tests/test_api.py:1118:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
crochet/_util.py:12: in _synced
    return method(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <crochet._eventloop.EventLoop object at 0x7fa5f0b46c10>

    @synchronized
    def no_setup(self):
        """
        Initialize the crochet library with no side effects.

        No reactor will be started, logging is uneffected, etc.. Future calls
        to setup() will have no effect. This is useful for applications that
        intend to run Twisted's reactor themselves, and so do not want
        libraries using crochet to attempt to start it on their own.

        If no_setup() is called after setup(), a RuntimeError is raised.
        """
        if self._started:
>           raise RuntimeError(
                "no_setup() is intended to be called once, by a"
                " Twisted application, before any libraries "
                "using crochet are imported and call setup().")
E           RuntimeError: no_setup() is intended to be called once, by a Twisted application, before any libraries using crochet are imported and call setup().

crochet/_eventloop.py:393: RuntimeError
_______________________________________________________________________ ThreadLogObserverTest.test_ioThreadUnchanged ________________________________________________________________________

self = <crochet.tests.test_logging.ThreadLogObserverTest testMethod=test_ioThreadUnchanged>

    def test_ioThreadUnchanged(self):
        """
        ThreadLogObserver does not change the Twisted I/O thread (which is
        supposed to match the thread the main reactor is running in.)
        """
        threadLog = ThreadLogObserver(None)
        threadLog.stop()
        threadLog._thread.join()
>       self.assertIn(
            threadable.ioThread,
            # Either reactor was never run, or run in thread running
            # the tests:
            (None, threading.current_thread().ident))

crochet/tests/test_logging.py:86:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <crochet.tests.test_logging.ThreadLogObserverTest testMethod=test_ioThreadUnchanged>, containee = 140350667507392, container = (None, 140350712317760), msg = None

    def assertIn(self, containee, container, msg=None):
        """
        Fail the test if C{containee} is not found in C{container}.

        @param containee: the value that should be in C{container}
        @param container: a sequence type, or in the case of a mapping type,
                          will follow semantics of 'if key in dict.keys()'
        @param msg: if msg is None, then the failure message will be
                    '%r not in %r' % (first, second)
        """
        if containee not in container:
>           raise self.failureException(msg or f"{containee!r} not in {container!r}")
E           twisted.trial.unittest.FailTest: 140350667507392 not in (None, 140350712317760)

/usr/lib/python3.8/site-packages/twisted/trial/_synctest.py:506: FailTest
===================================================================================== warnings summary ======================================================================================
crochet/tests/test_api.py:15
  /home/tkloczko/rpmbuild/BUILD/crochet-2.0.0/crochet/tests/test_api.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

crochet/tests/test_setup.py::ProcessSetupTests::test_posix
  /home/tkloczko/rpmbuild/BUILD/crochet-2.0.0/crochet/tests/test_setup.py:261: DeprecationWarning: Please use assertEqual instead.
    self.assertEquals(reaps, [1])

crochet/tests/test_setup.py::ProcessSetupTests::test_posix
  /home/tkloczko/rpmbuild/BUILD/crochet-2.0.0/crochet/tests/test_setup.py:263: DeprecationWarning: Please use assertEqual instead.
    self.assertEquals(reaps, [1, 1])

crochet/tests/test_setup.py::ProcessSetupTests::test_posix
  /home/tkloczko/rpmbuild/BUILD/crochet-2.0.0/crochet/tests/test_setup.py:265: DeprecationWarning: Please use assertEqual instead.
    self.assertEquals(reaps, [1, 1, 1])

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================== short test summary info ==================================================================================
SKIPPED [1] crochet/tests/test_api.py:497: This test is too fragile (and insufficient) on Python 3 - see https://github.com/itamarst/crochet/issues/43
SKIPPED [1] crochet/tests/test_mypy.py:280: Tests require mypy to be installed.
SKIPPED [1] crochet/tests/test_mypy.py:337: Tests require mypy to be installed.
SKIPPED [1] crochet/tests/test_mypy.py:320: Tests require mypy to be installed.
SKIPPED [1] crochet/tests/test_mypy.py:295: Tests require mypy to be installed.
SKIPPED [1] crochet/tests/test_mypy.py:366: Tests require mypy to be installed.
SKIPPED [1] crochet/tests/test_mypy.py:24: Tests require mypy to be installed.
SKIPPED [1] crochet/tests/test_mypy.py:381: Tests require mypy to be installed.
SKIPPED [1] crochet/tests/test_mypy.py:140: Tests require mypy to be installed.
SKIPPED [1] crochet/tests/test_mypy.py:246: Tests require mypy to be installed.
SKIPPED [1] crochet/tests/test_mypy.py:54: Tests require mypy to be installed.
SKIPPED [1] crochet/tests/test_mypy.py:31: Tests require mypy to be installed.
SKIPPED [1] crochet/tests/test_mypy.py:466: Tests require mypy to be installed.
SKIPPED [1] crochet/tests/test_mypy.py:433: Tests require mypy to be installed.
SKIPPED [1] crochet/tests/test_mypy.py:404: Tests require mypy to be installed.
SKIPPED [1] crochet/tests/test_setup.py:267: This test is for non-POSIX systems.
SKIPPED [1] crochet/tests/test_setup.py:329: This test is for Twisted < 15.2.
FAILED crochet/tests/test_api.py::ResultRegistryTests::test_runs_with_lock - Failed: NOTE: Incompatible Exception Representation, displaying natively:
FAILED crochet/tests/test_api.py::PublicAPITests::test_eventloop_api_reactor - RuntimeError: no_setup() is intended to be called once, by a Twisted application, before any libraries using crochet are imported and call setup().
FAILED crochet/tests/test_logging.py::ThreadLogObserverTest::test_ioThreadUnchanged - twisted.trial.unittest.FailTest: 140350667507392 not in (None, 140350712317760)
=================================================================== 3 failed, 94 passed, 17 skipped, 4 warnings in 7.79s ====================================================================

Here is list of installed modules in build env

Package                       Version
----------------------------- -------
alabaster                     0.7.13
asttokens                     2.2.1
attrs                         23.1.0
Automat                       22.10.0
Babel                         2.12.1
backcall                      0.2.0
build                         0.10.0
charset-normalizer            3.1.0
constantly                    15.1.0
decorator                     5.1.1
distro                        1.8.0
docutils                      0.19
exceptiongroup                1.1.1
executing                     1.2.0
gpg                           1.20.0
hyperlink                     21.0.0
idna                          3.4
imagesize                     1.4.1
importlib-metadata            6.7.0
incremental                   22.10.0
iniconfig                     2.0.0
installer                     0.7.0
ipython                       8.12.0
jedi                          0.18.2
Jinja2                        3.1.2
libcomps                      0.1.19
MarkupSafe                    2.1.2
matplotlib-inline             0.1.6
packaging                     23.1
parso                         0.8.3
pexpect                       4.8.0
pickleshare                   0.7.5
pluggy                        1.0.0
prompt-toolkit                3.0.38
ptyprocess                    0.7.0
pure-eval                     0.2.2
Pygments                      2.15.1
pyproject_hooks               1.0.0
pytest                        7.4.0
python-dateutil               2.8.2
pytz                          2023.2
requests                      2.31.0
setuptools                    68.0.0
six                           1.16.0
snowballstemmer               2.2.0
Sphinx                        6.2.1
sphinxcontrib-applehelp       1.0.4
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.5
stack-data                    0.6.2
tomli                         2.0.1
traitlets                     5.9.0
Twisted                       22.10.0
typing_extensions             4.6.3
urllib3                       1.26.15
wcwidth                       0.2.6
wheel                         0.40.0
wrapt                         1.14.1
zipp                          3.15.0
zope.event                    4.5.0
zope.interface                5.5.2

[itamarst]

The way to run tests is by running tox, e.g. tox -e py311. Or if you want to do it directly, python -m unittest discover -v crochet.tests.

I will however take a look since this suggests fragile tests.

[itamarst]

I found and fixed the issue, but in general I don't recommend just using pytest for any random project; it does Magic™ that occasionally breaks things, as in this case. And I still recommend you use the above command to run Crochet's tests.

[kloczek]

I found and fixed the issue, but in general I don't recommend just using pytest for any random project; it does Magic™ that occasionally breaks things, as in this case. And I still recommend you use the above command to run Crochet's tests.

I'm building packages with python modules in dedicated build envs in which are installed only resources listed in rpm spec file BuildRequirea. In that scenario use tox is kind of double wrapping
tox additionally if something is missing downloads .whl archives from pypi (I'm not interested to test anything against pypi but against my own packaged modules.

Just tested new 2.1.1 and all looks good now

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-crochet-2.1.1-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-crochet-2.1.1-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network'
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.8.17, pytest-7.4.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/crochet-2.1.1
plugins: mock-3.11.1, anyio-3.6.2, asyncio-0.21.0, aiohttp-1.0.4, httpbin-2.0.0
asyncio: mode=strict
collected 113 items

crochet/tests/test_api.py ................s..............................................                                                                                             [ 55%]
crochet/tests/test_logging.py ....                                                                                                                                                    [ 59%]
crochet/tests/test_mypy.py ..............                                                                                                                                             [ 71%]
crochet/tests/test_process.py .                                                                                                                                                       [ 72%]
crochet/tests/test_resultstore.py .....                                                                                                                                               [ 76%]
crochet/tests/test_setup.py ............s...s                                                                                                                                         [ 92%]
crochet/tests/test_shutdown.py .....                                                                                                                                                  [ 96%]
crochet/tests/test_util.py ....                                                                                                                                                       [100%]

================================================================================== short test summary info ==================================================================================
SKIPPED [1] crochet/tests/test_api.py:497: This test is too fragile (and insufficient) on Python 3 - see https://github.com/itamarst/crochet/issues/43
SKIPPED [1] crochet/tests/test_setup.py:267: This test is for non-POSIX systems.
SKIPPED [1] crochet/tests/test_setup.py:329: This test is for Twisted < 15.2.
========================================================================= 110 passed, 3 skipped in 67.77s (0:01:07) =========================================================================

Thank you 👍 😄