bpo-32972: Add unittest.AsyncioTestCase #10296

dave-shawley · 2018-11-02T11:25:04Z

This PR adds co-routine support to unittest by adding a new sub-class of unittest.TestCase named unittest.AsyncioTestCase. It extends unittest.TestCase by adding the following functionality:

a new event loop is created and terminated with each test method & the same event loop is used throughout the entire test method execution from setup through cleanup
asynchronous test setup and teardown are supported by new methods - asyncSetUp and asyncTearDown
test methods that are decorated with the async keyword are run on the event loop. Undecorated methods are run as normal methods.
cleanup hooks that are decorated with the async keyword are run on the event loop. Undecorated methods are run as normal methods.

I made the following conscious decisions during the implementation that may be somewhat controversial so I want to mention them here (in no particular order) along with the rationale.

there is no direct support for other event loop implementation: other event loop implementations can be plugged in using a custom EventLoopPolicy implementation. It can be inserted by overriding setUpClass. I believe that other loop implementations can either plug in using the event loop policy or provide their own asynchronous test case implementation (e.g., tornado.testing).
asyncio.iscoroutine is used to detect async methods: the two places that I need to detect whether a callable is a co-routine or not are: (a) test methods and (b) cleanup hooks added with addCleanup. Both are explicit (opt-in) actions so they are completely controlled by the test writer.
outstanding asyncio Tasks are not terminated: I'm still on the fence about whether the test should fail if there are outstanding tasks or if it should gather them.
async test case is a separate class: I decided to use a sub-class so that it is immediately obvious that a test methods are intended to run on the event loop. This also insulates unittest.TestCase from unexpected breakages and maintains it's current behavior.

Open Questions

What should happen when len(asyncio.all_tasks()) is non-zero after cleanups are run?
Should unittest.AsyncioTestCase include a global test run timeout?

I still have to amend commits to update documentation but I wanted to have the discussion about the implementation choices before I started writing up examples and updating the existing documentation suite.

https://bugs.python.org/issue32972

This is the core of running a test. It is also the portion of run() that requires specialization for asyncio support.

This method is essentially ``_runTest`` with a Outcome object that does nothing. This commit inserts a new class that is a no-op Outcome so that the method can be re-used.

See discussion under https://bugs.python.org/issue32972

This makes AsyncioTestCase.doCleanups() safe to call at any time. The previous version terminated the event loop unnecessarily.

matrixise · 2018-11-02T13:32:30Z

Hi,

Thanks for your PR, but this one is not related to the issue. In this PR, you add the support for asyncio but it is not the topic of the bpo issue.

Maybe you have to open an other issue and assign this PR to the issue.

dave-shawley · 2018-11-02T13:47:20Z

@matrixise ick ... early morning mistake. This is associated with BPO 32972, renaming the PR

matrixise

Thank for your contribution but could you add a NEWS entry with the blurb tool.

pip install blurb

matrixise · 2018-11-04T06:38:16Z

Lib/unittest/case.py

+    is stopped and closed.
+
+    When subclassing AsyncioTestCase, the event loop is available
+    via the ``event_loop`` property.  Tests can assume that


about event_loop I am not sure, usually we found loop in the documentation of asyncio

matrixise · 2018-11-04T06:41:39Z

Lib/unittest/case.py

+        self.__event_loop = None
+
+    def doCleanups(self):
+        """


The docstring should start on the first line, https://www.python.org/dev/peps/pep-0008/#documentation-strings

matrixise · 2018-11-04T06:47:44Z

Lib/unittest/case.py

@@ -81,6 +82,16 @@ def testPartExecutor(self, test_case, isTest=False):
            self.success = self.success and old_success


+class _IgnoredOutcome(object):


In Python3, we prefer to use the new style
https://docs.python.org/3/reference/compound_stmts.html#class-definitions

# new style class _IgnoredOutcome: pass

is equivalent to

# old style class _IgnoredOutcome(object): pass

so in this case, use the new style and not the old one.

matrixise · 2018-11-04T06:49:38Z

Lib/unittest/test/test_asyncio_test_case.py

+import unittest
+
+
+class Test(object):


See my previous comment about the construction of a class

class Test: pass

is better

matrixise · 2018-11-04T06:54:52Z

Lib/unittest/case.py

+        self.tearDown()
+
+    @property
+    def event_loop(self):


in the documentation, we prefer to use the loop variable for the reference to the event loop.

Examples:
:meth:`loop.call_soon` or :meth:`loop.set_debug` etc...

maybe you should do the same.

matrixise · 2018-11-04T06:56:10Z

Lib/unittest/test/test_asyncio_test_case.py

+        raise AssertionError(f'test {test_class} unexpectedly failed')
+
+
+class EventLoopTracking(object):


just class EventLoopTracking:

See python#10296

See python#10296.

`loop` is consistent with the Standard Library spelling of the current event loop. See python#10296

See python#10296

dave-shawley · 2018-11-04T13:07:47Z

@matrixise - thanks for the review. I amended a few commits to address your feedback. I still intend to add some documentation in Doc/library/unittest.rst and Doc/library/asyncio.rst as well. I wanted to make sure that the implementation was acceptable before starting the documentation related tasks.

PetterS · 2018-11-13T12:57:58Z

Lib/unittest/case.py

+        if self.__loop is None:
+            self.__loop = asyncio.new_event_loop()
+            self.__loop.set_debug(True)
+            asyncio.set_event_loop(self.__loop)


Does the test case have to modify the global state like this? It is calling self.loop.run_until_complete when running things.

I think that it does. This is the only way to ensure that the code under test gets the correct loop.

asvetlov · 2019-05-17T18:59:33Z

Please take a look on #13386

asvetlov · 2019-05-29T09:36:09Z

Implemented by #13386

dave-shawley added 7 commits November 2, 2018 06:44

unittest.TestRunner: introduce _runTest method.

564ada6

This is the core of running a test. It is also the portion of run() that requires specialization for asyncio support.

unittest: Refactor TestCase.debug.

9a780c6

This method is essentially ``_runTest`` with a Outcome object that does nothing. This commit inserts a new class that is a no-op Outcome so that the method can be re-used.

Add unittest.AsyncioTestCase.

60d3514

unittest: Introduce asyncSetUp/TearDown.

6fa2911

See discussion under https://bugs.python.org/issue32972

unittest: Introduce TestCase._terminateTest.

231aab5

This makes AsyncioTestCase.doCleanups() safe to call at any time. The previous version terminated the event loop unnecessarily.

unittest: Shutdown async generators.

88a697c

unittest: Enable event loop debug flag.

9f6263e

dave-shawley requested review from 1st1 and asvetlov as code owners November 2, 2018 11:25

the-knights-who-say-ni added the CLA signed label Nov 2, 2018

bedevere-bot added the awaiting review label Nov 2, 2018

dave-shawley changed the title ~~bpo-32971: Add unittest.AsyncioTestCase~~ bpo-32972: Add unittest.AsyncioTestCase Nov 2, 2018

matrixise requested changes Nov 4, 2018

View reviewed changes

bedevere-bot added awaiting core review and removed awaiting review labels Nov 4, 2018

dave-shawley added 4 commits November 4, 2018 07:40

bpo-32972: Switch to "newer style" classes.

694e6d5

See python#10296

bpo-32972: Docstring cleanup in unittest.case.

adb4c96

See python#10296.

bpo-32972: Rename AsyncioTestCase.event_loop.

3c0ac43

`loop` is consistent with the Standard Library spelling of the current event loop. See python#10296

bpo-32972: Add NEWS blurb.

7eb7899

See python#10296

Merge branch 'master' into bpo-32971-async-test-runner

eed94cf

PetterS reviewed Nov 13, 2018

View reviewed changes

asvetlov closed this May 29, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-32972: Add unittest.AsyncioTestCase #10296

bpo-32972: Add unittest.AsyncioTestCase #10296

dave-shawley commented Nov 2, 2018 •

edited

Loading

matrixise commented Nov 2, 2018

dave-shawley commented Nov 2, 2018

matrixise left a comment

matrixise Nov 4, 2018

matrixise Nov 4, 2018

matrixise Nov 4, 2018

matrixise Nov 4, 2018

matrixise Nov 4, 2018

matrixise Nov 4, 2018

dave-shawley commented Nov 4, 2018

PetterS Nov 13, 2018

dave-shawley Nov 15, 2018

asvetlov commented May 17, 2019

asvetlov commented May 29, 2019

		@@ -81,6 +82,16 @@ def testPartExecutor(self, test_case, isTest=False):
		self.success = self.success and old_success


		class _IgnoredOutcome(object):

		raise AssertionError(f'test {test_class} unexpectedly failed')


		class EventLoopTracking(object):

bpo-32972: Add unittest.AsyncioTestCase #10296

bpo-32972: Add unittest.AsyncioTestCase #10296

Conversation

dave-shawley commented Nov 2, 2018 • edited Loading

matrixise commented Nov 2, 2018

dave-shawley commented Nov 2, 2018

matrixise left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dave-shawley commented Nov 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asvetlov commented May 17, 2019

asvetlov commented May 29, 2019

dave-shawley commented Nov 2, 2018 •

edited

Loading