test console_task which aggregates test results #6646

illicitonion · 2018-10-17T19:21:12Z

First cut at #6006

stuhood

Looks good! I guess that testrunner_test_mixin.TestResult is fairly fluffy, so not much use in preserving it.

stuhood · 2018-10-17T22:13:04Z

tests/python/pants_test/engine/test_test.py

+  for test_result in test_results:
+    wrote_any_stdout |= bool(test_result.stdout)
+    # Assume \n-terminated
+    console.write_stdout(test_result.stdout)


I have so far been assuming (and I think it was in the v2 HLD) that we would render test runs completing as soon as they did. So the test stderr/stdout rendering would happen in #6003 by writing to a logger, and the run would "slow fail"... return a failing test result.

I thought the HLD said we would aggregate, and that we would run the thing running the test itself in some kind of interactive mode if we wanted things more eagerly...

I also think we started talking about introducing a "select first" version of Get so that we could be more eager if we wanted to be...

stuhood · 2018-10-17T22:13:48Z

tests/python/pants_test/engine/test_test.py

+    console.write_stdout('\n')
+  for (hydrated_target, test_result) in zip(hydrated_targets, test_results):
+    console.print_stdout('{0:80}.....{1:>10}'.format(hydrated_target.address.reference(), test_result.status))
+  # TODO: Communicate an exit code out of the console_rule (if tests failed)


This rule should raise an exception for failures.

stuhood · 2018-10-17T22:23:26Z

tests/python/pants_test/engine/test_test.py

+
+
+# TODO: Move to production code
+@console_rule('test', [Select(Console), Select(TestResult)])


This should Select(HydratedTargets) probably.

Downside of this unit test harness, I suppose... very ducktyped.

illicitonion · 2018-10-18T15:33:16Z

Everything except the HEAD commit are dependant PRs

stuhood

This looks awesome.

stuhood · 2018-10-18T16:05:36Z

src/python/pants/rules/core/core_test_model.py

+
+# This would be called TestResult, but then PyTest outputs a warning that it detected TestResult
+# was a test class, but couldn't treat it like a test class, because it has a constructor.
+class GenericTestResult(datatype([


https://stackoverflow.com/questions/42268401/how-can-i-get-pytest-to-ignore-test-classes-that-dont-subclass-unittest

Is there a particular thing you're suggesting here?

I'm not sure how the existing TestResult class doesn't fall foul of this, given I can't see any obvious pytest config in the repo...

I'm suggesting renaming this to TestResult and marking it:

__test__ = false

...to resolve the comment you have about naming here.

stuhood · 2018-10-18T16:08:17Z

src/python/pants/rules/core/python_test_runner.py

+from pants.engine.selectors import Select
+from pants.rules.core.core_test_model import GenericTestResult, Status
+
+


So, I think I just remembered that #6652 isn't quite accurate: commented there.

You should be able to declare this in src/python/pants/backend/python/register.py.

Will move tomorrow :)

stuhood · 2018-10-18T16:08:47Z

src/python/pants/rules/core/python_test_runner.py

+
+
+# This class currently exists so that other rules could be added which turned a HydratedTarget into
+# a language-specific test result, and could be installed alongside run_python_test.


Can reference #4535 (or not).

stuhood · 2018-10-18T16:10:45Z

src/python/pants/rules/core/test.py

+
+@rule(GenericTestResult, [Select(HydratedTarget)])
+def test_coordinator(target):
+  # This should do an instance match, or canonicalise the adaptor type, or something


This should definitely reference #4535.

stuhood · 2018-10-18T16:12:30Z

src/python/pants/rules/core/test.py

+
+@console_rule('test', [Select(Console), Select(HydratedTargets)])
+def fast_test(console, targets):
+  test_results = yield [Get(GenericTestResult, HydratedTarget, target) for target in targets]


So, I believe that further decoupling is possible here.

It should be possible for this rule to Select(BuildFileAddresses) (which is the first stage of parsing: "addresses for the spec roots"), and then select a TestResult for each of those. That should allow build file parsing to operate in parallel with test runs.

stuhood · 2018-10-18T16:13:23Z

tests/python/pants_test/engine/test_test.py

+from __future__ import absolute_import, division, print_function, unicode_literals
+
+from builtins import str
+


This file should maybe go under tests/python/pants_test/rules/ ...? Maybe?

stuhood · 2018-10-18T17:51:38Z

Rebased.

stuhood

Awesome. I made a minor tweak to remove a stale comment. Merging.

stuhood reviewed Oct 17, 2018

View reviewed changes

illicitonion force-pushed the dwagnerhall/v2/testrunner branch from 21e8856 to 9c6a801 Compare October 18, 2018 15:31

illicitonion changed the title ~~WIP: test console_task which aggregates test results~~ test console_task which aggregates test results Oct 18, 2018

stuhood approved these changes Oct 18, 2018

View reviewed changes

stuhood force-pushed the dwagnerhall/v2/testrunner branch from fd11d13 to 2d388a1 Compare October 18, 2018 17:51

illicitonion force-pushed the dwagnerhall/v2/testrunner branch from 2d388a1 to da57200 Compare October 19, 2018 10:05

Review comments

a962fb6

illicitonion force-pushed the dwagnerhall/v2/testrunner branch from da57200 to a962fb6 Compare October 19, 2018 11:02

Fix tests

dc24432

This was referenced Oct 19, 2018

Python test runner uses pytest #6661

Merged

Implement @console_rule for aggregate test runner/orchestrator #6006

Closed

Create initial @rules for pytest runner products #6003

Closed

Update core_test_model.py

c9e6f8c

stuhood approved these changes Oct 22, 2018

View reviewed changes

stuhood merged commit d146be8 into pantsbuild:master Oct 22, 2018

stuhood deleted the dwagnerhall/v2/testrunner branch October 22, 2018 21:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test console_task which aggregates test results #6646

test console_task which aggregates test results #6646

illicitonion commented Oct 17, 2018

stuhood left a comment

stuhood Oct 17, 2018 •

edited

Loading

illicitonion Oct 18, 2018

stuhood Oct 17, 2018

stuhood Oct 17, 2018

stuhood Oct 17, 2018

illicitonion commented Oct 18, 2018

stuhood left a comment

stuhood Oct 18, 2018

illicitonion Oct 18, 2018

stuhood Oct 18, 2018

stuhood Oct 18, 2018

illicitonion Oct 18, 2018

stuhood Oct 18, 2018

stuhood Oct 18, 2018

stuhood Oct 18, 2018

illicitonion Oct 18, 2018

stuhood Oct 18, 2018

stuhood commented Oct 18, 2018

stuhood left a comment



		# TODO: Move to production code
		@console_rule('test', [Select(Console), Select(TestResult)])

		from pants.engine.selectors import Select
		from pants.rules.core.core_test_model import GenericTestResult, Status



		# This class currently exists so that other rules could be added which turned a HydratedTarget into
		# a language-specific test result, and could be installed alongside run_python_test.

		from __future__ import absolute_import, division, print_function, unicode_literals

		from builtins import str

test console_task which aggregates test results #6646

test console_task which aggregates test results #6646

Conversation

illicitonion commented Oct 17, 2018

stuhood left a comment

Choose a reason for hiding this comment

stuhood Oct 17, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

illicitonion commented Oct 18, 2018

stuhood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood commented Oct 18, 2018

stuhood left a comment

Choose a reason for hiding this comment

stuhood Oct 17, 2018 •

edited

Loading