Clean up default test output

[JukkaL]

The current test output is a little messy. Consider this test run:

mypy $ ./runtests.py unit-test -a "*MultiAnd*"
PARALLEL 8
SUMMARY  12 tasks selected
0 test cases run, all passed.; running {#0, #1, #2, #3, #4, #5, #6, #7}         
*** OK ***
0 test cases run, all passed.; running {#0, #1, #2, #3, #4, #5, #7, #8}         
*** OK ***
0 test cases run, all passed.; running {#0, #1, #2, #3, #4, #5, #8, #9}         
*** OK ***
0 test cases run, all passed.
*** OK ***
0 test cases run, all passed.; running {#0, #3, #4, #5, #8, #9, #10, #11}       
*** OK ***
0 test cases run, all passed.
*** OK ***
0 test cases run, all passed.
*** OK ***
0 test cases run, all passed.; running {#0, #3, #5, #9, #10, #11}               
*** OK ***
0 test cases run, all passed.; running {#0, #3, #10, #11}                       
*** OK ***
0 test cases run, all passed.; running {#3, #10, #11}                           
*** OK ***
0 test cases run, all passed.0; running {#3, #11}                               
*** OK ***
2 test cases run, all passed.0; running {#3}                                    
*** OK ***
SUMMARY  all 12 tasks passed                    

I'd prefer something like this instead (the intermediate subsets {...} are useful as well and should not be removed, but they should be cleaned up properly):

mypy $ ./runtests.py unit-test -a "*MultiAnd*"
PARALLEL 8
SUMMARY  12 tasks selected
SUMMARY  all 12 tasks passed, all 2 test cases passed
*** OK ***

[o11c]

It might be as simple as just passing ['-q'] as the default arglist.

[JukkaL]

-q doesn't quite do what I want:

• It still shows messages from every suite, and these are intermingled with dots from the test runner.
• It doesn't the total number of tests run at the end, across all suites.

Displaying the {#0, #4, ...} messages is fine as long as they don't corrupt the actual test output.

[o11c]

I'm talking about myunit's -q, not runtest's -q, e.g. ./runtests.py -a -q unit-test. This gives absolutely no intermingling for me.

Total number across all suites is what #908 and the whole "collect first" then "lookup by name" is working towards (there's no reason those need to be done in the same process!)

[JukkaL]

Ah okay I was confused. In that case I'd love to hear more about "collect first" and "lookup by name" and how they can give us cleaner test output :-)

[o11c]

Basically, instead of running ~400 tasks, run all ~2500 individual testcases as direct children of the driver.

The complication comes from the fact that we probably don't want to be spawning a separate python interpreter for every one of those, so we need some sort of pipe with control commands. (Thus why it is out of scope for #908 for now, though the parallelism cost-benefit analysis will likely be beneficial for pythoneval).

[ilevkivskyi]

Is this still relevant? I like the current test output, it is quite terse.