pytest statuses "expected-fail" and "unexpectedly passing" not yet reflected in Category #47
Comments
|
In the meantime, |
|
Can't check because on the phone, but if Category is a flagint enum, it should hopefully be possible to cover both the more specific categories of pytest and frameworks that only offer pass/fail/skip with one logic. |
|
...if that's the reason you want to go for category.misc, that is. |
|
I think I may have misunderstood you. Why do you want The Category is not directly user visible, but it does determine the background colour (currently, green for Category.OK = passing tests, red for Category.FAIL = failing tests, blank for Category.PENDING = pending tests and grey for Category.SKIP = the rest). Do you think that xfail, xpass and skip should all have different colours? |
|
Ok, then I misunderstood. If it's primarily about the color scheme, I think xfail and xpass should be green, as they represent the test outcome matches the user's expectations, i.e. the test failed or passed, respectively. I wonder if pytest can colour output, that could also offer indications. |
|
I understand your reasoning but I'm wondering whether this may actually be more confusing. For the record, if you run py.test with the I'm going to leave this issue open for the moment. |
I'm afraid there is some misunderstanding here: |
|
Ah, sounds reasonable, thanks! |
Instead of writing events for each phase (setup, call, teardown) of a test (and eventually displaying them in the datatree widget) we now accumulate results over the individual test phases and only then write the final result of the entire test upon test completion using the pytest_runtest_logfinish hook. This ensures the following: 1) Capturing of output from the teardown phase even if it passes without an error. This fixes spyder-ide#127. 2) Differentiating between the outcome of the test itself (the call phase) and the entire test as a whole. An error during teardown, for instance, used to overwrite the (potentially passed) test result of the call phase. Now we get a passed test with an error during teardown. So in the datatree widget, the test status (first column) is no longer 1:1 linked with the color of this line (see also 3)). 3) Better handling of xfailed and xpassed tests which now show the correct status `xfailed` or `xpassed` and are colored green and red respectively (instead of status `skipped` and gray color for both of them). This fixes spyder-ide#47. 4) Better messages: the first message, which is usually the most important one, will be placed in the message column of the datatree widget. Further messages will be _appended_ to the extra_text field (instead of overwriting messages from previous test phases). If the first message spans over multiple lines then only its first line will be displayed in the message column and the complete message will be repeated in the extra_text field for better readability. This improves the visual impression of the datatree widget so that each (collapsed) row is exactly one line high.
Hi!
Just to note this (can't currently work on this), the pytest statuses "expected to fail" (XFAIL, shown as an "x" in test results) and "“unexpectedly passing” (XPASS) are not yet available in https://github.com/spyder-ide/spyder-unittest/blob/52af625e6a526c218d3a7930fdd348ab5716c724/spyder_unittest/backend/testrunner.py#L28, and probably should be at some point.
See also http://doc.pytest.org/en/latest/skipping.html?highlight=expected%20fail#skip-and-xfail-dealing-with-tests-that-cannot-succeed
The text was updated successfully, but these errors were encountered: