Execution order "by-defect" does not support parameterized tests with @dataProvider #3246
Comments
|
Hm, when deleting the incomplete test it still behaves badly and does not change the execution order at all; maybe this is because the failing test is a parameterized one? |
|
Oh yes, it looks like it is the
|
reinholdfuereder
changed the title
|
Rather irrelevant by comparison, but I am also wondering why the execution order "by-defect" also seems to consider incomplete tests: |
|
@epdenouden ping |
|
@reinholdfuereder thank you for the super detailed issue! I will look into it later this week, it is currently not impeding anything, right? |
|
@epdenouden No stress at all from my side, just take the time you need; thanks in advance... |
|
@sebastianbergmann Can you assign this issue to me? makes it much easier to keep track of it. :) |
|
@epdenouden GitHub only allows assigning of issues to people that have push access, sorry. |
|
@sebastianbergmann Thanks for the clarification. I had no idea. Quickfix by tracking it via my own issues list. |
|
@reinholdfuereder Apologies for the delay, I have some time this weekend to finally look into this. To answer one question quickly: but I am also wondering why the execution order "by-defect" also seems to consider incomplete tests?
Other modes of failure like skipping and incomplete are considered to be non-blocking annoyances. We tried a few different settings for this and quickly found out that sorting skipped/incomplete tests to the front would result in some TestSuites being first every single run. The real-world reason for this is often environment dependencies like "only run this test when we have run on a Unix-like system". Issue #2085 is an example of this as it requires OS-specific functionality. |
|
@epdenouden Thanks for the explanation: this sounds very reasonable |
In general, the
|
Okay, I nonetheless assume it should not be too difficult to write a small re-producer based on the aforementioned failing one. Unfortunately I will be in the office and have access to the code no earlier than October 30. However, I guess any further delay does not really matter... So please be patient... |
|
@reinholdfuereder No worries about the timing. As you can see, I'm also not as quick as I like with working on this, with the new(ish) job and all. I will just go ahead and try to break the edge cases I have sketched out above ^^^ and report back here. |
|
@reinholdfuereder I have found a scenario that fails in a way that looks a lot like your report. Have added a test that reproduces the failure. Caching previous test results does not sort tests with defective rows to the front. Remove the cache and do a first test run: A second run will sort the failing data rows to the front of their parent dataprovider, but the tests using those with The issue can be found somewhere in the logic that bubbles cached test results up the search tree. Although it's just a minor bug, it can make the tool feel unpredictable or make it look like the cache doesn't work. Good to have it fixed. It might even be a nice performance gain for developers working on unit tests for e.g. input validation logic with sizeable dataproviders like Thank you for providing such detailed and ongoing feedback! |
|
@reinholdfuereder After some more investigation I have opened #3396 and will provide an implementation for this in the coming week. |
|
@sebastianbergmann This ticket can be closed now. Everything here is covered by #3396 and its related pull request #3401 :-) |
mark the parent test failed.
In my test set (executed via XML configuration) there is (coincidentally) (1) one incomplete, (2) one skipped and (3) one failing test. I just started playing around with the new cool feature of running tests first that failed in a previous run, but it does not behave as expected:
=> My expectation would have been, that it starts with the single failing test (or in general all failing test cases or maybe all test classes with failing test cases?); and executing the other tests (including incomplete or skipped ones) in the previous/default order.
Not sure if it is needed or helpful, but the cache result file starts like this:
...whereas:
sendQuoteRequestTestis the incomplete testformatNumberForMobileTelLink with data set #8is the failing test (see above)testPhpWarningIsConvertedToPhpUnitException3is the skipped testThe text was updated successfully, but these errors were encountered: