Flaky tests: Several tests in AsyncTest fail when run individually #1065
Labels
code cleanup
Fix issues reported by Sonar or any other code analysis tools
Milestone
In #1024 I made the
AsyncTest
execute faster by reducing timeouts and delays. However, when researching #732 I happened to execute some tests individually, as opposed to executing the entireAsyncTest
or all tests in kiwi. One failed. I then executed every single test individually and found two more that consistently fail when run individually, but pass when you run all the tests or run all tests inAsyncTest
. These tests are failing on my new 2023 M2 MacBook Pro, as well as my old-as-dirt 2012 MacBook Pro (yes, 2012).However, they actually sometimes fail differently on the new vs. old laptop! On the 2023 M2, the
assertThatThrownBy
fails, with the message "Expecting code to raise a throwable" while on the 2012, theassertThatThrownBy
sometimes does not fail, but the subsequent assertion on the task count does fail. Given how much faster 2023 is than the 2012, this is not really all that surprising, and these tests became more likely to fail when I made them faster (#1024) because I reduced the task delay from 100 to 10 milliseconds, and the timeouts from 5 milliseconds to 1. This reduces the margin of error, apparently enough to cause these tests to be flaky.When run individually, I suspect that the threads are being started so quickly that the tasks (which now have a 10ms) are completing before the
waitForXxx
calls are made. And since thewaitForXxx
calls are made inside theassertThatThrownBy
, there is additional overhead by AssertJ that exists when running the tests individually, that may have already been executed when running the entire test suite.I'm not going to try and figure out the exact details of all this, but rather will make targeted changes to these three tests to increase the delays, which should ensure they pass when run individually or as part of the overall test or test suite.
The tests that fail individually are:
WaitFor#shouldThrowAsyncException_WhenTimesOut_BeforeTheFutureCompletes
WaitForAll#shouldThrowAsyncException_WhenTimesOut_BeforeAllFuturesComplete
WaitForAllIgnoringType#shouldThrowAsyncException_WhenTimesOut_BeforeAllFuturesComplete
The text was updated successfully, but these errors were encountered: