Unit test improvements #12153
Unit test improvements #12153
Conversation
Moreover, remove a left-over reference to `test.py` since that was ported to JavaScript a long time ago.
This might make debugging intermittent failures a bit easier in the future because it allows us to spot unexpected differences in the number of tests being run and allows us to run the tests locally in the same order in case of intermittent failures.
This is possible now that most intermittent unit test failures have been resolved by other patches. There is only one remaining, but it's very rare and doesn't have to block this update anymore.
|
I'm going to do a unit test run that includes the original commit with the failure that went unnoticed before to show that the unit tests now actually fail because of that. I'll remove that commit afterwards. /botio unittest |
From: Bot.io (Linux m4)ReceivedCommand cmd_unittest from @timvandermeij received. Current queue size: 0 Live output at: http://54.67.70.0:8877/e79cd113d2ab405/output.txt |
From: Bot.io (Windows)ReceivedCommand cmd_unittest from @timvandermeij received. Current queue size: 0 Live output at: http://54.215.176.217:8877/b62000140b5d30b/output.txt |
From: Bot.io (Linux m4)FailedFull output at http://54.67.70.0:8877/e79cd113d2ab405/output.txt Total script time: 3.85 mins
|
From: Bot.io (Windows)FailedFull output at http://54.215.176.217:8877/b62000140b5d30b/output.txt Total script time: 5.10 mins
|
|
|
||
| const total = suiteInfo.totalSpecsDefined; | ||
| const seed = suiteInfo.order.seed; | ||
| sendInfo(`Started ${total} tests for ${browser} with seed ${seed}.`); |
There was a problem hiding this comment.
This is really nice, and one of those things that you can't help but wonder why we haven't done years ago!
|
The alternate document commit is removed again. /botio unittest |
From: Bot.io (Linux m4)ReceivedCommand cmd_unittest from @timvandermeij received. Current queue size: 0 Live output at: http://54.67.70.0:8877/26ad8e8dc9e90c0/output.txt |
From: Bot.io (Windows)ReceivedCommand cmd_unittest from @timvandermeij received. Current queue size: 0 Live output at: http://54.215.176.217:8877/3ee03b43ac70729/output.txt |
From: Bot.io (Linux m4)SuccessFull output at http://54.67.70.0:8877/26ad8e8dc9e90c0/output.txt Total script time: 3.78 mins
|
From: Bot.io (Windows)SuccessFull output at http://54.215.176.217:8877/3ee03b43ac70729/output.txt Total script time: 4.60 mins
|
This commit series fixes the part of #12152 that caused the failure to go unnoticed and fixes the part of #12123 about updating Jasmine.
For the latter, I have run the unit tests locally around 100 times and the one remaining intermittent failure never happened once for me. Having thought about this more, I now also believe that it's no longer a blocker for the Jasmine update now that the other intermittent failures have actually been resolved (they happened far more often). We'll keep the issue open to track the last intermittent failure, but it's no longer a blocker for us.