some fixes for intermittent tests failing #1626
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Type of change
Description of change
Lately the tests have been intermittently failing more often, this pull-request solves some of the issues.
I think one of the big issues was related to #1536; it added an asynchronous test that was not treated asynchronously and also included a potential race-condition with between two
setTimeout
s. This had the effect of causing other suites to fail randomly (some of the "before each hook" or "after each hook" failures), which was pretty hard to track down.I think some of the other issues are the result of improper cleanup of stubbed methods and the global XMLHttpRequest object, of which I have a few fixes in this pull-request.
Even with the changes in this branch, I still had one of these fails:
But only once after many runs. I wasn't able to track down the true source of that error as a stack trace would be necessary considering many adapters use XHR handlers that contain a
seatbid
property, and considering the non-deterministic nature of these errors (I think it might depend on file load-order) I wasn't able to catch this bug as it happened.Other information
It may be worthwhile to make a requirement of all future testing suites added to Prebid.js that they must use the
beforeEach(() => {sandbox = sinon.sandbox.create() })
andafterEach(() => { sandbox.restore() })
at the root level of the their suite, and not allow any other uses of thesinon
global, onlysandbox
. This includes using thesandbox.useFakeServer()
for any server request testing. Doing so makes auditing of of proper test cleanup much easier.