You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It turns out that #4046 did have an effect on reported coverage, but only in the context of our CI. (For example, see before vs. after.)
For whatever reason, istanbul is now using the coverage reported for the IE test run, which has lower coverage due to ripple-related tests being skipped. It used to report the coverage numbers seen for Chrome / Firefox / iOS Safari, when we were running all CI unit tests through browsers on Sauce Labs.
I'd like to be able to say that it's reporting the coverage based on the last finished test, but I can't even conclude that, because IE always finished last, even previously when we were running all browsers on Sauce Labs.
There are multiple possible ways to work around this or attempt to resolve this:
Reduce the required coverage from 95% to 94% to account for the dip on IE (which is a <1% difference)
Figure out a way to not skip ripple-related tests in IE (in Dialog, Select, and Tab) so this becomes a non-issue (currently we're only able to test by detecting the ripple-upgraded class, which is never applied on IE)
Split up our karma conf into 2 configurations, to run sauce labs before local, so that the report reflects Chrome/Firefox coverage (this would make unit tests take a minute longer because they can't be parallelized between local and SL)
See whether upgrading to nyc (istanbul's new version) would pick up reports from multiple browsers better... but documentation seems confusing at best and migration documentation seems nonexistent, and I have no idea if it will even fix the issue
Improve coverage elsewhere to get us further away from the failure threshold (or add comments to avoid instrumenting code that is infeasible to cover)
Option 1 is the quickest short-term fix if we encounter more problems with dipping below coverage in PRs (we hit it in #4055). Option 2 would probably be preferable to solve this. Option 5 is probably also something we should pursue, eventually.
The text was updated successfully, but these errors were encountered:
It turns out that #4046 did have an effect on reported coverage, but only in the context of our CI. (For example, see before vs. after.)
For whatever reason, istanbul is now using the coverage reported for the IE test run, which has lower coverage due to ripple-related tests being skipped. It used to report the coverage numbers seen for Chrome / Firefox / iOS Safari, when we were running all CI unit tests through browsers on Sauce Labs.
I'd like to be able to say that it's reporting the coverage based on the last finished test, but I can't even conclude that, because IE always finished last, even previously when we were running all browsers on Sauce Labs.
There are multiple possible ways to work around this or attempt to resolve this:
nyc
(istanbul's new version) would pick up reports from multiple browsers better... but documentation seems confusing at best and migration documentation seems nonexistent, and I have no idea if it will even fix the issueOption 1 is the quickest short-term fix if we encounter more problems with dipping below coverage in PRs (we hit it in #4055). Option 2 would probably be preferable to solve this. Option 5 is probably also something we should pursue, eventually.
The text was updated successfully, but these errors were encountered: