Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stability script: normalize results format #5330

Merged

Conversation

jugglinmike
Copy link
Contributor

The "check stability" script parses test results into a List of tuples.
Later operations expect the fourth element to be a (possibly empty) List
of "messages". While this expectation is satisfied when the instability
is the result of a subtest's behavior, the parsing operation previously
violated the contract when interpreting test failures.

Update the parsing logic to provide an empty List in cases where a test
(rather than a subtest) is the source of instability.

The "check stability" script parses test results into a List of tuples.
Later operations expect the fourth element to be a (possibly empty) List
of "messages". While this expectation is satisfied when the instability
is the result of a subtest's behavior, the parsing operation previously
violated the contract when interpreting test failures.

Update the parsing logic to provide an empty List in cases where a test
(rather than a subtest) is the source of instability.
@ghost
Copy link

ghost commented Apr 3, 2017

View the complete job log.

Firefox (nightly channel)

Testing web-platform-tests at revision 161fed7
Using browser at version BuildID 20170403100136; SourceStamp 38894655c89e68bcd8f45d31a0d3005f2c2b53db
Starting 10 test iterations
All results were stable

All results

2 tests ran
/service-workers/service-worker/fetch-event.https.html
Subtest Results Messages
OK
Service Worker responds to fetch event with string PASS
Service Worker responds to fetch event with blob body PASS
Service Worker responds to fetch event with the referrer URL PASS
Service Worker responds to fetch event with the referrer policy FAIL assert_unreached: unexpected rejection: NetworkError when attempting to fetch resource. Reached unreachable code
Service Worker responds to fetch event with an existing client id PASS
Service Worker does not respond to fetch event PASS
Service Worker responds to fetch event with null response body PASS
Service Worker fetches other file in fetch event PASS
Service Worker responds to fetch event with POST form PASS
Multiple calls of respondWith must throw InvalidStateErrors PASS
Service Worker event.respondWith must set the used flag PASS
Service Worker should expose FetchEvent URL fragments. PASS
Service Worker responds to fetch event with the correct cache types PASS
Service Worker should intercept EventSource PASS
Service Worker responds to fetch event with the correct integrity_metadata PASS
/service-workers/service-worker/navigation-preload/redirect.https.html
Subtest Results Messages
TIMEOUT
Navigation Preload redirect response. TIMEOUT Test timed out
Navigation Preload no-location redirect response. NOTRUN
Navigation Preload no-location redirect response with body. NOTRUN
Navigation Preload redirect to the same scope. NOTRUN

@ghost
Copy link

ghost commented Apr 3, 2017

View the complete job log.

Chrome (unstable channel)

Testing web-platform-tests at revision 161fed7
Using browser at version 59.0.3053.3 dev
Starting 10 test iterations
All results were stable

All results

2 tests ran
/service-workers/service-worker/fetch-event.https.html
Subtest Results Messages
OK
Service Worker responds to fetch event with string PASS
Service Worker responds to fetch event with blob body PASS
Service Worker responds to fetch event with the referrer URL PASS
Service Worker responds to fetch event with the referrer policy FAIL assert_unreached: unexpected rejection: assert_equals: Service Worker should respond to fetch with the referrer URL when a member of RequestInit is present expected "Referrer: https://web-platform.test:8443/service-workers/service-worker/resources/simple.html?referrerPolicy\nReferrerPolicy: no-referrer-when-downgrade" but got "Referrer: https://web-platform.test:8443/service-workers/service-worker/resources/undefined\nReferrerPolicy: no-referrer-when-downgrade" Reached unreachable code
Service Worker responds to fetch event with an existing client id PASS
Service Worker does not respond to fetch event PASS
Service Worker responds to fetch event with null response body PASS
Service Worker fetches other file in fetch event PASS
Service Worker responds to fetch event with POST form PASS
Multiple calls of respondWith must throw InvalidStateErrors PASS
Service Worker event.respondWith must set the used flag PASS
Service Worker should expose FetchEvent URL fragments. FAIL assert_unreached: unexpected rejection: assert_equals: Service worker should expose URL fragments in request. expected "Fragment Found :#/some/fragment" but got "Fragment Not Found" Reached unreachable code
Service Worker responds to fetch event with the correct cache types FAIL assert_unreached: unexpected rejection: assert_equals: expected "default" but got "" Reached unreachable code
Service Worker should intercept EventSource FAIL assert_unreached: unexpected rejection: assert_equals: EventSource should bypass the http cache. expected (string) "no-store" but got (undefined) undefined Reached unreachable code
Service Worker responds to fetch event with the correct integrity_metadata PASS
/service-workers/service-worker/navigation-preload/redirect.https.html
Subtest Results Messages
TIMEOUT
Navigation Preload redirect response. TIMEOUT Test timed out
Navigation Preload no-location redirect response. NOTRUN
Navigation Preload no-location redirect response with body. NOTRUN
Navigation Preload redirect to the same scope. NOTRUN

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants