Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coop coep reporting testing plan #27659

Draft
wants to merge 18 commits into
base: master
Choose a base branch
from
Draft

Coop coep reporting testing plan #27659

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
584415c
WIP reporting testing plan for COOP/COEP
zcorpan Feb 9, 2021
07e071c
WIP
zcorpan Feb 10, 2021
509aae7
Move Reporting API testing plan to /reporting/README.md
zcorpan Feb 11, 2021
e8a89e4
COEP reporting testing plan
zcorpan Feb 17, 2021
10585d0
Elaborate on relevant settings object
zcorpan Mar 22, 2021
39c5542
Clean up
zcorpan Mar 29, 2021
d719338
Remove reference to fixed spec bug
zcorpan Mar 29, 2021
9e746f1
Point to PR for CSP/X-Frame-Options
zcorpan Mar 29, 2021
01b3472
Test coverage for local URL schemes
zcorpan Mar 29, 2021
490e052
I was confused about how agent clusters work. Dedicated workers are (…
zcorpan Mar 29, 2021
db8fd83
Note more existing coverage
zcorpan Mar 29, 2021
cf5c0bb
Link to PR for 'test multiple headers with COEP/COEPRO'
zcorpan Mar 29, 2021
172d653
Note remaining TODOs for COEP reporting
zcorpan Mar 29, 2021
66e0fa8
empty webidl sequence means array with length 0
zcorpan Mar 30, 2021
6e098ab
Link to spec issue for reporting api
zcorpan Mar 30, 2021
dc6cf39
Link to new tests
zcorpan Mar 31, 2021
bba78be
doctype/meta charset done in #28398
zcorpan Apr 8, 2021
d610e5f
Update COOP test plan and point to PR 28653
zcorpan Apr 23, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions html/cross-origin-embedder-policy/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,52 @@
See `../cross-origin-opener-policy/README.md`.

## Reporting

### Existing tests ###

- cache-storage-reporting-dedicated-worker.https.html
- cache-storage-reporting-document.https.html
- cache-storage-reporting-service-worker.https.html
- cache-storage-reporting-shared-worker.https.html
- reporting-navigation.https.html
- reporting-subresource-corp.https.html
- reporting-to-endpoint.https.html
- reporting-to-owner.https.html

### Testing plan ###

- test multiple values with COEPRO
- test "relevant settings object" is the right one
zcorpan marked this conversation as resolved.
Show resolved Hide resolved

https://html.spec.whatwg.org/multipage/origin.html#embedder-policy-checks

"check a navigation response's adherence to its embedder policy"
-> https://html.spec.whatwg.org/multipage/browsing-the-web.html#navigating-across-documents:check-a-navigation-response's-adherence-to-its-embedder-policy
zcorpan marked this conversation as resolved.
Show resolved Hide resolved

-> "Otherwise, if the result of Should navigation response to navigation request of type in target be blocked by Content Security Policy? given navigationParams's request, response, navigationType, and browsingContext is "Blocked", then set failure to true. [CSP]"
zcorpan marked this conversation as resolved.
Show resolved Hide resolved
- test that a CSP failure doesn't give a COEP report.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a step between these two which invokes "check a navigation response's adherence to its embedder policy". Why is that omitted?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to test the ordering of these 3 checks, which should be possible with 2 tests: for steps A, B, C, test these combinations: (A, B) and (B, C)

-> "Otherwise, if the result of checking a navigation response's adherence to `X-Frame-Options` given response, browsingContext, and navigationParams's origin is false, then set failure to true."
- test that X-Frame-Options check happens after COEP check (expect a COEP report) (is this order of checks intentional?)

"check a global object's embedder policy"
-> https://html.spec.whatwg.org/multipage/workers.html#worker-processing-model:check-a-global-object's-embedder-policy
zcorpan marked this conversation as resolved.
Show resolved Hide resolved

Run a Worker and SharedWorker.
-> "If response's url's scheme is a local scheme, then set worker global scope's embedder policy to owner's embedder policy."
- test that "about" (is it testable?), "blob", and "data" inherit owner's embedder policy, and that other schemes don't.

-> "If worker global scope's embedder policy is "require-corp" and is shared is true, then set agent's agent cluster's cross-origin isolation mode to "logical" or "concrete". The one chosen is implementation-defined."
- "is shared" means it's a SharedWorker and not a Worker. So dedicated worker don't get cross-origin isolation mode changed. Test with .crossOriginIsolated ?

-> "If is shared is false and response's url's scheme is "data", then set worker global scope's cross-origin isolated capability to false."
- dedicated worker with data: URL should have .crossOriginIsolated === false.

- test that type (note: different from "report type") is "worker initialization"

- test that blockedURL trims username, password, fragment.
zcorpan marked this conversation as resolved.
Show resolved Hide resolved


https://w3c.github.io/ServiceWorker/

How does this spec use embedder policy? Does it cause COEP reports to be sent?
151 changes: 151 additions & 0 deletions html/cross-origin-opener-policy/reporting/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
## Existing tests ##

- access-reporting
- access-from-coop-page-to-openee_coop-ro.https.html
- COOP reports are to the opener when the opener used COOP-RO+COEP and then it tries to access a same-origin openee.
- access-from-coop-page-to-openee_coop-ro_cross-origin.https.html
- COOP reports are to the opener when the opener used COOP-RO+COEP and then it
tries to access a cross-origin openee.
- access-from-coop-page-to-opener_coop-ro.https.html
- COOP reports are sent when the openee used COOP-RO+COEP and then tries to access its same-origin opener.
- access-from-coop-page-to-opener_coop-ro_cross-origin.https.html
- COOP reports are sent when the openee used COOP-RO+COEP and then tries to access its cross-origin opener.
- access-from-coop-page-to-other_coop-ro.https.html
- One window accesses a second one. They are aren't related by an opener/openee relationship. The first window has set Cross-Origin-Opener-Policy-Report-Only:same-origin, so it receives a "access-from-coop-page-to-other" report.
- access-from-coop-page-to-other_coop-ro_cross-origin.https.html
- One window accesses a second one. They are aren't related by an opener/openee relationship. The first window has set Cross-Origin-Opener-Policy-Report-Only:same-origin, so it receives a "access-from-coop-page-to-other" report.
- access-to-coop-page-from-openee_coop-ro.https.html
- COOP reports are to the opener when the opener used COOP-RO+COEP and then its
same-origin openee tries to access it.
- access-to-coop-page-from-openee_coop-ro_cross-origin.https.html
- COOP reports are to the opener when the opener used COOP-RO+COEP and then its
cross-origin openee tries to access it.
- access-to-coop-page-from-opener_coop-ro.https.html
- COOP reports are sent when the openee used COOP-RO+COEP and then its
same-origin opener tries to access it.
- access-to-coop-page-from-opener_coop-ro_cross-origin.https.html
- COOP reports are sent when the openee used COOP-RO+COEP and then its
cross-origin opener tries to access it.
- access-to-coop-page-from-other_coop-ro.https.html
- One window accesses a second one. They are aren't related by an opener/openee relationship. The second window has set Cross-Origin-Opener-Policy-Report-Only:same-origin, so it receives a "access-to-coop-page-from-other" report.
- access-to-coop-page-from-other_coop-ro_cross-origin.https.html
- One window accesses a second one. They are aren't related by an opener/openee relationship. The second window has set Cross-Origin-Opener-Policy-Report-Only:same-origin, so it receives a "access-to-coop-page-from-other" report.
- property-blur.https.html
- Check openee.blur() access is checked
- property-close.https.html
- Check openee.close() access is checked
- property-closed.https.html
- Check openee.closed access is checked
- property-focus.https.html
- Check openee.focus() access is checked
- property-frames.https.html
- Check openee.frames access is checked
- property-indexed-getter.https.html
- Check reports are sent for the indexed getter
- property-length.https.html
- Check openee.length access is checked
- property-location-get.https.html
- Check openee.location access is checked
- property-location-set.https.html
- Check openee.location access is checked (TODO rename title)
- property-named-getter.https.html
- Check reports are sent for the indexed getter
- property-opener-get.https.html
- Check openee.opener access is checked
- property-opener-set.https.html
- Check openee.opener access is checked (TODO rename title)
- property-postmessage-1.https.html
- Check openee.postMessage(arg1, arg2) access is checked
- property-postmessage-2.https.html
- Check openee.postMessage(arg1) access is checked
- property-self.https.html
- Check openee.self access is checked
- property-top.https.html
- Check openee.top access is checked
- property-window.https.html
- Check openee.window access is checked
- report-to-both_coop-ro.https.html
- Both the openee and the opener have a COOP reporter. The report are sent to both side.
- reporting-observer.
- Check the ReportingObserver(s) are notified about the coop-access-violation events.
- navigation-reporting
- report-only-four-reports.https.html
- A test with both COOP and COOP report only setup
- report-only-from-unsafe-none.https.html
- Report only tests for an opener without any COOP/COOP report only set
- report-only-same-origin-report-to.https.html
- reporting same origin with report-to
- report-only-same-origin-with-coep-report-only.https.html
- reporting same origin with report-to (TODO fix title?)
- report-only-same-origin-with-coep.https.html
- reporting same origin with report-to
- report-only-same-origin.https.html
- reporting same origin with report-to
- reporting-coop-navigated-opener.https.html
- Reports a browsing context group switch when an opener with COOP navigates.
- reporting-coop-navigated-popup.https.html
- Cross-Origin-Opener-Policy: a navigated popup with reporting
- reporting-popup-same-origin-allow-popups-report-to.https.html
- reporting same origin with report-to
- reporting-popup-same-origin-coep-report-to.https.html
- reporting same origin with report-to (TODO fix title?)
- reporting-popup-same-origin-report-to.https.html
- reporting same origin with report-to (TODO fix title?)
- reporting-popup-same-origin.https.html
- reporting same origin
- reporting-popup-unsafe-none-report-to.https.html
- reporting same origin with report-to
- reporting-redirect-with-same-origin-allow-popups.https.html
- Tests the redirect interaction with COOP same-origin-allow-popups.

## Testing plan ##



- access-reporting
- Test all members of Window with testAccessProperty? https://html.spec.whatwg.org/#the-window-object ...or are these enough? https://whatpr.org/html/5518/895fd80...c8265a7/browsers.html#crossoriginproperties-(-o-)
- navigation-reporting
- report-only-same-origin-with-coep.https.html is identical to report-only-same-origin-with-coep-report-only.https.html (but different .headers) -- could have clearer titles.



### HTML ###

https://whatpr.org/html/5518/895fd80...c8265a7/browsers.html#virtual-browsing-context-group-id

- How to test? (TODO) (Also "initial URL", "opener origin at creation")
- window.open
- iframe
- https://whatpr.org/html/5518/895fd80...c8265a7/browsers.html#creating-a-new-auxiliary-browsing-context step 6, 7

https://whatpr.org/html/5518/895fd80...c8265a7/browsers.html#crossoriginproperties-(-o-)
- Property access for these should be tested already.

https://whatpr.org/html/5518/895fd80...c8265a7/browsing-the-web.html#navigate

- step 7...

https://whatpr.org/html/5518/895fd80...c8265a7/origin.html#the-headers

- update Structured Fields parsing tests

https://whatpr.org/html/5518/895fd80...c8265a7/origin.html#browsing-context-group-switches-due-to-cross-origin-opener-policy

- test same COOPRO with same-origin navigation, expect no report
- test initial about:blank (expect no report)
- test non-initial about:blank (expect report)
- test sandboxFlags
- test COOP+COOPRO with different (expect report)
- test COOP+COOPRO with same values (expect no report)

https://whatpr.org/html/5518/895fd80...c8265a7/origin.html#reporting

- test step 2. Nested iframes: originA nests originB nests originA. The innermost iframe is accessor. (expect no report)
- test step 3. Come up with interesting cases where virtual browsing context group ID has unexpected values.
- ...

### General todos/bugs ###

- Add <!doctype html> to all tests
- Add meta charset to all tests
- Make sure titles are unique
46 changes: 46 additions & 0 deletions reporting/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
## Testing plan ##

### Reporting API ###

https://w3c.github.io/reporting/#generic-reporting

- test that the `content-type` request header is application/reports+json
- test that the HTTP method is POST

https://w3c.github.io/reporting/#try-delivery

- check the 'Origin' request header
- spec bug? where does it set the method to POST?
zcorpan marked this conversation as resolved.
Show resolved Hide resolved

https://w3c.github.io/reporting/#queue-report

- change the value of navigator.userAgent, check that the original value is reported
- test that report destination URL's username/password, fragment are stripped
- test a URL that doesn't parse
- test javascript: mailto: data: ws: wss: etc URLs
- test settings objects
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The environment settings object may influence two observable steps in this algorithm: the derivation of the report's URL, and the decision to notify reporting observer. Does this note describe tests for both of those things? Or something else?

Also, there might be another opportunity to improve the spec here. In the algorithm, both "url" and "settings" are optional. Step two assumes that if "url" is not provided, then "settings" will be. Should there be an assertion stating that at least one of those two parameters will always be specified? Or is the algorithm missing a possible condition?


https://w3c.github.io/reporting/#notify-observers

- test report buffer > 100 reports
tests: bufferSize.html

https://w3c.github.io/reporting/#add-report

- test that the right report types are visible or not visible to ReportingObserver
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good if this test plan enumerated the report types that you want to cover because they aren't listed in the specification itself and because that information will probably inform effort estimates.

- test `options` with a `types` member that throws on getting
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm under the impression that the options object will always generated from specification language, which would make this condition impossible. Take that with a grain of salt, though; I haven't reviewed the relevant algorithms in quite some time.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

options is passed to the ReportingObserver() constructor by author JS, so it can be any value. This would be more of a WebIDL test, though. Also, options is a dictionary and types is a sequence<DOMString>, so a types getter that is a function doesn't get to throwing the exception because a function isn't iterable... But these two cases should throw (when the constructor is called, because webidl conversion of the arguments):

new ReportingObserver(() => {}, { types: { *[Symbol.iterator]() { throw 1; } } });
new ReportingObserver(() => {}, { types: [ { valueOf: () => { throw 2; } } ] });

https://heycam.github.io/webidl/#create-sequence-from-iterable

Demo: http://software.hixie.ch/utilities/js/live-dom-viewer/saved/8942

Only the first one throws in chromium and gecko, though. Are they waiting with actually iterating the sequence? Or am I misunderstanding the WebIDL spec?

- test `options` with an empty (?) `types` member. (spec bug? what does empty mean?)
- "queue a task" - which task queue? (spec bug?)
- "how to polymorphically initialize body?" (spec bug?) - help solve this spec issue and test it.
- "Let global be observer’s relevant global object." - test that the right global object is used.

https://w3c.github.io/reporting/#interface-reporting-observer

- Check if this API is well-tested (TODO).
tests: idlharness.any.js
- Test unpaired UTF-16 surrogates where possible.
- Is `Report` web compatible? In chromium it's marked `LegacyNoInterfaceObject` https://source.chromium.org/chromium/chromium/src/+/master:third_party/blink/renderer/core/frame/report.idl (also see spec issue https://github.com/w3c/reporting/issues/216)

https://w3c.github.io/reporting/#disable

- "User agents MUST allow users to disable reporting with some reasonable amount of granularity in order to maintain the priority of constituencies espoused in [HTML-DESIGN-PRINCIPLES]." - manual test