RFC: Add TestDriver support for the "Set Timezone" WebDriver extension command

[FrankYFTang]

Summary

I propose to add TestDriver support for the "Set Timezone" WebDriver command, which will allow the web platform tests to set the host timezone in the user-agent. This command is specced in here.

Motivation

In order to test how web apps behave in different time zones, developers currently have to do the following:

• change the operating system-level settings (e.g. by setting the TZ environment variable)
• restart the browser
• test the web app
• restore the operating system-level setting

This is an unnecessarily painful and inefficient workflow, which we aim to simplify by adding first-class timezone emulation support to user-agent via WebDriver extension command. The command empower the web platform tests to inform the user-agent to switch to a different host timezone during the tests.

Risks

I am not aware of any risks.

Details

No additional details.

[jgraham]

RFCs are usually pull requests.

In this case I have concerns about implmenting something where the "spec" is a Google doc. I think this would need to go through a proper standards process before we could consider it for testdriver.

[stephenmcgruer]

Apologies @jgraham (and to Frank), that's on me. Frank is following a guide I own, which asks for an RFC to be filed and points to an old RFC (here) which I guess is from before we changed to pull requests?

However, I'm not actually clear that we need to make an RFC part of the process. It was done for generate test report, but not for permissions nor webauthn I don't think. I suggest we accept the existence of a merged PR in a standards document (not a google doc; separately I'm working with Frank on that) as sufficient cause to accept a pull request to WPT without an RFC?

Reasoning:

1. We have a goal to reduce the pain of webdriver automation, and this seems like an easy step to remove.
2. Testdriver is written to be quite extensible and so lack of support for a given webdriver feature in a given vendor doesn't have serious consequences. (I think?)

[stephenmcgruer]

Indeed, https://web-platform-tests.org/writing-tests/testdriver-extension-tutorial.html doesn't mention writing an RFC, though as I understand it you weren't familiar with that guide until recently @jgraham so quite possibly you don't consider it authoritative either :)

[jgraham]

I understand the desire to make this easy for developers. But, especially in cases like this where the feature might make sense independent of any other web platform feature, I wonder what's needed to communicate the new requirement to other parties and ensure that the design is implementable across multiple engines.

In particular one can imagine something that goes through a pseudo-standards venue like WICG without getting significant feedback from multiple parties. Then it's added to testdriver without any input, and later other vendors are surprised by failing tests which turn out to because of a feature that they had no idea exists and which they are unable to implement in a reasonable way.

For things that are part of a spec with real multi-party buy in like webauthn, the concern about the spec not having sufficient review is less relevant, but there is still a problem with the communication. Currently the process for figuring out that the endpoints were added to testdriver still seems to be "notice that the tests are failing". And from the point of view of a second implementor that actually makes things worse, because it means the webauthn team are the most likely to notice the failures, and they then have to reach out to the test infra team to figure out what's going on in an area outside their expertise. Whereas if there's an RFC th einfra team can proactively communicate with relevant implementation team and ensure that they prioritise the work to implement the endpoint.

cc @web-platform-tests/wpt-core-team for more feedback on the policy piece here.

[stephenmcgruer]

(cc @foolip )

With regards to 'are webdriver extensions in WICG actually reasonable standards?', I think that's a fair point.

It seems awkward to have WPT be the communication centre-point, though. This would mean any RFC to add a testdriver extension now needs all major vendors to go talk to their implementation team(s) and wait for their response to confirm that the extension is reasonable, which seems like it would add a lot of slowdown.

Do we really need webdriver extensions to stand up to the same standard (hah, pun not intended) as general web features? I understand that WebDriver is visible to and used by web developers for testing and so they can come to rely on it. However perhaps it could accept more churn than 'proper' web APIs and thus have a lower bar for initial implementations?

Alternate idea: maybe WebDriver extensions by default should be marked 'experimental' in some way (perhaps even requiring you pass a capability to WebDriver to enable them?), and then go through a separate vetting process before they are considered 'stable'? That way we could lower the bar for WPT and reduce the friction, but be clear to web developers that these might change (e.g. if other vendors say "we can't implement that").

[FrankYFTang]

In this case I have concerns about implmenting something where the "spec" is a Google doc. I think this would need to go through a proper standards process before we could consider it for testdriver.

OK, I also file

w3c/webdriver#1559

[FrankYFTang]

Could someone explain to me what is WICG ?

[FrankYFTang]

@jgraham Since the value of the Timezone ID fired by this SetTimezone WebDriver extension could only be observed by the calls defined in ECMA402. Would you recommend me to talk to ECMA402 or webdriver to incorporate the Google Doc I put together into their standard?

[FrankYFTang]

@anba @sffc @littledan @gibson042 @zbraniecki @rwaldron @spectranaut @rkirsling @mathiasbynens - For the impact to testing about ECMA402 features

[stephenmcgruer]

Could someone explain to me what is WICG ?

The Web Platform Incubator Community Group, which is a incubator for web specs that are not yet widely accepted by multiple browser vendors. Here jgraham is using it to highlight the risk that people propose a bunch of WebDriver extensions that end up being very browser specific and not interoperable.

To be clear, much of the above conversation likely does not apply to the proposed timezone command. I believe this command to be fairly low risk, and once you have found an appropriate place for it to be specified I will help you get an RFC pull request done for this. The discussion jgraham and I are having here is on the meta level of what we should do in general for new webdriver extensions, not this specific one.

[FrankYFTang]

once you have found an appropriate place for it to be specified

Now, what is the minimum criteria of such " appropriate place "? By default, my prefer place is the current Google Doc. If that is not acceptable for web-platform-tests to accept, then web-platform-tests need to define what is the minimum criteria location for such and I can work toward that.

1. Is it just need to be a document in public that everyone accessible? (then my current google doc fit, right?)
2. Is it also need to be agreed by developers from at least 2 browser engines? (would me and an engineer from a random browser project agree good enough?)
3. Is it need to be agreed by developers from at least 3 browser engine?
4. Is it also need to be PART of a "standard" ? (then could someone list the set of standard which is acceptable?)
5. Is it need to be part of W3C and not just ECMA TC39?
6. Is it ok to be either part of W3C or ECMA TC39?
   I need some clear criteria to define the minimum criteria of such " appropriate place " so I can move forward to it. I am willing to move toward those places, but there are no way I can guess what is in the mind of decision holders in web-platform-tests now.

[stephenmcgruer]

Now, what is the minimum criteria of such " appropriate place "? By default, my prefer place is the current Google Doc. If that is not acceptable for web-platform-tests to accept, then web-platform-tests need to define what is the minimum criteria location for such and I can work toward that.

Apologies, I should have been more explicit. What I meant by 'appropriate' was that it is either a submitted PR or an LGTM'd PR to a web specification. As per our internal thread I (and I believe @domenic ) felt that HTML was a good choice given you were adding the timezonechange event to that spec, but you have since argued for both ECMA402 and the WebDriver spec itself. Any of those 3 is fine by me.

jgraham has been clear, and I agree, that a google doc (or other public document) does not suffice.

EDIT: Edited to clarify that any of the HTML spec, ECMA402, or WebDriver seems a fine location for the extension command to be specified for me.

[FrankYFTang]

OK, I also file issue in ECMA402 and placed on the Monthly Meeting Agenda of this coming Thursday to discuss would ECMA402 wiling to include a section about this. How does other features got added before?

[jgraham]

It seems awkward to have WPT be the communication centre-point, though. This would mean any RFC to add a testdriver extension now needs all major vendors to go talk to their implementation team(s) and wait for their response to confirm that the extension is reasonable, which seems like it would add a lot of slowdown.

Hopefully the relevant team has already been involved in the standards process and has already agreed that the proposal is reasonable. The additional piece of information is "this will be used in wpt tests, and therefore we need to implement it to get coverage of those tests". That may seem obvious from the point of view of people working on test infra, but it's quite possible for product teams to not make the connection between WebDriver extensions and web-platform-tests exposed APIs.

[stephenmcgruer]

How does other features got added before?

This question is ambiguous. I am interpreting it as "What specs were other webdriver extensions added to?", but let me know if that is not what you were asking. For my interpretation:

• Permissions spec (initial PR that added it)
• Reporting spec (initial PR that added it)
• Generic Sensor API spec (initial PR that added it)
• WebAuthn spec (initial PR that added it)
• Background Fetch spec (initial PR that added it)

[FrankYFTang]

@littledan - would you mind to add an "Automation" section similar to above into your whatwg/html#3047 ?
My draft of such is in https://docs.google.com/document/d/12vdPTH_Vlkt4pxwlWzR1CjKtLE6y2Z-Qs-GPi_hWXqc/edit#

[FrankYFTang]

ECMA402 subcommittee talk about the concept of adding "Set Timezone'" WebDriver extension to ECMA402 today. The conclusion is we do not think ECMA402 should host such specification since ECMA402 should not be the source of truth of system timezone.
That narrow down the possible places for this spec to

1. WebDriver
2. HTML

[stephenmcgruer]

I have opened #73 as an RFC for this extension now that it is in an (in progress) PR on the HTML spec.

I am closing this in favour of the PR, but long term I do expect to come back to the discussions jgraham and I had above about reasonable expectations around specs/etc for WebDriver support.