Selectively testing commits to master

[jugglinmike]

Today, we are running all of WPT in experimental releases of Chrome and Firefox for every commit to master via the TaskCluster service. (We hope to gather up the results from these tasks for publication on https://wpt.fyi).

Moving forward, we would like to also run all of WPT in stable releases of those two browsers. This does not need to occur for every commit to master; only commits designated by an external "announcer" system need to trigger these jobs.

[jugglinmike]

From the TaskCluster documentation

The core of Taskcluster-Github's operation is this: when an event occurs on Github, such as a push or a pull request, it loads .taskcluster.yml from the commit specified in the event, renders it with JSON-e, and then calls Queue.createTask for each of the specified tasks.

In order to trigger TaskCluster, we will need to perform some git operation in the GitHub-hosted repository. Tagging the desired commit seems like the most appropriate operation. Which part of the system ought to be responsible for that?

• The revision announcer could push tags whenever it identifies an "interesting" commit
• TravisCI (acting via a cron job) could query the revision announcer on a regular interval and push tags as necessary

There are trade-offs here, so I'm wondering what @jgraham and @mdittmer think.

[foolip]

Presumably merely tagging won't trigger the Taskcluster integration, but we could perhaps have a branch that always points to something the revision announcer has picked, updated daily.

Or, have a faux repo/branch which only exists to trigger Taskcluster and doesn't share any history with the WPT repo. But defining the tasks in the wpt repo itself probably becomes problematic then, one would have to emulate the processing of .taskcluster.yml perfectly.

[jugglinmike]

Presumably merely tagging won't trigger the Taskcluster integration

It ought to, in theory. The GitHub docs on the "push" event state that it is fired for tags, and TaskCluster's GitHub integration has code for that case.

I've filed an issue to see if this use case is supported, but your suggestion for an "announcement branch" is a more common scenario and more likely to be supported.

[jgraham]

Yeah, I think a tag or a branch would both work. If the revision announcer can just create an approproate ref of some sort, that seems like a perfect system; it will integrate with the current taskcluster scheduling with few changes.

If we wanted to do this earlier than that is ready, we could create an explicit decision task (this will eventually be requried for PRs anyway) and just have it check if e.g. the current PR is the first merged PR since midnight UTC using the GH API.

[foolip]

This was done in #12679.

[foolip]

I've filed #12907 about the branch maintenance.