Unify ./wpt check-stability and ./wpt run --verify

[foolip]

Previous issues and PRs (now closed/merged) on the topic of stability checking:

• Move stability checking to wpt run --verify
• Replace check-stability script with wpt run --stability and document how to run locally
• Make --stability just use the --verify code
• stability/verify unification

However, with --stability gone, we're still left with two different things. On Travis we use ./wpt check-stability:

wpt/tools/ci/ci_stability.sh

Line 12 in 654dc43

./wpt check-stability $PRODUCT $extra_arg --output-bytes $((1024 * 1024 * 3)) --metadata ~/meta/ --install-fonts

And on Taskcluster we use ./wpt run --verify:

wpt/.taskcluster.yml

Line 100 in 207fa91

extra_args: '--verify'

It would be great with just a single approach that we can use everywhere.

@gsnedders @jgraham @jugglinmike

[jugglinmike]

This is more than just interface design. The semantics of the two commandsdiffer in a way that influences what passes for "stable". More on this here:

#12657 (comment)

[foolip]

@jugglinmike thanks for that summary! Beyond the AABB vs. ABAB ordering, you can from the Taskcluster logs that there are also differences in restarts. Example from #13405:

:::
::: Running test verification step "Running tests in a loop 10 times"...
:::

All results

/webstorage/event_initstorageevent.html

Subtest	Results	Messages
	OK
initStorageEvent with 0 arguments	FAIL	assert_throws: function "() => event.initStorageEvent()" did not throw
initStorageEvent with 1 argument	FAIL	assert_equals: event.key expected (object) null but got (string) "undefined"
initStorageEvent with 8 sensible arguments	PASS
initStorageEvent with 8 null arguments	FAIL	assert_equals: event.key expected (object) null but got (string) "null"
initStorageEvent with 8 undefined arguments	FAIL	assert_equals: event.key expected (object) null but got (string) "undefined"

:::
::: Running test verification step "Running tests in a loop with restarts 5 times"...
:::

All results

/webstorage/event_initstorageevent.html

Subtest	Results	Messages
	OK
initStorageEvent with 0 arguments	FAIL	assert_throws: function "() => event.initStorageEvent()" did not throw
initStorageEvent with 1 argument	FAIL	assert_equals: event.key expected (object) null but got (string) "undefined"
initStorageEvent with 8 sensible arguments	PASS
initStorageEvent with 8 null arguments	FAIL	assert_equals: event.key expected (object) null but got (string) "null"
initStorageEvent with 8 undefined arguments	FAIL	assert_equals: event.key expected (object) null but got (string) "undefined"

::: Running tests in a loop 10 times : PASS
::: Running tests in a loop with restarts 5 times : PASS
:::
::: Test verification PASS
:::

Anyway, it seems to me that --verify is the more featureful variant. @jgraham, after we've turned off Travis stability checking in favor of Taskcluster, is there any reason to keep check-stability around as well?

[jgraham]

No.

[foolip]

Alright! @jugglinmike, how are things looking for making Taskcluster stability task failures blocking?

[mdittmer]

Ping from your friendly neighbourhood ecosystem infra rotation

If this is priority:roadmap should it have an assignee?

[foolip]

I'll downgrade the priority. Once we've gotten rid of stability jobs from Travis entirely we can do this.