Move stability checking to wpt run --verify

[jgraham]

Since we developed the stability checker, wptrunner got a --verify flag for checking test stability. Ideally instead of having separate codepaths here we could keep all the travis handling logic from the check_stability script but do the actual run using the --verify option to wpt run, so that we are confident that the same CI checks apply to both.

[akshitac8]

Hey @jgraham . I would like to work on this issue. Can I get a little insight on how to proceed.

[kriti21]

@akshitac8 Are you still working on this one ?

[jgraham]

Sorry, I somehow missed the comments here. I think this isn't a trivial issue and probably not suitable as a first bug because I currently don't know what's required to land this.

[akshitac8]

yes @kriti21, Its taking a little time. sorry for the inconvience:)

[akshitac8]

Hello,
I wanted to confirm that this file has to be modified?

[foolip]

@jgraham, maybe #10269 is now a dupe of this? Is it --verify or --stability?

[foolip]

Hmm, both --verify and --stability exist. There's presumably some difference. @kereliuk, can you take a look, and assign to yourself if this makes sense as part of fixing #7660?

[gsnedders]

@jgraham

We have both:

  --stability           Stability check tests
  --verify              Run a stability check on the selected tests

What's the difference between these?

[jgraham]

--stability does exactly what travis does right now (except the ci command has extra integration with Travis) and --verify is the new thing that we should use instead.

[gsnedders]

@jgraham do we want to support both? should we drop support for one? we should at least document the difference somewhere, because at the moment the --help doesn't give any indication as to which to use!

[jgraham]

The intent of this issue is to move to using --verify exclusively.

[foolip]

There's a great deal of overlap with #10269 and perhaps they'll be fixed by the same PR, but for now I'll just align the labels and assign both to @gsnedders.

[gsnedders]

OK, so in short, the current flow is:

1. tools/ci/ci_stability.sh does little more than start tools/ci/check_stability.py
2. tools/ci/check_stability.py sets stuff up, calls tools.wpt.stability.run, aggregates results, and posts to pulls.wpt.org
3. tools/wpt/stability.py is the old thing we want to replace here (and in Make --stability just use the --verify code #10921)

The hardest part is that we need to get the data structure that tools.wpt.stability.run returns somehow out of ./wpt run --verify (and I'm not entirely sure what that data structure is yet!), though I think that's probably just copy/pasting code?

[foolip]

@gsnedders, could ./wpt run --verify not simply write a wptreport? If that format doesn't currently support recording multiple runs, how could we make it do that?

@lukebjerring @Hexcles, is that what we'd want for web-platform-tests/wpt.fyi#118?

[gsnedders]

@foolip Without having tested, I would assume it would support any of wptrunner's output formats, given it's part of wptrunner? That said, pulls.web-platform-tests.org expects to get data POSTed in a specific format, so somewhere we need to convert the data into that form (or kill the dashboard entirely; see #10923) before making the request there.

[foolip]

If, as a part of the work on this issue, posting to pulls.web-platform-tests.org no longer happens (because it would require additional work) I think that'd be fine, since it's no longer commenting, and when it was it was incredible hard to figure out if there were regressions or not. If we want to revive it before solving the same problem with wpt.fyi, lumping that into the same work seems OK. @jugglinmike, does that sound reasonable, or too willing to break things?

[jugglinmike]

I have yet to contribute to the pull request dashboard, but out of coincidence (i.e. our effort to centralize secrets), I'm almost in a position to deploy changes to that application. My preference would be to diagnose and correct the existing problem before making further changes since proceeding in the reverse order (or consolidating two steps) would make bug fixing more difficult.

when it was it was incredible hard to figure out if there were regressions or not

Do you mean "regressions [in the WPT pull request being validated]" or "regressions [in the integration between WPT and the pull request dashboard"?

[foolip]

I mean "regressions [in the WPT pull request being validated]".

[gsnedders]

I'm pretty sure the immediate state after we fix this isn't going to stop posting results, FWIW.

[jugglinmike]

Got it, @foolip. I was preparing to file a new issue to discuss simplifications to debugging the integration, but since you were referring to the test review process, I think that's documented by gh-7475

[Hexcles]

@foolip I don't know how multiple runs will interact with wptreprot. Even if they end up in the report, I don't think various readers on the wpt.fyi project can understand multiple runs of a same test.

However, the whole code path shouldn't be too hard to fix. And using wptreport is the right way to go IMHO.

[foolip]

@Hexcles, I suppose the options are N wptreport JSON files, or 1 wptreport JSON files representing all runs. I take it you prefer separate files? But how would one represent a run with AABB test order? (I'm guessing that's possible to achieve with some combination of flags.)

[gsnedders]

So what I haven't done so far is move over to running the multi-part verify (with/without restarts, with/without chaos mode in Firefox), but it does now use the verify code.

At the moment, it's still calling into the code through Python, rather than letting it run separately and then reading the logs. If we want to avoid going through the logs, we'd need to either make check_stability (in tools/wptrunner/wptrunner/stability.py) return something useful or call get_steps ourselves.