Miri UI tests aren't run on CI (or locally)

[CraftSpider]

Issue

As the title says - Miri UI tests aren't run when invoked through x.py, or on CI:
https://github.com/rust-lang-ci/rust/actions/runs/4636594388/jobs/8204646214#step:26:13316

The actual issue appears to be that compiletest.rs in the miri directory expects no non-filter args to be passed to it when being run, but the args -Z unstable-options --format json are getting passed.

Discovery

I was trying to test miri locally, and I ran the command x.py test src/tools/miri --stage 2. I expected the miri UI tests to get run, but instead, they were all always getting filtered out, despite not passing a filter. After looking into it a bit, I discovered the filter vector was always ["-Z", "unstable-options", "--format", "json"], and decided to check whether bors was running them.

[jyn514]

This seems ... pretty unfortunate 😓

cc @rust-lang/release @rust-lang/miri, I know miri is nightly only but it makes me pretty uncomfortable to be shipping tools that aren't running tests, even if rust-lang/miri is still running the tests.

[saethlin]

If my shell history is anything to go by, this is recent breakage, and x test miri used to run all of Miri's tests.

[oli-obk]

Yea, this is not breakage on the miri side afaict, but must be a change on the cargo or bootstrap side.

I'm fine fixing this on our side tho. I can replicate the -- behaviour of the default test runner and adjust the wrappers in the miri repo to insert that

[CraftSpider]

For making it follow normal -- behavior, I'm just replacing the skip(1).filter(...) in compiletest with skip_while(|arg| arg != "--") for now. I'd be willing to PR this change as a standalone if desired.

[oli-obk]

For making it follow normal -- behavior, I'm just replacing the skip(1).filter(...) in compiletest with skip_while(|arg| arg != "--") for now. I'd be willing to PR this change as a standalone if desired.

Thanks! That would be great

[RalfJung]

Yeah I am pretty sure I tested this when we changed Miri to a subtree, and it worked back then.

How does looking for -- help? Looks to me like we'd have to know exactly which flags the regular test runner understands and parse them all the same way. E.g. -Zx filter should apply the filer but -Z filter should not. Which is of course ridiculous.

Can we stop bootstrap from passing wrong flags to test runners?

[RalfJung]

I suggest that we change the test runner argument parsing to error out if there is any filter starting with - -- except after a --. So, something like:

// Arguments before the first '--' are filters, except if they start with `-` which are probably
// meant to be flags, and we don't support any flags.
for arg in args.as_ref().take_while(|arg| arg != "--") {
  if arg.starts_with("-") {
    eprintln!("Unknown command-line flag {arg}");
  }
  filters.push(arg);
}
// Append remaining arguments as filters
filters.extend(args);

[RalfJung]

The source of the regression is #108659.

Meanwhile #110177 should fix bootstrap invoking the Miri test suite.