Skipping based on runtime detection support

[mathstuf]

Long ago I posted a pre-RFC thread about skipping tests at runtime. I'd like to make sure that some kind of support ends up here and wanted to participate with it.

I do have patches against an old rustc to support the proposed attribute, but they surely need rebased by this time.

[epage]

We support this internally and at runtime but we should keep track of this for when we get to writing the macros.

This is blocked on #16.

[epage]

First off, I haven't gone through all of the thread yet.

First, a question for us to resolve is what we mean by "ignore" and "skip". Within libtest today, the macro is ignore and it gets reported as such. Considerations for defining things

• libtest a --include-ignored flag, implying that ignores can be overridden
• What do we expect people to do based off of reporting?
  • e.g. how do people make sure a test runs
  • e.g. how do logging systems not get confused at the result

One idea I've been playing with is:

• Skip: can't run. A common occurrence would be that the fixture failed. Platform conditionals would also fall into this camp.
• Ignore: disabled by default, may be run with --include-ignored

In both cases, we should actively encourage people to give a reason for something being ignored/skipped. Granted, we can infer this from the predicate if nothing else.

As for how to express this, I think the macro could easily handle #[ignore = pred()] though some might like the more explicit #[ignore_if = pred()]. pred() could return either bool or Option<impl Display> where Some is the reason.

Under the above, we could have #[skip_cfg(windows)] (hmm, the negative logic might confuse people)

[mathstuf]

Note that I'm mostly interested in runtime skipping. #[ignore] and #[cfg] are fine for statically-known details like platform. Instead, this is for things like "kernel is too old", "kernel is missing some critical feature", "you're running Wayland, not X", "you seem to lack a sufficiently new CUDA driver/hardware/configuration", or "the test is running as root" that is not tied to any ABI or platform detail representable as a cfg query. It's not something that a flag to cargo test should be able to affect through some "run skipped tests anyways" flag either as the test is likely to give either bogus results or panic.

Note that there should also be support for test executables to be able to perform this skipping logic.