Add libtest JSON emitter

[kirtchev-adacore]

As discussed in the #t-infra/bootstrap > x test src/tools/clippy and JSON output Zulip thread, oli suggested:

Adding json support for ui_test shouldn't be too hard (famous last words). I wrote the entire system emitter-agnostic. So you "just" need to implement the various emitter traits for a new json emitter struct. If you want to upstream it to ui_ test I'll happily maintain it, so there's only a one time cost for you (implementing, adding tests and likely addressing a round or two of review feedback)

For the record, I looked into creating a ui_test-to-libtest adapter, where implementations of the various emitter traits would convert the ui_test types to libtest types and just use the libtest JSON emission circuitry, however libtest does not expose most of the relevant traits and types in its public API.

[oli-obk]

yea we'll need to add the types ourselves to ui_test. Unless cargo_metadata has them already, I'm not sure how much stuff we added to cargo_metadata

[Zalathar]

For comparison: When implementing libtest-compatible JSON output in compiletest (rust-lang/rust#139660), I ended up writing the JSON emitters by hand, because I found the equivalent serde declarations to be much harder to write and read.

[oli-obk]

Yea, I don't think a serde style serialization is possible here in general the way the emitter traits work. The outer layers will need to be custom, or you need to store everything in one big data structure and serialize at the end

[kirtchev-adacore]

I have been looking at the status_emitter traits and the implementation in text since it will be the closest implementation to the libtest JSON, and I am scratching my head a bit.

1. Please check my understanding of how ui_test works with respect to emitters:
   1. ui_test calls StatusEmitter::register_test() for each discovered test, getting back a TestStatus.
   2. The mapping between a test and its TestStatus is maintained by ui_test.
   3. Once the test completes, ui_test calls TestStatus::done() for the corresponding TestStatus.
   4. When is TestStatus::failed_test() called with respect to TestStatus::done()?
   5. What is this "drop guard" mentioned in the comment of TestStatus::failed_test()? What does it do?
   6. Once all tests complete, ui_test calls StatusEmitter::finalize() to get back a Summary.
   7. ui_test calls Summary::test_failure() if I assume there was at least one failed test. What happens when there are no failures? How is the Summary emitted?
   8. What is the role of TestStatus::for_path() and TestStatus::for_revision()? Why would ui_test need a copy of a TestStatus with some modification?
   9. What is the role of TestStatus::path() given that ui_test already has the mapping between a test and its TestStatus?
   10. What is the role of TestStatus::revision() given that there is no way to specify the revision in the first place?
2. text uses the feature "indicatif". What is this, and should I care about it?
3. text seems to be employing tasking to deal with multiple progress bars. Should I care about this?

Probably more questions to come.

[oli-obk]

8. What is the role of TestStatus::for_path() and TestStatus::for_revision()? Why would ui_test need a copy of a TestStatus with some modification?

Revisions and subtests like rustfix tests need a separate test. I don't think libtest has a concept of nested tests, so you'll still need to do this but just treat the subtest as its own test

[oli-obk]

2. uses the feature "indicatif". What is this, and should I care about it?

You shouldn't. That's just a terminal redrawing crate so I can have spinners and progress bars

[oli-obk]

3. seems to be employing tasking to deal with multiple progress bars. Should I care about this?

Nope

[oli-obk]

For the rest, maybe add docs to the trait when you figure them out

[oli-obk]

I think https://github.com/oli-obk/ui_test/blob/main/src/status_emitter/debug.rs is a better starting point for you

[kirtchev-adacore]

Here is the libtest JSON format in a nutshell:

/// The format of the libtest JSON is as follows:
///
///     { "type": "suite", "event": "started", "test_count": TEST_COUNT }
///     { "type": "test", "event": "started", "name": "NAME" }
///     { "type": "test", "name": "NAME", "event": "TEST_STATUS" }
///     { "type": "test", "name": "NAME", "event": "failed", "stdout": "OUTPUT" }
///     { "type": "suite", "event": "TEST_SUITE_STATUS", "passed": PASSED, "failed": FAILED, "ignored": IGNORED, "measured": MEASURED, "filtered_out": FILTERED, "exec_time": TIME }
///
/// Where
///     FAILED - The number of failed tests in the test suite.
///     FILTERED - The number of filgered tests in the test suite.
///     IGNORED - The number of ignored tests in the test suite.
///     MEASURED - The number of measured tests in the test suite.
///     NAME - The name of the test.
///     OUTPUT - The output of a failed test test.
///     PASSED - The number of passed tests in the test suite.
///     TEST_COUNT - The number of total tests in the test suite.
///     TEST_STATUS - Either "ignored", "ok", or "timeout".
///     TEST_SUITE_STATUS - Either "failed" or "ok".
///     TIME - The time (in seconds) it took to execute the test suite.

As you can see, each test suite commences with a started event, which among other things contains the number of tests to be executed.

Since StatusEmitter does not have an appropriate entry point for such an event, I will have to delay the output of everything until the Summary is finalized and emitted.

Which brings me to one of my previous questions:

ui_test calls Summary::test_failure() if I assume there was at least one failed test. What happens when there are no failures? How is the Summary emitted?

[oli-obk]

Do you need a realiable test count? Can't you just ignore that value? Like... this format is unstable and I don't think the test count is a useful value in the initial suite start. I would just not emit the test_count field.

ui_test calls Summary::test_failure() if I assume there was at least one failed test. What happens when there are no failures? How is the Summary emitted?

I would have to look at the source, please do so for me and document that method on the trait 😆

[kirtchev-adacore]

Do you need a realiable test count?

Unfortunately we do need this value. We use it to sanity check a test suite run, in particular detect whether the number of executed tests agrees with our oracle. Here is a good example from our infrastructure system:

+ Different number of executed tests for alloc
+   expected: 2245
+   actual  : 2795
+ Different number of executed tests for cargo
+   expected: 3089
+   actual  : 3430
+ Different number of executed tests for clippy
+   expected: 1043
+   actual  : 1520
+ Different number of executed tests for compiler
+   expected: 2227
+   actual  : 2500
+ Different number of executed tests for compiletest
+   expected: 18325
+   actual  : 20633
+ Different number of executed tests for core
+   expected: 6630
+   actual  : 7275
+ Different number of executed tests for rust_analyzer
+   expected: 15
+   actual  : 0
+ Different number of executed tests for rustfmt
+   expected: 201
+   actual  : 212
+ Different number of executed tests for std
+   expected: 2120
+   actual  : 2413
+ Test suite rust_analyzer_extension was not executed