feat: add nargo test --format json

[asterite]

Description

Problem

Part of #6733

Summary

Adds --format json to nargo test.

Here's the output for a single suite and test that passes:

{"event":"started","name":"one","test_count":1,"type":"suite"}
{"event":"ok","exec_time":0.049250625,"name":"one","type":"test"}
{"event":"ok","failed":0,"passed":1,"skipped":0,"type":"suite"}

Here's one when there's a failure:

{"event":"started","name":"one","test_count":1,"type":"suite"}
{"event":"failed","exec_time":0.435959917,"name":"keys::getters::test::test_get_public_keys_unknown","stdout":"error: Test failed with the wrong message. \nExpected: Invalid public keys hint for address \nGot: 0 output values were provided as a foreign call result for 13 destination slots\nAssertion failed: '0 output values were provided as a foreign call result for 13 destination slots'\nat aztec/src/test/helpers/cheatcodes.nr\nCall stack:\n1. aztec/src/keys/getters/test.nr:17:19\n2. aztec/src/keys/getters/test.nr:10:19\n3. aztec/src/test/helpers/cheatcodes.nr:51:5\n","type":"test"}
{"event":"failed","failed":1,"passed":0,"skipped":0,"type":"suite"}

Additional Context

Documentation

Check one:

• No documentation needed.
• Documentation included in this PR.
• [For Experimental Features] Documentation to be submitted in a separate PR.

PR Checklist

• I have tested the changes locally.
• I have formatted the changes with Prettier and/or cargo fmt on default settings.

[github-actions]

Peak Memory Sample

Program	Peak Memory	%
keccak256	79.14M	0%
workspace	121.94M	-1%
regression_4709	295.99M	0%
ram_blowup_regression	2.44G	0%
private-kernel-tail	210.43M	0%
private-kernel-reset	862.22M	0%
private-kernel-inner	307.94M	0%
parity-root	175.78M	0%

[github-actions]

Compilation Sample

Program	Compilation Time	%
sha256_regression	0m1.485s	-13%
regression_4709	0m0.792s	-2%
ram_blowup_regression	0m17.012s	-1%
private-kernel-tail	0m1.209s	-21%
private-kernel-reset	0m9.660s	10%
private-kernel-inner	0m2.471s	-9%
parity-root	0m1.147s	22%
noir-contracts	2m50.039s	0%

[michaeljklein]

LGTM

[TomAFrench]

Just comparing this to the libtest output (would be nice to be compatible)

[TomAFrench]

Differences I see are:

• The name key for suites doesn't exist in libtest.
• We include an exec_time key for test completion.
• We don't print a start event for individual tests.

Can you add this start event? The other two fields should be fine to keep (exec_time in particular is useful).

[asterite]

One difference I remember with libtest is that they show the elapsed time (exec_time) of the entire suite but not for individual tests. Here I chose to show it for individual tests and not for the suite, because:

1. that's what we had before this PR
2. I think it's more useful, to know which tests are slow
3. because tests run in parallel, I'm not sure how useful it is for the entire suite

Another difference is that because we use "skipped" instead of "ignored" I use that word... but we could change it.

[asterite]

I'll add the start event 👍 I wasn't sure how useful it is, or if there are tools that already use the JSON output from libtest (the other formatters don't "use" the start event)

[TomAFrench]

Yeah, let's change "skipped" to "ignored".

[TomAFrench]

🤷 It's cheap to do and improves compatibility so might as well.

I wasn't sure how useful it is, or if there are tools that already use the JSON output from libtest (the other formatters don't "use" the start event)

If nargo test hangs we'll be able to tell which test caused it (or at least narrow down to a couple).

[asterite]

I'm going to be doing a few changes, then.

The thing is, Rust seems to parallelize tests by package. In our case we parallelize all tests in all packages.

So, in Rust the events are like this

• start suite one
• start test a
• start test b
• end test a
• end test b
• end suite one
• start suite two
• start test c
• etc.

In our case the events are like this:

• start suite one
• start suite two
• start test a
• start test b
• start test c
• (ends can happen in any order)
• end suite one
• end suite two

However, the output we show to users is this:

[one] Running 2 test functions
[one] Testing a.. ok
[one] Testing b... ok
[one] 2 tests passed
[two] Running 1 test function
[two] Testing c... ok
[two] 1 test passed

So you can see that for output for humans we go package by package, packages are not mixed. But in the case of events, they will be mixed.

So I think I'll just have two types of formatter callbacks, "sync" and "async", where the "sync" ones are used to show output for humans (pretty and terse), and the "async" ones are for output for computers (just the JSON format).

Alternatively, we could just fake one "test_start" event right before the "test_end" event, but then we won't be able to find out if a test hangs.

[asterite]

Done! If there are tools that use the libtest output and rely on the specific order they use we could later adjust the code on our side.

[asterite]

@TomAFrench Let me know if the output looks fine now.

[asterite]

I think I'll undo the last commit in this PR. I wanted to start working on the filtering so the first step was running external checks, but I'll do that in a separate PR.