T2. add(test): add test API that checks process logs for failures

[teor2345]

Motivation

We want to check lightwalletd logs and Zebra logs for failures like:

• missing RPCs
• RPC argument parse errors
• RPC argument data errors
• RPC response parse errors
• RPC response data errors
• other specific errors logged by lightwalletd

But our testing framework makes it very hard to do that.

This PR adds an API to check test logs for failures, but it doesn't actually do those checks yet.

This is related to:

• Detect warnings, errors and panics in full sync test #3500
• Print lightwalletd output during integration test #3856
• Add lightwalletd full sync integration test #3511

Designs

Add a set of failure regexes to each child process, and check each log line for those regexes. Panic if any failures are logged.

Make sure test processes are cleaned up correctly, so their logs are read and checked for failures.

Solution

Failure logs:

• Add a set of failure regexes to test child processes, and use them to check process output
• When the child is dropped, check any remaining output
• Add internal tests for failure regex panics

Reading process output:

• Add test APIs for consuming child output lines, without checking them for a success regex

Also document some edge cases, and some things we might want to fix later.

Review

This PR is not urgent.

It is based on PR #3892.

Reviewer Checklist

• Code implements Specs and Designs
• Tests for Expected Behaviour
• Tests for Errors

Follow Up Work

• Check lightwalletd logs for failures

[oxarbitrage]

Please rebase and fix conflicts when you get the time. Thanks!

[teor2345]

@Mergifyio update

[mergify]

update

❌ Base branch update has failed

merge conflict between base and head
err-code: 94415

[teor2345]

@Mergifyio rebase

[mergify]

rebase

❌ Base branch update has failed

Git reported the following error:

Rebasing (1/23)
Auto-merging zebra-test/src/command.rs
CONFLICT (content): Merge conflict in zebra-test/src/command.rs
Auto-merging zebra-test/src/command/to_regex.rs
CONFLICT (add/add): Merge conflict in zebra-test/src/command/to_regex.rs
error: could not apply 5ce0dee5... Make command test matching code accept generic regexes
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply 5ce0dee5... Make command test matching code accept generic regexes

err-code: A5E44

[teor2345]

Please rebase and fix conflicts when you get the time. Thanks!

Done!

The conflicts aren't real, they're just caused by squashing PRs.

[oxarbitrage]

Looks great, very complete with a lot of testing so it should work pretty much as we expect.