Enforce zebra_test::command timeouts regardless of command output

[teor2345]

Version

zebra-test 3.0.0-alpha.0 (current main branch, commit eaf5473)

Platform

Linux 5.4.68 #1-NixOS SMP x86_64 GNU/Linux

Description

The zebra_test::command acceptance test framework contains a few undocumented surprises. The timeout is only checked in expect_stdout, and only after each newline. And the timeout kill check is inverted. (It appears that it has never been tested.)

Here's what I think we should fix:

• Check if the command is running before killing it (Test sync on testnet #1053)
• Enforce acceptance test command timeouts, even if the command doesn't output any lines
  • See the ignored tests in zebra-test/tests/command.rs
• Enforce timeouts during wait_with_output
• Limit the length of the output printed in the context
  • Allow developers to override the default via an env var, so they can see more output if needed
  • Print the first few and last few lines of output?

[teor2345]

If we had this fix, we could probably have diagnosed the hang in #1773:

running 2 tests
test sync_large_checkpoints_mainnet ... ok
test sync_large_checkpoints_testnet ... test sync_large_checkpoints_testnet has been running for over 60 seconds
Error: The operation was canceled.

(after 1 hour)
https://github.com/ZcashFoundation/zebra/pull/1773/checks?check_run_id=1930291039#step:9:414

[teor2345]

This hasn't been much of a problem recently, we fixed all the hangs.