[Merged by Bors] - fix: fluvio-test harness bug

[davidbeesley]

Addresses #2695

A race condition existed in the logic to evaluate whether or not tests run with fluvio-test passed, making it possible for a non-passing-test to appear to succeed.

Also, tests that timed-out were erroneously set as passing. Note that this effect was obscured as default timeout of one-hour was higher than github action timeout.

1. Added test harness validation tests that "failed to fail" using previous logic.
2. Added --expect-fail and --expect-timeout flags for tests that are supposed to fail or timeout and added --expect-timeout to longevity tests.
3. Tightened up logic so as to eliminate race condition.

Note: This fix exposes genuine failing tests, so it will not consistently pass CI pipeline.

validate-test-harness example output (ignoring stderr)

./build-scripts/install_target.sh
cargo build --bin fluvio-test -p fluvio-test   
cargo  build --bin fluvio -p fluvio-cli    
cargo build --bin fluvio-run -p fluvio-run    
echo "clean up previous installation"
clean up previous installation
./target/debug/fluvio cluster delete
./target/debug/fluvio-test expected_pass --local  --cluster-start
Start running fluvio test runner
Starting cluster and testing connection
remove cluster skipped
installing cluster
starting test in child process
Creating the topic: unused
topic "unused" created

Starting example test that passes
test complete, signaling to parent
Test passed
+----------+--------------+
| Pass?    | true         |
|----------+--------------|
| Duration | 827.659539ms |
+----------+--------------+
./target/debug/fluvio-test expected_fail --expect-fail
Start running fluvio test runner
Testing connection to Fluvio cluster in profile
starting test in child process
Creating the topic: unused
topic "unused" already exists

Starting example test that fails
test failed as expected, signaling parent
test complete, signaling to parent
Test passed
+----------+--------------+
| Pass?    | true         |
|----------+--------------|
| Duration | 2.731820019s |
+----------+--------------+
./target/debug/fluvio-test expected_fail_join_fail_first --expect-fail
Start running fluvio test runner
Testing connection to Fluvio cluster in profile
starting test in child process
Creating the topic: unused
topic "unused" already exists

Starting example test that fails
test failed as expected, signaling parent
test complete, signaling to parent
Test passed
+----------+--------------+
| Pass?    | true         |
|----------+--------------|
| Duration | 2.511258335s |
+----------+--------------+
./target/debug/fluvio-test expected_fail_join_success_first --expect-fail
Start running fluvio test runner
Testing connection to Fluvio cluster in profile
starting test in child process
Creating the topic: unused
topic "unused" already exists

Starting example test that fails
test failed as expected, signaling parent
test complete, signaling to parent
Test passed
+----------+--------------+
| Pass?    | true         |
|----------+--------------|
| Duration | 925.028775ms |
+----------+--------------+
./target/debug/fluvio-test expected_timeout --timeout 5sec --expect-timeout
Start running fluvio test runner
Testing connection to Fluvio cluster in profile
starting test in child process
Creating the topic: unused
topic "unused" already exists

Starting example test that timeouts
Test timed out as expected after 5 seconds
killing child test pid 1250657 name fluvio-test
killing child test pid 1250686 name fluvio-test
+----------+--------------+
| Pass?    | true         |
|----------+--------------|
| Duration | 5.405322312s |
+----------+--------------+

[sehz]

LGTM

[sehz]

bors r+

[bors]

Build failed:

• Done

[sehz]

bors r+

[bors]

Build failed:

• Done

[davidbeesley]

bors r+

[bors]

Build failed:

• Done

[davidbeesley]

bors r+

[bors]

Pull request successfully merged into master.

Build succeeded:

• Done