Lunatic test harness

[bkolobara]

The #[lunatic::test] macro turns a test into a process, but it still uses the default test harness underneath and is not perfect.

The biggest issue comes from the fact that rust compiled to WebAssembly uses panic=abort, meaning that every time a panic occurs the process is terminated and there is no unwinding or panic catching capability. This can lead to silent errors. Let us have a look at the following test output:

running 6 tests
test message_custom_type ... ok
test message_integer ... ok
test message_resource ... ok
test message_vector ... ok
test request_reply ... ok
test timeout ... thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `2`', tests/messaging_test.rs:102:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
     Running tests/process_test.rs (target/wasm32-wasi/debug/deps/process_test-20848690c2dbf9e1.wasm)

running 4 tests
test compute_limit ... ok
test link_with_tags ... ok
test memory_limit ... memory allocation of 1200000 bytes failed
ok
test spawn_link ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

The timeout test failed on a assert_eq(1, 2), but an assert failure will cause a panic and will terminate the execution of all the other tests following and not print any statistics how many passed or failed. It's still possible to notice that the test failed, but with a lot of tests and output it can be easily missed.

Even we can wrap tests into processes and use links to check for failures, I'm not aware of a way to report a failure without a panic. Tests can also return a Result to indicate success or failure, but under the hood rust will still use panics in the default test harness.

I think that the best way forward would be the way many testing frameworks for embedded Rust work, where panic=abort also applies (like defmt-test). This would require us to develop a specific testing framework around lunatic's capabilities.

I'm opening this issue just to kick off the discussion about finding "the perfect" testing story around Rust code on lunatic.

[teymour-aldridge]

I guess to start having something very similar to the vanilla Rust test harness would be great.

More long term, in terms of testing, I think that it would be excellent if Lunatic could make a lot of testing techniques that are highly effective, but sadly not sufficiently ubiquitous easier to use (or more prominent).

For example:

• property testing (e.g. proptest)
• model checking
• simulation
• fuzzing
• chaos engineering
• etc, etc

[bkolobara]

Closed by #15!