HIL updates

[SergioGasquez]

Thank you for your contribution!

We appreciate the time and effort you've put into this pull request.
To help us review it efficiently, please ensure you've gone through the following checklist:

Submission Checklist 📝

• I have updated existing examples or added new ones (if applicable).
• My changes were added to the CHANGELOG.md in the proper section.

Extra:

• I have read the CONTRIBUTING.md guide and followed its instructions.

Pull Request Details 📖

Description

This PR addresses a few different things:

• Adds a -t/--test to only run a test with the xtask run-tests command
• Document the new HIL xtask subcommands in the readme
• Formated the hil-tests deps
• Updated embedded-tests to allow custom executors, see Configurable executor probe-rs/embedded-test#30. Thanks @bjoernQ!
• When executing all the test, it prints the list of failed tests and exits with a failure if there is any failed test.
• Fix xtask package Clippy lints

I wonder if its still worth to keep the hil-tests aliases that run all the tests for a target.

Testing

Execution of a single test:

esp-hal/xtask on  feat/hil-update is  v0.0.0 via  v1.79.0-nightly 
❯ cargo xtask run-tests esp32h2 -t uart
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.03s
     Running `/Users/sergio/Documents/Espressif/forks/esp-hal/target/debug/xtask run-tests esp32h2 -t uart`
[2024-04-08T08:58:17Z INFO  xtask] Building example '/Users/sergio/Documents/Espressif/forks/esp-hal/hil-test/tests/uart.rs' for 'esp32h2'
[2024-04-08T08:58:17Z INFO  xtask] Package: "tests/uart.rs"
    Finished `release` profile [optimized + debuginfo] target(s) in 0.15s
     Running tests/uart.rs (target/riscv32imac-unknown-none-elf/release/deps/uart-5f44d4b03a865568)
      Erasing ✔ [00:00:00] [####################################################################################] 192.00 KiB/192.00 KiB @ 1.29 MiB/s (eta 0s )
  Programming ✔ [00:00:00] [#####################################################################################] 35.75 KiB/35.75 KiB @ 39.81 KiB/s (eta 0s )    Finished in 1.059s

running 1 test
test tests::test_send_receive ... ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 289207279 while buffer size is 1024 for "up" channel 0 (defmt)
ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.49s

Execution of all tests:

esp-hal/xtask on  feat/hil-update is  v0.0.0 via  v1.79.0-nightly took 2s 
❯ cargo xtask run-tests esp32h2 
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.03s
     Running `/Users/sergio/Documents/Espressif/forks/esp-hal/target/debug/xtask run-tests esp32h2`
[2024-04-08T08:59:06Z INFO  xtask] Building example '/Users/sergio/Documents/Espressif/forks/esp-hal/hil-test/tests/aes.rs' for 'esp32h2'
[2024-04-08T08:59:06Z INFO  xtask] Package: "tests/aes.rs"
   Compiling hil-test v0.0.0 (/Users/sergio/Documents/Espressif/forks/esp-hal/hil-test)
    Finished `release` profile [optimized + debuginfo] target(s) in 1.13s
     Running tests/aes.rs (target/riscv32imac-unknown-none-elf/release/deps/aes-0bacdfb841a99582)
      Erasing ✔ [00:00:00] [##################################################################################] 192.00 KiB/192.00 KiB @ 984.93 KiB/s (eta 0s )
  Programming ✔ [00:00:00] [#####################################################################################] 35.96 KiB/35.96 KiB @ 60.16 KiB/s (eta 0s )    Finished in 0.809s
ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 598283353 while buffer size is 1024 for "up" channel 0 (defmt)

running 2 tests
test tests::test_aes_encryption ... ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 1973464005 while buffer size is 1024 for "up" channel 0 (defmt)
ok
test tests::test_aes_decryption ... ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 773238864 while buffer size is 1024 for "up" channel 0 (defmt)
ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.97s

[2024-04-08T08:59:09Z INFO  xtask] Building example '/Users/sergio/Documents/Espressif/forks/esp-hal/hil-test/tests/spi_full_duplex.rs' for 'esp32h2'
[2024-04-08T08:59:09Z INFO  xtask] Package: "tests/spi_full_duplex.rs"
   Compiling hil-test v0.0.0 (/Users/sergio/Documents/Espressif/forks/esp-hal/hil-test)
    Finished `release` profile [optimized + debuginfo] target(s) in 0.95s
     Running tests/spi_full_duplex.rs (target/riscv32imac-unknown-none-elf/release/deps/spi_full_duplex-a183c7e207700c0f)
      Erasing ✔ [00:00:00] [##################################################################################] 192.00 KiB/192.00 KiB @ 853.19 KiB/s (eta 0s )
  Programming ✔ [00:00:00] [#####################################################################################] 37.64 KiB/37.64 KiB @ 53.62 KiB/s (eta 0s )    Finished in 0.943s
ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 303502708 while buffer size is 1024 for "up" channel 0 (defmt)

running 4 tests
test tests::test_symestric_transfer                      ... ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 290607941 while buffer size is 1024 for "up" channel 0 (defmt)
ok
test tests::test_asymestric_transfer                     ... ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 1341298459 while buffer size is 1024 for "up" channel 0 (defmt)
ok
test tests::test_symestric_transfer_huge_buffer          ... ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 135274955 while buffer size is 1024 for "up" channel 0 (defmt)
ok
test tests::test_symestric_transfer_huge_buffer_no_alloc ... ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 2247497692 while buffer size is 1024 for "up" channel 0 (defmt)
ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.93s

[2024-04-08T08:59:14Z INFO  xtask] Building example '/Users/sergio/Documents/Espressif/forks/esp-hal/hil-test/tests/uart_async.rs' for 'esp32h2'
[2024-04-08T08:59:14Z INFO  xtask] Package: "tests/uart_async.rs"
   Compiling hil-test v0.0.0 (/Users/sergio/Documents/Espressif/forks/esp-hal/hil-test)
    Finished `release` profile [optimized + debuginfo] target(s) in 1.00s
     Running tests/uart_async.rs (target/riscv32imac-unknown-none-elf/release/deps/uart_async-6403fcaac0a504a3)
      Erasing ✔ [00:00:00] [##################################################################################] 192.00 KiB/192.00 KiB @ 887.71 KiB/s (eta 0s )
  Programming ✔ [00:00:00] [#####################################################################################] 38.79 KiB/38.79 KiB @ 56.94 KiB/s (eta 0s )    Finished in 0.916s
ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 3734281554 while buffer size is 1024 for "up" channel 0 (defmt)

running 1 test
test tests::test_send_receive ... ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 982331164 while buffer size is 1024 for "up" channel 0 (defmt)
ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.50s

[2024-04-08T08:59:17Z INFO  xtask] Building example '/Users/sergio/Documents/Espressif/forks/esp-hal/hil-test/tests/gpio.rs' for 'esp32h2'
[2024-04-08T08:59:17Z INFO  xtask] Package: "tests/gpio.rs"
   Compiling hil-test v0.0.0 (/Users/sergio/Documents/Espressif/forks/esp-hal/hil-test)
    Finished `release` profile [optimized + debuginfo] target(s) in 1.03s
     Running tests/gpio.rs (target/riscv32imac-unknown-none-elf/release/deps/gpio-8a798c29a24e8e6a)
      Erasing ✔ [00:00:00] [##################################################################################] 192.00 KiB/192.00 KiB @ 866.50 KiB/s (eta 0s )
  Programming ✔ [00:00:00] [#####################################################################################] 39.68 KiB/39.68 KiB @ 55.87 KiB/s (eta 0s )    Finished in 0.95s
ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 4094361401 while buffer size is 1024 for "up" channel 0 (defmt)

running 6 tests
test tests::test_async_edge     ... ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 194126822 while buffer size is 1024 for "up" channel 0 (defmt)
ok
test tests::test_a_pin_can_wait ... ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 2729777517 while buffer size is 1024 for "up" channel 0 (defmt)
ok
test tests::test_gpio_input     ... ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 3204006546 while buffer size is 1024 for "up" channel 0 (defmt)
ok
test tests::test_gpio_output    ... ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 1691747781 while buffer size is 1024 for "up" channel 0 (defmt)
ok
test tests::test_gpio_interrupt ... ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 3566446721 while buffer size is 1024 for "up" channel 0 (defmt)
ok
test tests::test_gpio_od        ... ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 1232102943 while buffer size is 1024 for "up" channel 0 (defmt)
ok

test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.99s

[2024-04-08T08:59:22Z INFO  xtask] Building example '/Users/sergio/Documents/Espressif/forks/esp-hal/hil-test/tests/uart.rs' for 'esp32h2'
[2024-04-08T08:59:22Z INFO  xtask] Package: "tests/uart.rs"
    Finished `release` profile [optimized + debuginfo] target(s) in 0.14s
     Running tests/uart.rs (target/riscv32imac-unknown-none-elf/release/deps/uart-5f44d4b03a865568)
      Erasing ✔ [00:00:00] [##################################################################################] 192.00 KiB/192.00 KiB @ 862.97 KiB/s (eta 0s )
  Programming ✔ [00:00:00] [#####################################################################################] 35.75 KiB/35.75 KiB @ 51.28 KiB/s (eta 0s )    Finished in 0.937s
ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 4062726515 while buffer size is 1024 for "up" channel 0 (defmt)

running 1 test
test tests::test_send_receive ... ERROR probe_rs::util::rtt: 
Error reading from RTT: The control block has been corrupted. write pointer is 1267861596 while buffer size is 1024 for "up" channel 0 (defmt)
ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.50s

probe-rs is still throwing some werid RTT errors, but those can be still ignored. probe-rs used for tests is the one in the VMs (ddd59fa), I've also tried the latest probe-rs but same output

[SergioGasquez]

Also, I just noticed that when running all the tests, if a test failed, we keep running all the other tests. (I guess this should also happen for the examples at it reuses the same code), changed it to fail and exit if a test fails (04800cf). Happy to revert it in case we dont want this.

[SergioGasquez]

Clippy lints should be resolved by #1410

[bjoernQ]

Thanks! That's very useful! Running a single test was not too funny before :)

I guess we can keep the aliases - they won't hurt but no strong opinion on that

Also, I just noticed that when running all the tests, if a test failed, we keep running all the other tests. (I guess this should also happen for the examples at it reuses the same code), changed it to fail and exit if a test fails (04800cf). Happy to revert it in case we dont want this.

I think ideally it would be good to try all tests and exit with a non-zero exit code if one failed. Otherwise, if multiple tests fail it might take several attempts to get CI green (but on the other hand users can just run them locally until it's fine - I guess that's preferrable).

Locally I often run into the problem that if one test failed the following tests fail for no reason - so maybe there is no use to try the remaining tests, then?

Probably good to hear other's voices here

[SergioGasquez]

By the way, as also @bjoernQ noticed, the GPIO test test_gpio_interrupt is failing for Xtensa targets (at least for S2 and S3), issue #1413

[jessebraham]

I wonder if its still worth to keep the hil-tests aliases that run all the tests for a target.

I think these can probably be removed. The xtask package hadn't been created yet when these were added, and we're now recommending using the xtask, so let's just get rid of them I think.

Also, I just noticed that when running all the tests, if a test failed, we keep running all the other tests. (I guess this should also happen for the examples at it reuses the same code), changed it to fail and exit if a test fails (04800cf). Happy to revert it in case we dont want this.

I think, assuming a failing test does not cause the failure of subsequent tests, it would be best to continue running all tests. In the case of a PR where multiple peripherals are affected, it could potentially take a number of HIL test runs in order to get things passing otherwise. While it would be nice to assume that everybody tests everything locally, I do not think this is a safe assumption to make.

[SergioGasquez]

I reverted my changes and now, upon a test fail the other tests are also run, as it was initially.

I've also removed the aliases as they wont be of any use when we grow our tests suit as theyll fail if they can't run all the tests regardless if the test is supported for that target unless we add probe-rs/probe-rs#2344 and maintain the skipped list.

[SergioGasquez]

Just noticed at the moment, the workflow will get green even with some tests failing, so I will add some logic to run all the test but tracking the failed tests and throw an error printing that list at the end.

Example of HIL run with fails that is green: https://github.com/esp-rs/esp-hal/actions/runs/8615443424/job/23610994749

[jessebraham]

Thanks for updating the run-tests subcommand, handy to be able to build just a single test 😁

[JurajSadel]

LGTM, thanks!