Set up hardware-in-loop testing

[jessebraham]

We have already set up self-hosted runners for GitHub actions.

We will need to better define what is required in order to bring this to fruition.

EDIT: There have been a number of updates on this task, please see the comments below!

[jessebraham]

Having issues with probe-rs, same error as reported here: probe-rs/probe-rs#1829

[SergioGasquez]

The probe-rs issue is now fixed (probe-rs/probe-rs#1832) but not yet released

[jessebraham]

I have managed to flash and run tests on-device using teleprobe/probe-rs and defmt, so we're unblocked and back on track here. I've written a simple test for GPIO so far, which I've played with a bit and seems to be working as expected.

I will continue writing tests, and also start working on getting the CI workflow set up to invoke them.

[bugadani]

Test cases that would be useful, a working list:

• embedded-hal DMA driver returns error when transfering from ROM (I don't have a good reproducer for someone's issue around this)
• short waits should not block embassy (systimer driver is/was buggy, workaround in Fix us waits in embassy hanging #906)

[jessebraham]

This week had some unexpected work pop up, so I unfortunately have not made as much progress and I had hoped. However, I will still provide some updates on the current status of things:

• I have written a CI workflow which should handle building the tests and running them on-device via our self-hosted runners. This needs a bit of work still, but is mostly complete.
• There are some logistical/devops issues which need solving still, which I will discuss with a colleague who works with CI as soon as possible. These are not technically blocking issues, however, but still need to be addressed eventually.
• The plan is to initially start with with just the single GPIO test binary, to prove out the process/workflow, and then additional tests can be added.
• @t-moe has opened a PR adding RISC-V support to defmt-test and panic-probe: (risc-v support for defmt-test knurling-rs/defmt#796). Once this is merged we will use this.

[jessebraham]

Things are moving fast in the embedded testing world, and only two weeks later things are looking a bit different!

• I have switched from defmt-test to embedded-test
  • https://github.com/probe-rs/embedded-test
• I have simplified much of the configuration, and came up with solutions for passing the chip and target to the test runner
• I have begun documenting how to write and run tests
• GPIO tests are still working, planning to write simple UART and Full-Duplex SPI tests too, probably, just to help get things started
• I still need to sort out some details with the CI workflow, but I'd say it's probably 80% of the way there

I think I can actually get a draft PR opened for this soon, and I will add the CI workflow at a later date, once it is ready. At least would be nice to get some eyes on what I've come up with so I can elicit some feedback.

[jessebraham]

@SergioGasquez has graciously offered to resurrect this from the dead 😁

[SergioGasquez]

Minor update, we already had our first successful HIL CI runs: https://github.com/esp-rs/esp-hal/actions/runs/8204261710/job/22440035135 🚀

[SergioGasquez]

Quick update regarding HIL:

• Its working fine for H2 and C6 and testing (at least, partially) aes, gpio, uart and spi_full_duplex, : https://github.com/esp-rs/esp-hal/actions/runs/8280334758
• We had several issues with HIL VMs:
  • GH actions jobs were running under tomas user. Now they run under espressif.
  • Some VMs had problems detecting (or they were disappearing after being detected) the targets and reloading them.
    • We still dont know the root cause of this. Tomas removed an unused VM (with a rustboard) and everything started working.
  • probe-rs was disappearing after running the HIL workflow:
    • swatinem/rust-cache action has a known issue:

      The cache cleaning process currently removes all the files from ~/.cargo/bin that were present before the action ran (for example rustc).

      • Either we don't use the action or we make the file immutable sudo chattr +i /home/espressif/.cargo/bin/probe-rs
• Board support:
  • Maybe tomorrow we will have a rebased version of the probe-rs branch that adds support for probe-rs test which will enable C3 and C2.
  • semihosting and embedded-test do not support Xtensa
• Documentation could be improved, atm we document the wiring and VM setup in: https://github.com/esp-rs/esp-hal/tree/feat/hil-testing/hil-test#running-tests-remotes-ie-on-self-hosted-runners

[jessebraham]

Thank you again for your work on this @SergioGasquez, I'm really excited to have this at our disposal. Hopefully the probe-rs issues can be resolved soon and we can start to move forward with this; I think we're pretty close to being able to open at least a draft PR and start scrutinizing things a bit.

I guess we still have not really decided which event(s) will trigger this workflow. I still don't think our merge queues are configured quite right, but I had originally thought of this somehow being a final check before merging. Not sure if this is possible with the merge_queue event or not, or if we need to think of something else. I think trying to run HIL tests each time a PR is updated will quickly lead to problems, so we need to find some happy medium.

[SergioGasquez]

I think we're pretty close to being able to open at least a draft PR and start scrutinizing things a bit.

Definitely, I think with what I have I can already create the first PR (as a draft at least), so we can start giving feedback.

I guess we still have not really decided which event(s) will trigger this workflow.

This is one of the things that we still need to figure out. Each time a PR gets updated seems a bit too much to me too, and I have not experience with merge_queues, as you mention having something that runs just before merging and that if fails it does not merge would be very sweet. Will have a look at merge_queues

[SergioGasquez]

I've gathered a list of the esp-hal modules and how hard/easy I think it would be to test those (take it with a grain of salt as I may have misevaluated some). Also, some modules can't be tested, or it does not make sense:

• aes - Easy
  • DMA
  • Non- DMA
• analog - Hard
• assist_debug - Hard
• clock -
• delay - Easy
• dma -
• ecc - Easy
• fmt -
• gpio - Easy
• hmac - Easy
• i2c - Hard
• i2s - Hard
• interrupt -
• lcd_cam -
• ledc -
• mcpwm - Hard
• otg_fs - Hard
• parl_io - Hard
• pcnt - Hard
• rmt - Hard?
• rng - Hard
• rom -
• rsa - Easy
• rtc_cntl -
• sha - Easy
• soc -
• spi
  • Half duplex - Hard
  • Full duplex - Easy
  • DMA
    • Half duplex
    • Full duplex
• systimer - Hard
• timer - Hard
• trace - Hard
• twai - Hard
• uart - Easy
• usb_serial_jtag - Easy?

[SergioGasquez]

Just opened a PR: #1297

[SergioGasquez]

Minor update: C3 is now also being tested on ci https://github.com/esp-rs/esp-hal/actions/runs/8326344742

[SergioGasquez]

Initial HIL has been merged! #1297

Current board support

• C3, C6 and H2 are supported
• We need to setup the S3 machine and include it on the CI workflow.
• C2 should be supported soon
• Xtensa devices would require adding Xtensa support in semihosting and probe-rs
  • See Support Xtensa (OpenOCD Semihosting) taiki-e/semihosting#9 and Xtensa semihosting support probe-rs/probe-rs#2301
  • S3 is now supported! We still need to upstream changes and setup a VM with it, so it will probably be in another PR.

Current tests

• Only aes, gpio, uart and spi_full_duplex are partially being tested. See full list of peripherals pending in: Set up hardware-in-loop testing #809 (comment)

[jessebraham]

I think we can probably close this issue at this point. There is definitely still work to be done, but I would say that HIL has been "set up", and the remaining work can be split out into their own issues.

@SergioGasquez if you think this is premature please feel free to reopen.

I've opened #1332 and #1334 for now.