Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decide on the testing strategy #78

Open
igrr opened this issue Jan 19, 2023 · 0 comments
Open

Decide on the testing strategy #78

igrr opened this issue Jan 19, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@igrr
Copy link
Member

igrr commented Jan 19, 2023

There are a few options for setting up tests for esptool-js:

  1. Unit tests with a mocked serial port, similar to the ones in https://github.com/espressif/esp-serial-flasher/blob/a61f7449d7d3c4e63e5188c398b6ea1b1d35a002/test/test.cpp#L183-L239.
    • ➕ these tests should be very reliable
    • ➕ contributors can run the tests locally
    • ➖ these tests check only what was envisioned by the test author, which is good for detecting regressions but not so good at finding unknown problems
    • ➖ writing them usually requires some understanding of the underlying protocol
  2. Integration test against an emulator (such as QEMU), similar to https://github.com/espressif/esp-serial-flasher/blob/a61f7449d7d3c4e63e5188c398b6ea1b1d35a002/test/run_test.sh.
    • ➕ requires no understanding of the protocol (so e.g. I could write such a test)
    • ➕ contributors can run the tests locally (although less easy than the unit tests)
    • ➖ involves communication between processes, could be flaky
    • ➖ only the ESP32 chip is emulated in QEMU now
  3. Integration test against hardware, similar to https://github.com/espressif/esptool/blob/3f3d88a82477848b8fd055e73de33937aeb896e6/test/test_esptool.py#L349
    • ➕ requires no understanding of the protocol
    • ➕ checks against real hardware, which means a pretty good level of assurance
    • ➕ can run tests with chips which we don't have an emulator for
    • ➖ involves real hardware and a self-hosted CI runner, can be flaky for many reasons
    • ➖ extra maintenance cost for the self-hosted runner
    • ➖ tests running in PRs from forks might require approval to run
    • ➖ running such tests locally might be more tricky (need to manually test on multiple chips)
@igrr igrr added the enhancement New feature or request label Jan 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant