Description
The problem you're addressing (if any)
Currently, a full run of DTS E2E tests takes around 2.5 hour (example). And most of the time is spent on QEMU rebooting and Dasharo Firmware configuration for network boot (for example for E2E003.010 it is about 3 minutes, with entire test about 3 minutes and 20 seconds, that is 90% of the test is spent in firmware).
Considering the fact that during the CI run not all test cases were implemented yet, we can conclude that at final state of DTS E2E test suite full run may take much more time.
Describe the solution you'd like
As was said, around 90% of tests time takes QEMU reboot, firmware setup and DTS network boot. But are those steps needed in every DTS E2E test?
The DST E2E test are primarily targeting testing on QEMU only DTS part, that is all scripts and applications launched inside Linux. There are no hardware/firmware modifications due to mocking of all hardware-specific functions and no firmware/hardware inf. reading. The only reason why QEMU reboot is may be needed - is the need to start every test from known environment state. But is the reboot the only way to start a DTS E2E test from a known state?
Because of the nature of DTS (DTS is composed of two parts: Linux image and Linux user space scripts/apps) we can consider change DTS E2E entry point from QEMU boot up to DTS main menu. That is, the QEMU boot up, firmware configuration and DTS boot could be done at the test suite start, and every DTS E2E test should expect DTS main menu to be launched. Then, all that will need to be done to make sure, that DTS E2E entry point is in the known state is to clean up some files and shell environment variables after every completed test. Here is an example list of whet will have to be cleaned up:
- DPP credentials;
- Error and log files;
- Any other files written during previous test.
- Shell env. variables
Where is the value to a user, and who might that user be?
This will drastically reduce time needed for DTS E2E tests execution. Targets all users using DTS E2E tests suite, but most importantly meta-dts workflows, so the workflows could be run not only on every release as it is being done currently but in every PR, improving quality of every contribution.
Describe alternatives you've considered
No response
Additional context
No response