Reducing time needed for DTS E2E tests execution

[DaniilKl]

The problem you're addressing (if any)

Currently, a full run of DTS E2E tests takes around 2.5 hour (example). And most of the time is spent on QEMU rebooting and Dasharo Firmware configuration for network boot (for example for E2E003.010 it is about 3 minutes, with entire test about 3 minutes and 20 seconds, that is 90% of the test is spent in firmware).

Considering the fact that during the CI run not all test cases were implemented yet, we can conclude that at final state of DTS E2E test suite full run may take much more time.

Describe the solution you'd like

As was said, around 90% of tests time takes QEMU reboot, firmware setup and DTS network boot. But are those steps needed in every DTS E2E test?

The DST E2E test are primarily targeting testing on QEMU only DTS part, that is all scripts and applications launched inside Linux. There are no hardware/firmware modifications due to mocking of all hardware-specific functions and no firmware/hardware inf. reading. The only reason why QEMU reboot is may be needed - is the need to start every test from known environment state. But is the reboot the only way to start a DTS E2E test from a known state?

Because of the nature of DTS (DTS is composed of two parts: Linux image and Linux user space scripts/apps) we can consider change DTS E2E entry point from QEMU boot up to DTS main menu. That is, the QEMU boot up, firmware configuration and DTS boot could be done at the test suite start, and every DTS E2E test should expect DTS main menu to be launched. Then, all that will need to be done to make sure, that DTS E2E entry point is in the known state is to clean up some files and shell environment variables after every completed test. Here is an example list of whet will have to be cleaned up:

1. DPP credentials;
2. Error and log files;
3. Any other files written during previous test.
4. Shell env. variables

Where is the value to a user, and who might that user be?

This will drastically reduce time needed for DTS E2E tests execution. Targets all users using DTS E2E tests suite, but most importantly meta-dts workflows, so the workflows could be run not only on every release as it is being done currently but in every PR, improving quality of every contribution.

Describe alternatives you've considered

No response

Additional context

No response

[m-iwanicki]

Even worse is when most tests fail. Here is one example of almost 4 hours with most tests failing: https://github.com/Dasharo/meta-dts/actions/runs/12434149601

[m-iwanicki]

With #658:

	Time
Prepare test suite	21:38
HCL report	23:24
Rest	06:02
Total runtime	51:04

To reduce runtime more we would need to mock HCL report or reduce preparation time.
Depending on amount of tests and how long they take (per tag) preparation time can be significant proportion of runtime e.g.:

• novacustom_heads - preparation takes up around 88% of runtime
• novacustom_comm - preparation takes up around 22% of runtime

[DaniilKl]

Results:

Without #658 and this Dasharo/dts-scripts#63:

λ time robot -b command_log.txt -v snipeit:no -L TRACE -v config:qemu -v rte_ip:127.0.0.1 -v dpp_logs_key:'x' -v dpp_download_key:'x' -v dpp_password:'x' -v boot_dts_from_ipxe_shell:True -v dts_ipxe_link:http://192.168.4.175:8080/ipxe -i msi_dpp dts/dts-e2e.robot
==============================================================================
Dts-E2E                                                                       
==============================================================================
E2E003.003 MSI PRO Z690-A DDR-4 initial deployment (legacy -> Core... | PASS |
------------------------------------------------------------------------------
E2E003.004 MSI PRO Z690-A initial deployment (legacy -> Coreboot +... | PASS |
------------------------------------------------------------------------------
E2E003.005 MSI PRO Z690-A DDR-4 initial deployment (legacy -> Core... | PASS |
------------------------------------------------------------------------------
E2E003.006 MSI PRO Z690-A initial deployment (legacy -> Coreboot +... | PASS |
------------------------------------------------------------------------------
E2E003.009 MSI PRO Z690-A DDR-4 update (Coreboot + UEFI -> Coreboo... | PASS |
------------------------------------------------------------------------------
E2E003.010 MSI PRO Z690-A update (Coreboot + UEFI -> Coreboot + UE... | PASS |
------------------------------------------------------------------------------
Dts-E2E                                                               | PASS |
6 tests, 6 passed, 0 failed
==============================================================================
Debug:   /home/danillklimuk/Projects/DTS/open-source-firmware-validation/command_log.txt
Output:  /home/danillklimuk/Projects/DTS/open-source-firmware-validation/output.xml
Log:     /home/danillklimuk/Projects/DTS/open-source-firmware-validation/log.html
Report:  /home/danillklimuk/Projects/DTS/open-source-firmware-validation/report.html
robot -b command_log.txt -v snipeit:no -L TRACE -v config:qemu -v  -v  -v  -v 
284.90s user 3.05s system 23% cpu 20:12.88 total

Without this Dasharo/dts-scripts#63:

λ time robot -b command_log.txt -v snipeit:no -L TRACE -v config:qemu -v rte_ip:127.0.0.1 -v dpp_logs_key:'x' -v dpp_download_key:'x' -v dpp_password:'x' -v boot_dts_from_ipxe_shell:True -v dts_ipxe_link:http://192.168.4.175:8080/ipxe -i msi_dpp dts/dts-e2e.robot
==============================================================================
Dts-E2E                                                                       
==============================================================================
E2E003.003 MSI PRO Z690-A DDR-4 initial deployment (legacy -> Core... | PASS |
------------------------------------------------------------------------------
E2E003.004 MSI PRO Z690-A initial deployment (legacy -> Coreboot +... | PASS |
------------------------------------------------------------------------------
E2E003.005 MSI PRO Z690-A DDR-4 initial deployment (legacy -> Core... | PASS |
------------------------------------------------------------------------------
E2E003.006 MSI PRO Z690-A initial deployment (legacy -> Coreboot +... | PASS |
------------------------------------------------------------------------------
E2E003.009 MSI PRO Z690-A DDR-4 update (Coreboot + UEFI -> Coreboo... | PASS |
------------------------------------------------------------------------------
E2E003.010 MSI PRO Z690-A update (Coreboot + UEFI -> Coreboot + UE... | PASS |
------------------------------------------------------------------------------
Dts-E2E                                                               | PASS |
6 tests, 6 passed, 0 failed
==============================================================================
Debug:   /home/danillklimuk/Projects/DTS/open-source-firmware-validation/command_log.txt
Output:  /home/danillklimuk/Projects/DTS/open-source-firmware-validation/output.xml
Log:     /home/danillklimuk/Projects/DTS/open-source-firmware-validation/log.html
Report:  /home/danillklimuk/Projects/DTS/open-source-firmware-validation/report.html
robot -b command_log.txt -v snipeit:no -L TRACE -v config:qemu -v  -v  -v  -v 
51.44s user 1.94s system 10% cpu 8:10.60 total

With both PRs:

λ time robot -b command_log.txt -v snipeit:no -L TRACE -v config:qemu -v rte_ip:127.0.0.1 -v dpp_logs_key:'x' -v dpp_download_key:'x' -v dpp_password:'x' -v boot_dts_from_ipxe_shell:True -v dts_ipxe_link:http://192.168.4.175:8080/ipxe -i msi_dpp dts/dts-e2e.robot
==============================================================================
Dts-E2E                                                                       
==============================================================================
E2E003.003 MSI PRO Z690-A DDR-4 initial deployment (legacy -> Core... | PASS |
------------------------------------------------------------------------------
E2E003.004 MSI PRO Z690-A initial deployment (legacy -> Coreboot +... | PASS |
------------------------------------------------------------------------------
E2E003.005 MSI PRO Z690-A DDR-4 initial deployment (legacy -> Core... | PASS |
------------------------------------------------------------------------------
E2E003.006 MSI PRO Z690-A initial deployment (legacy -> Coreboot +... | PASS |
------------------------------------------------------------------------------
E2E003.009 MSI PRO Z690-A DDR-4 update (Coreboot + UEFI -> Coreboo... | PASS |
------------------------------------------------------------------------------
E2E003.010 MSI PRO Z690-A update (Coreboot + UEFI -> Coreboot + UE... | PASS |
------------------------------------------------------------------------------
Dts-E2E                                                               | PASS |
6 tests, 6 passed, 0 failed
==============================================================================
Debug:   /home/danillklimuk/Projects/DTS/open-source-firmware-validation/command_log.txt
Output:  /home/danillklimuk/Projects/DTS/open-source-firmware-validation/output.xml
Log:     /home/danillklimuk/Projects/DTS/open-source-firmware-validation/log.html
Report:  /home/danillklimuk/Projects/DTS/open-source-firmware-validation/report.html
robot -b command_log.txt -v snipeit:no -L TRACE -v config:qemu -v  -v  -v  -v
 35.97s user 1.03s system 14% cpu 4:07.11 total

So it is now 5 times faster for MSI DPP tests.

[DaniilKl]

@artur-rs, we should consider reducing preparation time as well, but this only be possible after we could create one set of credentials for all subscriptions, that is after infrastructure changes.

[m-iwanicki]

With current changes test time got reduced from 2 hours to 35 minutes (calculated by adding runtime of all Run DTS tests jobs)