Add tests for few upcoming changes #654

marmarek · 2025-02-13T15:52:05Z

See individual commit messages for details

codecov · 2025-02-13T16:31:25Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.73%. Comparing base (9a89893) to head (d75733d).
Report is 11 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #654      +/-   ##
==========================================
- Coverage   69.75%   69.73%   -0.02%     
==========================================
  Files          58       58              
  Lines       12550    12550              
==========================================
- Hits         8754     8752       -2     
- Misses       3796     3798       +2

Flag	Coverage Δ
unittests	`69.73% <ø> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

qubesos-bot · 2025-02-13T23:20:45Z

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025030301-4.3&flavor=pull-requests

Test run included the following:

Assure Update window width does not break OpenQA qubes-desktop-linux-manager#248 (QubesOS/qubes-desktop-linux-manager@1af77ee)
Revert to red label while restoring on missing labels qubes-core-admin-client#335 (QubesOS/qubes-core-admin-client@c4940be)
Add tests for few upcoming changes #654 (d75733d)
tests: integration tests for boot mode #664 (fcca635)
Reproduce bug in qrexec tests #661 (87f187d)
Add Arch packaging qubes-core-admin-client#332 (QubesOS/qubes-core-admin-client@e22c531)
Mark VMs with prohibit-start feature in updater qubes-desktop-linux-manager#247 (QubesOS/qubes-desktop-linux-manager@0c8493d)
Move all binaries from /usr/sbin to /usr/bin qubes-core-qubesdb#68 (QubesOS/qubes-core-qubesdb@658ce33)
Add boot mode support #653 (975b945)
Add the feature to prohibit starting a qube qubes-manager#408 (QubesOS/qubes-manager@0fea5d5)
Update firmware packages from bookworm-backports on Debian 12 qubes-core-admin-linux#184 (QubesOS/qubes-core-admin-linux@078c9db)
Unify method to query available terminals qubes-core-agent-linux#556 (QubesOS/qubes-core-agent-linux@5e5fc52)
Allow equal sign and coma in feature requests #665 (60ed744)

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025021804-4.3&flavor=update

system_tests_basic_vm_qrexec_gui
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=3b01c8...
system_tests_extra
- TC_00_QVCTest_whonix-gateway-17: test_010_screenshare (failure)
  ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 0 == 0
system_tests_qrexec
- TC_00_Qrexec_whonix-gateway-17: test_065_qrexec_exit_code_vm (failure)
  ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^... AssertionError: b'125\n' != b'0\n'
system_tests_kde_gui_interactive
- gui_keyboard_layout: wait_serial (wait serial expected)
  # wait_serial expected: "echo -e '[Layout]\nLayoutList=us,de' | sud...
- gui_keyboard_layout: Failed (test died)
  # Test died: command 'test "$(cd ~user;ls e1*)" = "$(qvm-run -p wor...
system_tests_basic_vm_qrexec_gui_zfs
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=4b3bbc...
system_tests_audio
- TC_20_AudioVM_Pulse_whonix-workstation-17: test_221_audio_rec_muted_pulseaudio (error + cleanup)
  raise TimeoutError from exc_val... TimeoutError
system_tests_basic_vm_qrexec_gui_btrfs
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=2094a4...
system_tests_basic_vm_qrexec_gui_ext4
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=3d69b2...
system_tests_basic_vm_qrexec_gui_xfs
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=fddc82...
system_tests_basic_vm_qrexec_gui@hw7
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=1bd747...

Failed tests

11 failures

system_tests_basic_vm_qrexec_gui
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=3b01c8...
system_tests_extra
- TC_00_QVCTest_whonix-gateway-17: test_010_screenshare (failure)
  ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 0 == 0
system_tests_qrexec
- TC_00_Qrexec_whonix-gateway-17: test_065_qrexec_exit_code_vm (failure)
  ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^... AssertionError: b'125\n' != b'0\n'
system_tests_kde_gui_interactive
- gui_keyboard_layout: wait_serial (wait serial expected)
  # wait_serial expected: "echo -e '[Layout]\nLayoutList=us,de' | sud...
- gui_keyboard_layout: Failed (test died)
  # Test died: command 'test "$(cd ~user;ls e1*)" = "$(qvm-run -p wor...
system_tests_basic_vm_qrexec_gui_zfs
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=4b3bbc...
system_tests_audio
- TC_20_AudioVM_Pulse_whonix-workstation-17: test_221_audio_rec_muted_pulseaudio (error + cleanup)
  raise TimeoutError from exc_val... TimeoutError
system_tests_basic_vm_qrexec_gui_btrfs
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=2094a4...
system_tests_basic_vm_qrexec_gui_ext4
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=3d69b2...
system_tests_basic_vm_qrexec_gui_xfs
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=fddc82...
system_tests_basic_vm_qrexec_gui@hw7
- TC_00_Basic: test_211_bootmode_standalone (failure)
  AssertionError: 'opv1=val1' not found in 'systemd.machine_id=1bd747...

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/129058#dependencies

15 fixed

system_tests_suspend
- suspend: unnamed test (unknown)
- suspend: Failed (test died)
  # Test died: no candidate needle with tag(s) 'SUSPEND-FAILED' match...
system_tests_qrexec
- TC_00_Qrexec_whonix-workstation-17: test_070_qrexec_vm_simultaneous_write (failure)
  exited with non-zero code 125; stderr: b"dd: error writing 'standa...
system_tests_kde_gui_interactive
- clipboard_and_web: unnamed test (unknown)
- clipboard_and_web: Failed (test died)
  # Test died: no candidate needle with tag(s) 'clipboard-paste-notif...
system_tests_audio
- TC_20_AudioVM_Pulse_whonix-workstation-17: test_223_audio_play_hvm (failure)
  AssertionError: too short audio, expected 10s, got 7.33528344671201...
- TC_20_AudioVM_PipeWire_whonix-workstation-17: test_226_audio_playback_pipewire (failure)
  AssertionError: too short audio, expected 10s, got 9.19027210884353...
- TC_20_AudioVM_PipeWire_whonix-workstation-17: test_251_audio_playback_audiovm_pipewire_late_start (failure)
  AssertionError: too short audio, expected 10s, got 9.22900226757369...
system_tests_whonix@hw7
- whonixcheck: fail (unknown)
  Whonixcheck for sys-whonix failed...
- whonixcheck: unnamed test (unknown)
system_tests_basic_vm_qrexec_gui_btrfs
- TC_03_QvmRevertTemplateChanges: test_000_revert_linux (error)
  subprocess.CalledProcessError: Command '['sha1sum', '/var/lib/qubes...
system_tests_suspend@hw1
- suspend: unnamed test (unknown)
- suspend: Failed (test died)
  # Test died: no candidate needle with tag(s) 'SUSPEND-FAILED' match...
system_tests_whonix
- whonixcheck: fail (unknown)
  Whonixcheck for sys-whonix failed...
- whonixcheck: unnamed test (unknown)

Unstable tests

Performance Tests

Performance degradation:

31 performance degradations

debian-12-xfce_exec: 8.47 :small_red_triangle_up: ( previous job: 7.15, degradation: 118.39%)
debian-12-xfce_exec-data-simplex: 71.98 :small_red_triangle_up: ( previous job: 48.93, degradation: 147.11%)
debian-12-xfce_exec-data-duplex: 69.42 :small_red_triangle_up: ( previous job: 50.76, degradation: 136.76%)
debian-12-xfce_exec-data-duplex-root: 73.53 :small_red_triangle_up: ( previous job: 64.91, degradation: 113.28%)
debian-12-xfce_socket-data-duplex: 163.41 :small_red_triangle_up: ( previous job: 81.49, degradation: 200.54%)
fedora-41-xfce_exec-data-simplex: 79.01 :small_red_triangle_up: ( previous job: 49.65, degradation: 159.14%)
fedora-41-xfce_exec-data-duplex: 72.56 :small_red_triangle_up: ( previous job: 49.08, degradation: 147.83%)
fedora-41-xfce_exec-data-duplex-root: 101.24 :small_red_triangle_up: ( previous job: 81.65, degradation: 123.99%)
fedora-41-xfce_socket-data-duplex: 144.14 :small_red_triangle_up: ( previous job: 78.62, degradation: 183.33%)
whonix-gateway-17_exec-data-simplex: 75.85 :small_red_triangle_up: ( previous job: 48.76, degradation: 155.55%)
whonix-gateway-17_exec-data-duplex: 75.18 :small_red_triangle_up: ( previous job: 48.55, degradation: 154.85%)
whonix-gateway-17_exec-data-duplex-root: 102.93 :small_red_triangle_up: ( previous job: 70.13, degradation: 146.78%)
whonix-gateway-17_socket-data-duplex: 172.33 :small_red_triangle_up: ( previous job: 82.74, degradation: 208.28%)
whonix-workstation-17_exec-data-simplex: 70.48 :small_red_triangle_up: ( previous job: 47.01, degradation: 149.91%)
whonix-workstation-17_exec-data-duplex: 74.21 :small_red_triangle_up: ( previous job: 49.48, degradation: 149.99%)
whonix-workstation-17_exec-data-duplex-root: 101.23 :small_red_triangle_up: ( previous job: 79.93, degradation: 126.65%)
whonix-workstation-17_socket-data-duplex: 175.16 :small_red_triangle_up: ( previous job: 81.71, degradation: 214.36%)
dom0_root_seq1m_q8t1_write 3:write_bandwidth_kb: 54344.00 :small_red_triangle_up: ( previous job: 276742.00, degradation: 19.64%)
dom0_root_seq1m_q1t1_read 3:read_bandwidth_kb: 123293.00 :small_red_triangle_up: ( previous job: 423495.00, degradation: 29.11%)
dom0_root_seq1m_q1t1_write 3:write_bandwidth_kb: 90898.00 :small_red_triangle_up: ( previous job: 185030.00, degradation: 49.13%)
dom0_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 12512.00 :small_red_triangle_up: ( previous job: 100699.00, degradation: 12.43%)
dom0_varlibqubes_rnd4k_q32t1_write 3:write_bandwidth_kb: 3952.00 :small_red_triangle_up: ( previous job: 8767.00, degradation: 45.08%)
fedora-41-xfce_root_seq1m_q1t1_read 3:read_bandwidth_kb: 305529.00 :small_red_triangle_up: ( previous job: 343795.00, degradation: 88.87%)
fedora-41-xfce_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 3126.00 :small_red_triangle_up: ( previous job: 3785.00, degradation: 82.59%)
fedora-41-xfce_private_seq1m_q8t1_read 3:read_bandwidth_kb: 342448.00 :small_red_triangle_up: ( previous job: 401907.00, degradation: 85.21%)
fedora-41-xfce_private_seq1m_q1t1_read 3:read_bandwidth_kb: 276159.00 :small_red_triangle_up: ( previous job: 357875.00, degradation: 77.17%)
fedora-41-xfce_private_rnd4k_q32t1_read 3:read_bandwidth_kb: 32766.00 :small_red_triangle_up: ( previous job: 87999.00, degradation: 37.23%)
fedora-41-xfce_private_rnd4k_q32t1_write 3:write_bandwidth_kb: 1974.00 :small_red_triangle_up: ( previous job: 3885.00, degradation: 50.81%)
fedora-41-xfce_private_rnd4k_q1t1_write 3:write_bandwidth_kb: 831.00 :small_red_triangle_up: ( previous job: 1613.00, degradation: 51.52%)
fedora-41-xfce_volatile_seq1m_q1t1_write 3:write_bandwidth_kb: 52560.00 :small_red_triangle_up: ( previous job: 78093.00, degradation: 67.30%)
fedora-41-xfce_volatile_rnd4k_q1t1_write 3:write_bandwidth_kb: 1809.00 :small_red_triangle_up: ( previous job: 2693.00, degradation: 67.17%)

Remaining performance tests:

41 tests

debian-12-xfce_exec-root: 29.33 :small_red_triangle_up: ( previous job: 27.97, degradation: 104.88%)
debian-12-xfce_socket: 8.71 :small_red_triangle_up: ( previous job: 8.33, degradation: 104.63%)
debian-12-xfce_socket-root: 8.61 :small_red_triangle_up: ( previous job: 8.20, degradation: 104.96%)
fedora-41-xfce_exec: 9.20 :small_red_triangle_up: ( previous job: 9.13, degradation: 100.68%)
fedora-41-xfce_exec-root: 66.29 :small_red_triangle_up: ( previous job: 61.17, degradation: 108.37%)
fedora-41-xfce_socket: 8.57 🟢 ( previous job: 8.66, improvement: 98.99%)
fedora-41-xfce_socket-root: 8.20 🟢 ( previous job: 8.61, improvement: 95.30%)
whonix-gateway-17_exec: 7.01 🟢 ( previous job: 7.87, improvement: 89.08%)
whonix-gateway-17_exec-root: 38.99 :small_red_triangle_up: ( previous job: 38.36, degradation: 101.64%)
whonix-gateway-17_socket: 7.85 :small_red_triangle_up: ( previous job: 7.54, degradation: 104.02%)
whonix-gateway-17_socket-root: 8.32 :small_red_triangle_up: ( previous job: 8.27, degradation: 100.58%)
whonix-workstation-17_exec: 7.85 🟢 ( previous job: 8.23, improvement: 95.32%)
whonix-workstation-17_exec-root: 57.69 :small_red_triangle_up: ( previous job: 52.56, degradation: 109.75%)
whonix-workstation-17_socket: 8.84 :small_red_triangle_up: ( previous job: 8.21, degradation: 107.66%)
whonix-workstation-17_socket-root: 9.00 :small_red_triangle_up: ( previous job: 8.20, degradation: 109.81%)
dom0_root_seq1m_q8t1_read 3:read_bandwidth_kb: 453536.00 :small_red_triangle_up: ( previous job: 486352.00, degradation: 93.25%)
dom0_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 4119.00 :green_circle: ( previous job: 3277.00, improvement: 125.69%)
dom0_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 11846.00 :green_circle: ( previous job: 10163.00, improvement: 116.56%)
dom0_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 342.00 :green_circle: ( previous job: 282.00, improvement: 121.28%)
dom0_varlibqubes_seq1m_q8t1_read 3:read_bandwidth_kb: 469792.00 :small_red_triangle_up: ( previous job: 475329.00, degradation: 98.84%)
dom0_varlibqubes_seq1m_q8t1_write 3:write_bandwidth_kb: 174268.00 :green_circle: ( previous job: 95209.00, improvement: 183.04%)
dom0_varlibqubes_seq1m_q1t1_read 3:read_bandwidth_kb: 433474.00 :green_circle: ( previous job: 433474.00, improvement: 100.00%)
dom0_varlibqubes_seq1m_q1t1_write 3:write_bandwidth_kb: 206843.00 :green_circle: ( previous job: 164133.00, improvement: 126.02%)
dom0_varlibqubes_rnd4k_q32t1_read 3:read_bandwidth_kb: 102343.00 :green_circle: ( previous job: 99808.00, improvement: 102.54%)
dom0_varlibqubes_rnd4k_q1t1_read 3:read_bandwidth_kb: 9448.00 :green_circle: ( previous job: 7053.00, improvement: 133.96%)
dom0_varlibqubes_rnd4k_q1t1_write 3:write_bandwidth_kb: 5080.00 :green_circle: ( previous job: 3868.00, improvement: 131.33%)
fedora-41-xfce_root_seq1m_q8t1_read 3:read_bandwidth_kb: 377593.00 :small_red_triangle_up: ( previous job: 396586.00, degradation: 95.21%)
fedora-41-xfce_root_seq1m_q8t1_write 3:write_bandwidth_kb: 229648.00 :green_circle: ( previous job: 99783.00, improvement: 230.15%)
fedora-41-xfce_root_seq1m_q1t1_write 3:write_bandwidth_kb: 120771.00 :green_circle: ( previous job: 44770.00, improvement: 269.76%)
fedora-41-xfce_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 90394.00 :green_circle: ( previous job: 86742.00, improvement: 104.21%)
fedora-41-xfce_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 8742.00 :green_circle: ( previous job: 8623.00, improvement: 101.38%)
fedora-41-xfce_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 1407.00 :green_circle: ( previous job: 1126.00, improvement: 124.96%)
fedora-41-xfce_private_seq1m_q8t1_write 3:write_bandwidth_kb: 266271.00 :green_circle: ( previous job: 116848.00, improvement: 227.88%)
fedora-41-xfce_private_seq1m_q1t1_write 3:write_bandwidth_kb: 111419.00 :green_circle: ( previous job: 41375.00, improvement: 269.29%)
fedora-41-xfce_private_rnd4k_q1t1_read 3:read_bandwidth_kb: 8148.00 :small_red_triangle_up: ( previous job: 8744.00, degradation: 93.18%)
fedora-41-xfce_volatile_seq1m_q8t1_read 3:read_bandwidth_kb: 390531.00 :small_red_triangle_up: ( previous job: 392725.00, degradation: 99.44%)
fedora-41-xfce_volatile_seq1m_q8t1_write 3:write_bandwidth_kb: 144771.00 :green_circle: ( previous job: 139933.00, improvement: 103.46%)
fedora-41-xfce_volatile_seq1m_q1t1_read 3:read_bandwidth_kb: 302619.00 :green_circle: ( previous job: 294875.00, improvement: 102.63%)
fedora-41-xfce_volatile_rnd4k_q32t1_read 3:read_bandwidth_kb: 87349.00 :green_circle: ( previous job: 71108.00, improvement: 122.84%)
fedora-41-xfce_volatile_rnd4k_q32t1_write 3:write_bandwidth_kb: 3925.00 :small_red_triangle_up: ( previous job: 3959.00, degradation: 99.14%)
fedora-41-xfce_volatile_rnd4k_q1t1_read 3:read_bandwidth_kb: 8914.00 :green_circle: ( previous job: 8408.00, improvement: 106.02%)

Create a bunch of files/dirs and check if expected files remained and unexpected did not. QubesOS/qubes-issues#1006

Logging from testvm1 already checked for test failure, but not for testvm2.

marmarek added the openqa-pending label Feb 13, 2025

This was referenced Feb 13, 2025

vmupdate: load plugins in sorted order QubesOS/qubes-core-admin-linux#183

Merged

Add filter for qubes stored on inaccessible storage pools QubesOS/qubes-manager#399

Draft

marmarek force-pushed the tests20250213 branch 2 times, most recently from cd5c644 to 26d0080 Compare February 16, 2025 10:54

This was referenced Feb 16, 2025

Fix updating template switcher progress bar QubesOS/qubes-manager#404

Merged

Do not close FDs 0, 1, or 2 QubesOS/qubes-core-qrexec#186

Open

Improved Icons and UX QubesOS/qubes-desktop-linux-manager#242

Merged

marmarek force-pushed the tests20250213 branch 2 times, most recently from a210046 to 6d51fa9 Compare February 20, 2025 11:59

marmarek force-pushed the tests20250213 branch from 6d51fa9 to 531c821 Compare February 21, 2025 03:47

marmarek force-pushed the tests20250213 branch from 531c821 to e3dfe65 Compare February 27, 2025 09:13

marmarek added 2 commits February 27, 2025 11:04

tests: add test for custom-persist feature

ce5ebaa

Create a bunch of files/dirs and check if expected files remained and unexpected did not. QubesOS/qubes-issues#1006

tests: log .xsession-errors from testvm2 only on failure

d75733d

Logging from testvm1 already checked for test failure, but not for testvm2.

marmarek force-pushed the tests20250213 branch from e3dfe65 to d75733d Compare February 27, 2025 10:04

marmarek merged commit 258f153 into QubesOS:main Mar 3, 2025
3 of 5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tests for few upcoming changes #654

Add tests for few upcoming changes #654

marmarek commented Feb 13, 2025

codecov bot commented Feb 13, 2025 •

edited

Loading

qubesos-bot commented Feb 13, 2025 •

edited

Loading

Add tests for few upcoming changes #654

Add tests for few upcoming changes #654

Conversation

marmarek commented Feb 13, 2025

codecov bot commented Feb 13, 2025 • edited Loading

Codecov Report

qubesos-bot commented Feb 13, 2025 • edited Loading

OpenQA test summary

New failures, excluding unstable

Failed tests

Fixed failures

Unstable tests

Performance Tests

Performance degradation:

Remaining performance tests:

codecov bot commented Feb 13, 2025 •

edited

Loading

qubesos-bot commented Feb 13, 2025 •

edited

Loading