-
Notifications
You must be signed in to change notification settings - Fork 62
Open
Labels
Test FlakeTests that work. Wait, no. Actually yes. Hang on. Something is broken.Tests that work. Wait, no. Actually yes. Hang on. Something is broken.
Description
As seen in #7731 and #7551, but this seems like a different flake mode? I wouldn't be surprised if this is more closely related to 7731 though.
This test failed in a CI run on #8728:
Log showing the specific test failure:
Excerpt from the log showing the failure:
7104 2025-09-17T22:21:14.076Z stderr ───
7105 2025-09-17T22:21:14.076Z log file: /var/tmp/omicron_tmp/omicron_nexus-7c3777add95cb98d-test_action_failure_can_unwind_no_pantry.12907.0.log
7106 2025-09-17T22:21:14.076Z note: configured to log to "/var/tmp/omicron_tmp/omicron_nexus-7c3777add95cb98d-test_action_failure_can_unwind_no_pantry.12907.0.log"
7107 2025-09-17T22:21:14.076Z DB URL: postgresql://root@[::1]:55473/omicron?sslmode=disable
7108 2025-09-17T22:21:14.076Z DB address: [::1]:55473
7109 2025-09-17T22:21:14.076Z log file: /var/tmp/omicron_tmp/omicron_nexus-7c3777add95cb98d-test_action_failure_can_unwind_no_pantry.12907.2.log
7110 2025-09-17T22:21:14.076Z note: configured to log to "/var/tmp/omicron_tmp/omicron_nexus-7c3777add95cb98d-test_action_failure_can_unwind_no_pantry.12907.2.log"
7111 2025-09-17T22:21:14.076Z
7112 2025-09-17T22:21:14.076Z thread 'app::sagas::snapshot_create::test::test_action_failure_can_unwind_no_pantry' panicked at /work/oxidecomputer/omicron/nexus/test-utils/src/resource_helpers.rs:222:13:
7113 2025-09-17T22:21:14.076Z failed to make "DELETE" request to /v1/disks/disky-mcdiskface?project=springfield-squidport: making request to server
7114 2025-09-17T22:21:14.076Z stack backtrace:
7115 2025-09-17T22:21:14.076Z 0: __rustc::rust_begin_unwind
7116 2025-09-17T22:21:14.076Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:697:5
7117 2025-09-17T22:21:14.076Z 1: core::panicking::panic_fmt
7118 2025-09-17T22:21:14.076Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:75:14
7119 2025-09-17T22:21:14.076Z 2: nexus_test_utils::resource_helpers::object_delete::{{closure}}::{{closure}}
7120 2025-09-17T22:21:14.076Z at ./test-utils/src/resource_helpers.rs:222:13
7121 2025-09-17T22:21:14.076Z 3: core::result::Result<T,E>::unwrap_or_else
7122 2025-09-17T22:21:14.076Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/result.rs:1521:23
7123 2025-09-17T22:21:14.076Z 4: {async_fn#0}
7124 2025-09-17T22:21:14.076Z at ./test-utils/src/resource_helpers.rs:221:10
7125 2025-09-17T22:21:14.077Z 5: {async_fn#0}
7126 2025-09-17T22:21:14.077Z at ./test-utils/src/resource_helpers.rs:543:33
7127 2025-09-17T22:21:14.077Z 6: {async_block#0}
7128 2025-09-17T22:21:14.077Z at ./src/app/sagas/snapshot_create.rs:2383:66
...
Notably this is a failure to make the DELETE request, but the test had been running for >600 seconds at that point and the test Nexus had processed a DELETE a few seconds earlier. So my wild guess is that here and #7731 are us witnessing a timeout getting reached and the test environment coming down around the test runner.
Metadata
Metadata
Assignees
Labels
Test FlakeTests that work. Wait, no. Actually yes. Hang on. Something is broken.Tests that work. Wait, no. Actually yes. Hang on. Something is broken.