Skip to content

test failed in CI: test_action_failure_can_unwind_no_pantry #9041

@iximeow

Description

@iximeow

As seen in #7731 and #7551, but this seems like a different flake mode? I wouldn't be surprised if this is more closely related to 7731 though.

This test failed in a CI run on #8728:

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01K5CRKT4BRQZNCGR0YX0T9FNS/gs2sRBxOJ3N0uak0e7vbD0IHYNK1WvQu8YkenbimiD3ZqnN4/01K5CRN247N8KCCMSRXXJP1R9F

Excerpt from the log showing the failure:

7104  2025-09-17T22:21:14.076Z    stderr ─── 
7105  2025-09-17T22:21:14.076Z      log file: /var/tmp/omicron_tmp/omicron_nexus-7c3777add95cb98d-test_action_failure_can_unwind_no_pantry.12907.0.log
7106  2025-09-17T22:21:14.076Z      note: configured to log to "/var/tmp/omicron_tmp/omicron_nexus-7c3777add95cb98d-test_action_failure_can_unwind_no_pantry.12907.0.log"
7107  2025-09-17T22:21:14.076Z      DB URL: postgresql://root@[::1]:55473/omicron?sslmode=disable
7108  2025-09-17T22:21:14.076Z      DB address: [::1]:55473
7109  2025-09-17T22:21:14.076Z      log file: /var/tmp/omicron_tmp/omicron_nexus-7c3777add95cb98d-test_action_failure_can_unwind_no_pantry.12907.2.log
7110  2025-09-17T22:21:14.076Z      note: configured to log to "/var/tmp/omicron_tmp/omicron_nexus-7c3777add95cb98d-test_action_failure_can_unwind_no_pantry.12907.2.log"
7111  2025-09-17T22:21:14.076Z  
7112  2025-09-17T22:21:14.076Z      thread 'app::sagas::snapshot_create::test::test_action_failure_can_unwind_no_pantry' panicked at /work/oxidecomputer/omicron/nexus/test-utils/src/resource_helpers.rs:222:13:
7113  2025-09-17T22:21:14.076Z      failed to make "DELETE" request to /v1/disks/disky-mcdiskface?project=springfield-squidport: making request to server
7114  2025-09-17T22:21:14.076Z      stack backtrace:
7115  2025-09-17T22:21:14.076Z         0: __rustc::rust_begin_unwind
7116  2025-09-17T22:21:14.076Z                   at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:697:5
7117  2025-09-17T22:21:14.076Z         1: core::panicking::panic_fmt
7118  2025-09-17T22:21:14.076Z                   at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:75:14
7119  2025-09-17T22:21:14.076Z         2: nexus_test_utils::resource_helpers::object_delete::{{closure}}::{{closure}}
7120  2025-09-17T22:21:14.076Z                   at ./test-utils/src/resource_helpers.rs:222:13
7121  2025-09-17T22:21:14.076Z         3: core::result::Result<T,E>::unwrap_or_else
7122  2025-09-17T22:21:14.076Z                   at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/result.rs:1521:23
7123  2025-09-17T22:21:14.076Z         4: {async_fn#0}
7124  2025-09-17T22:21:14.076Z                   at ./test-utils/src/resource_helpers.rs:221:10
7125  2025-09-17T22:21:14.077Z         5: {async_fn#0}
7126  2025-09-17T22:21:14.077Z                   at ./test-utils/src/resource_helpers.rs:543:33
7127  2025-09-17T22:21:14.077Z         6: {async_block#0}
7128  2025-09-17T22:21:14.077Z                   at ./src/app/sagas/snapshot_create.rs:2383:66
...

Notably this is a failure to make the DELETE request, but the test had been running for >600 seconds at that point and the test Nexus had processed a DELETE a few seconds earlier. So my wild guess is that here and #7731 are us witnessing a timeout getting reached and the test environment coming down around the test runner.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Test FlakeTests that work. Wait, no. Actually yes. Hang on. Something is broken.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions