test failed in CI: test_migration_failed_everyone_died_can_unwind #9237
Description
This test failed on a CI run on #9232:
https://github.com/oxidecomputer/omicron/pull/9232/checks?check_run_id=52936622364
Log showing the specific test failure:
Excerpt from the log showing the failure:
2025-10-16T17:28:32.611Z | FAIL [ 36.930s] omicron-nexus app::sagas::instance_update::test::test_migration_failed_everyone_died_can_unwind
-- | --
8071 | 2025-10-16T17:28:32.611Z | stdout ───
8072 | 2025-10-16T17:28:32.612Z |
8073 | 2025-10-16T17:28:32.612Z | running 1 test
8074 | 2025-10-16T17:28:32.612Z | test app::sagas::instance_update::test::test_migration_failed_everyone_died_can_unwind ... FAILED
8075 | 2025-10-16T17:28:32.612Z |
8076 | 2025-10-16T17:28:32.612Z | failures:
8077 | 2025-10-16T17:28:32.612Z |
8078 | 2025-10-16T17:28:32.612Z | failures:
8079 | 2025-10-16T17:28:32.612Z | app::sagas::instance_update::test::test_migration_failed_everyone_died_can_unwind
8080 | 2025-10-16T17:28:32.612Z |
8081 | 2025-10-16T17:28:32.612Z | test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 184 filtered out; finished in 36.85s
8082 | 2025-10-16T17:28:32.612Z |
8083 | 2025-10-16T17:28:32.612Z | stderr ───
8084 | 2025-10-16T17:28:32.612Z | log file: /var/tmp/omicron_tmp/omicron_nexus-f2fbcdf152a0edc4-test_migration_failed_everyone_died_can_unwind.99156.0.log
8085 | 2025-10-16T17:28:32.612Z | note: configured to log to "/var/tmp/omicron_tmp/omicron_nexus-f2fbcdf152a0edc4-test_migration_failed_everyone_died_can_unwind.99156.0.log"
8086 | 2025-10-16T17:28:32.612Z | DB URL: postgresql://root@[::1]:43055/omicron?sslmode=disable
8087 | 2025-10-16T17:28:32.612Z | DB address: [::1]:43055
8088 | 2025-10-16T17:28:32.612Z | log file: /var/tmp/omicron_tmp/omicron_nexus-f2fbcdf152a0edc4-test_migration_failed_everyone_died_can_unwind.99156.2.log
8089 | 2025-10-16T17:28:32.612Z | note: configured to log to "/var/tmp/omicron_tmp/omicron_nexus-f2fbcdf152a0edc4-test_migration_failed_everyone_died_can_unwind.99156.2.log"
8090 | 2025-10-16T17:28:32.612Z | log file: /var/tmp/omicron_tmp/omicron_nexus-f2fbcdf152a0edc4-test_migration_failed_everyone_died_can_unwind.99156.3.log
8091 | 2025-10-16T17:28:32.612Z | note: configured to log to "/var/tmp/omicron_tmp/omicron_nexus-f2fbcdf152a0edc4-test_migration_failed_everyone_died_can_unwind.99156.3.log"
8092 | 2025-10-16T17:28:32.612Z |
8093 | 2025-10-16T17:28:32.613Z | thread 'app::sagas::instance_update::test::test_migration_failed_everyone_died_can_unwind' panicked at sled-agent/src/sim/collection.rs:274:65:
8094 | 2025-10-16T17:28:32.613Z | called `Result::unwrap()` on an `Err` value: InternalError { internal_message: "Communication Error: error sending request for url (http://127.0.0.1:42319/vmms/c7c7857c-9df6-4c7a-9011-b7b3abaac4e2)" }
8095 | 2025-10-16T17:28:32.613Z | stack backtrace:
8096 | 2025-10-16T17:28:32.613Z | 0: __rustc::rust_begin_unwind
8097 | 2025-10-16T17:28:32.613Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:697:5
8098 | 2025-10-16T17:28:32.613Z | 1: core::panicking::panic_fmt
8099 | 2025-10-16T17:28:32.613Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:75:14
8100 | 2025-10-16T17:28:32.613Z | 2: core::result::unwrap_failed
8101 | 2025-10-16T17:28:32.613Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/result.rs:1761:5
8102 | 2025-10-16T17:28:32.613Z | 3: unwrap<(), omicron_common::api::external::error::Error>
8103 | 2025-10-16T17:28:32.613Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/result.rs:1167:23
8104 | 2025-10-16T17:28:32.613Z | 4: {async_fn#0}<omicron_sled_agent::sim::instance::SimInstance>
8105 | 2025-10-16T17:28:32.613Z | at /work/oxidecomputer/omicron/sled-agent/src/sim/collection.rs:274:65
8106 | 2025-10-16T17:28:32.613Z | 5: {async_fn#0}
8107 | 2025-10-16T17:28:32.613Z | at /work/oxidecomputer/omicron/sled-agent/src/sim/sled_agent.rs:523:58
8108 | 2025-10-16T17:28:32.613Z | 6: {async_fn#0}
8109 | 2025-10-16T17:28:32.613Z | at /work/oxidecomputer/omicron/sled-agent/src/sim/http_entrypoints.rs:836:38
8110 | 2025-10-16T17:28:32.613Z | 7: {async_block#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>, fn(dropshot::handler::RequestContext<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>) -> omicron_sled_agent::sim::http_entrypoints::{impl#1}::from::instance_poke_post::{async_fn_env#0}, omicron_sled_agent::sim::http_entrypoints::{impl#1}::from::instance_poke_post::{async_fn_env#0}, dropshot::handler::HttpResponseUpdatedNoContent, dropshot::error::HttpError, dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>>
8111 | 2025-10-16T17:28:32.614Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.4/src/handler.rs:664:78
8112 | 2025-10-16T17:28:32.614Z | 8: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<http::response::Response<dropshot::body::Body>, dropshot::handler::HandlerError>> + core::marker::Send), alloc::alloc::Global>>
8113 | 2025-10-16T17:28:32.614Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
8114 | 2025-10-16T17:28:32.614Z | 9: {async_block#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>, fn(dropshot::handler::RequestContext<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>) -> omicron_sled_agent::sim::http_entrypoints::{impl#1}::from::instance_poke_post::{async_fn_env#0}, (dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>), dropshot::handler::HttpResponseUpdatedNoContent>
8115 | 2025-10-16T17:28:32.614Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.4/src/handler.rs:793:16
8116 | 2025-10-16T17:28:32.614Z | 10: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<http::response::Response<dropshot::body::Body>, dropshot::handler::HandlerError>> + core::marker::Send), alloc::alloc::Global>>
8117 | 2025-10-16T17:28:32.614Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
8118 | 2025-10-16T17:28:32.614Z | 11: {async_block#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>
8119 | 2025-10-16T17:28:32.614Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.4/src/server.rs:937:69
8120 | 2025-10-16T17:28:32.614Z | 12: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
8121 | 2025-10-16T17:28:32.614Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/core.rs:365:24
8122 | 2025-10-16T17:28:32.614Z | 13: with_mut<tokio::runtime::task::core::Stage<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>>, core::task::poll::Poll<()>, tokio::runtime::task::core::{impl#6}::poll::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>
8123 | 2025-10-16T17:28:32.614Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/loom/std/unsafe_cell.rs:16:9
8124 | 2025-10-16T17:28:32.614Z | 14: poll<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>
8125 | 2025-10-16T17:28:32.614Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/core.rs:354:30
8126 | 2025-10-16T17:28:32.614Z | 15: tokio::runtime::task::harness::poll_future::{{closure}}
8127 | 2025-10-16T17:28:32.614Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/harness.rs:535:30
8128 | 2025-10-16T17:28:32.614Z | 16: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
8129 | 2025-10-16T17:28:32.614Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panic/unwind_safe.rs:272:9
8130 | 2025-10-16T17:28:32.614Z | 17: do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<()>>
8131 | 2025-10-16T17:28:32.614Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:589:40
8132 | 2025-10-16T17:28:32.614Z | 18: __rust_try
8133 | 2025-10-16T17:28:32.614Z | 19: catch_unwind<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>>
8134 | 2025-10-16T17:28:32.614Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:552:19
8135 | 2025-10-16T17:28:32.614Z | 20: catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<()>>
8136 | 2025-10-16T17:28:32.616Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panic.rs:359:14
8137 | 2025-10-16T17:28:32.616Z | 21: tokio::runtime::task::harness::poll_future
8138 | 2025-10-16T17:28:32.616Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/harness.rs:523:18
8139 | 2025-10-16T17:28:32.616Z | 22: tokio::runtime::task::harness::Harness<T,S>::poll_inner
8140 | 2025-10-16T17:28:32.616Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/harness.rs:210:27
8141 | 2025-10-16T17:28:32.616Z | 23: tokio::runtime::task::harness::Harness<T,S>::poll
8142 | 2025-10-16T17:28:32.616Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/harness.rs:155:20
8143 | 2025-10-16T17:28:32.616Z | 24: tokio::runtime::task::raw::poll
8144 | 2025-10-16T17:28:32.616Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/raw.rs:325:13
8145 | 2025-10-16T17:28:32.616Z | 25: tokio::runtime::task::raw::RawTask::poll
8146 | 2025-10-16T17:28:32.616Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/raw.rs:255:18
8147 | 2025-10-16T17:28:32.616Z | 26: tokio::runtime::task::LocalNotified<S>::run
8148 | 2025-10-16T17:28:32.616Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/mod.rs:509:13
8149 | 2025-10-16T17:28:32.616Z | 27: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
8150 | 2025-10-16T17:28:32.616Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:789:30
8151 | 2025-10-16T17:28:32.616Z | 28: with_budget<(), tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure_env#1}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
8152 | 2025-10-16T17:28:32.616Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/task/coop/mod.rs:167:5
8153 | 2025-10-16T17:28:32.617Z | 29: budget<(), tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure_env#1}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
8154 | 2025-10-16T17:28:32.617Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/task/coop/mod.rs:133:5
8155 | 2025-10-16T17:28:32.617Z | 30: {closure#0}<(), tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure_env#1}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
8156 | 2025-10-16T17:28:32.617Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:368:43
8157 | 2025-10-16T17:28:32.617Z | 31: tokio::runtime::scheduler::current_thread::Context::enter
8158 | 2025-10-16T17:28:32.617Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:432:19
8159 | 2025-10-16T17:28:32.617Z | 32: tokio::runtime::scheduler::current_thread::Context::run_task
8160 | 2025-10-16T17:28:32.617Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:368:28
8161 | 2025-10-16T17:28:32.617Z | 33: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
8162 | 2025-10-16T17:28:32.617Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:785:43
8163 | 2025-10-16T17:28:32.617Z | 34: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
8164 | 2025-10-16T17:28:32.617Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:829:68
8165 | 2025-10-16T17:28:32.617Z | 35: tokio::runtime::context::scoped::Scoped<T>::set
8166 | 2025-10-16T17:28:32.617Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context/scoped.rs:40:9
8167 | 2025-10-16T17:28:32.617Z | 36: tokio::runtime::context::set_scheduler::{{closure}}
8168 | 2025-10-16T17:28:32.617Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context.rs:176:38
8169 | 2025-10-16T17:28:32.617Z | 37: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#8}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
8170 | 2025-10-16T17:28:32.622Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:315:12
8171 | 2025-10-16T17:28:32.623Z | 38: std::thread::local::LocalKey<T>::with
8172 | 2025-10-16T17:28:32.623Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:279:20
8173 | 2025-10-16T17:28:32.623Z | 39: tokio::runtime::context::set_scheduler
8174 | 2025-10-16T17:28:32.623Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context.rs:176:17
8175 | 2025-10-16T17:28:32.623Z | 40: tokio::runtime::scheduler::current_thread::CoreGuard::enter
8176 | 2025-10-16T17:28:32.623Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:829:27
8177 | 2025-10-16T17:28:32.623Z | 41: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
8178 | 2025-10-16T17:28:32.623Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:729:24
8179 | 2025-10-16T17:28:32.623Z | 42: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
8180 | 2025-10-16T17:28:32.623Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:200:33
8181 | 2025-10-16T17:28:32.623Z | 43: tokio::runtime::context::runtime::enter_runtime
8182 | 2025-10-16T17:28:32.623Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context/runtime.rs:65:16
8183 | 2025-10-16T17:28:32.623Z | 44: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
8184 | 2025-10-16T17:28:32.623Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:188:9
8185 | 2025-10-16T17:28:32.623Z | 45: tokio::runtime::runtime::Runtime::block_on_inner
8186 | 2025-10-16T17:28:32.623Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/runtime.rs:356:52
8187 | 2025-10-16T17:28:32.623Z | 46: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
8188 | 2025-10-16T17:28:32.623Z | at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/runtime.rs:330:18
8189 | 2025-10-16T17:28:32.623Z | 47: test_migration_failed_everyone_died_can_unwind
8190 | 2025-10-16T17:28:32.623Z | at ./src/app/sagas/instance_update/mod.rs:2254:5
8191 | 2025-10-16T17:28:32.623Z | 48: omicron_nexus::app::sagas::instance_update::test::test_migration_failed_everyone_died_can_unwind::{{closure}}
8192 | 2025-10-16T17:28:32.623Z | at ./src/app/sagas/instance_update/mod.rs:2257:6
8193 | 2025-10-16T17:28:32.623Z | 49: core::ops::function::FnOnce::call_once
8194 | 2025-10-16T17:28:32.623Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
8195 | 2025-10-16T17:28:32.623Z | 50: core::ops::function::FnOnce::call_once
8196 | 2025-10-16T17:28:32.623Z | at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
8197 | 2025-10-16T17:28:32.623Z | note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
8198 | 2025-10-16T17:28:32.623Z |
8199 | 2025-10-16T17:28:32.623Z | thread 'app::sagas::instance_update::test::test_migration_failed_everyone_died_can_unwind' panicked at clients/sled-agent-client/src/lib.rs:378:14:
8200 | 2025-10-16T17:28:32.623Z | instance_finish_transition() failed unexpectedly: reqwest::Error { kind: Request, url: "http://[::1]:43095/vmms/c7c7857c-9df6-4c7a-9011-b7b3abaac4e2/poke", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(IncompleteMessage)) }
8201 | 2025-10-16T17:28:32.623Z | stack backtrace:
2025-10-16T17:28:32.611Z FAIL [ 36.930s] omicron-nexus app::sagas::instance_update::test::test_migration_failed_everyone_died_can_unwind
[8071](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8071) 2025-10-16T17:28:32.611Z stdout ───
[8072](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8072) 2025-10-16T17:28:32.612Z
[8073](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8073) 2025-10-16T17:28:32.612Z running 1 test
[8074](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8074) 2025-10-16T17:28:32.612Z test app::sagas::instance_update::test::test_migration_failed_everyone_died_can_unwind ... FAILED
[8075](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8075) 2025-10-16T17:28:32.612Z
[8076](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8076) 2025-10-16T17:28:32.612Z failures:
[8077](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8077) 2025-10-16T17:28:32.612Z
[8078](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8078) 2025-10-16T17:28:32.612Z failures:
[8079](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8079) 2025-10-16T17:28:32.612Z app::sagas::instance_update::test::test_migration_failed_everyone_died_can_unwind
[8080](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8080) 2025-10-16T17:28:32.612Z
[8081](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8081) 2025-10-16T17:28:32.612Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 184 filtered out; finished in 36.85s
[8082](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8082) 2025-10-16T17:28:32.612Z
[8083](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8083) 2025-10-16T17:28:32.612Z stderr ───
[8084](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8084) 2025-10-16T17:28:32.612Z log file: /var/tmp/omicron_tmp/omicron_nexus-f2fbcdf152a0edc4-test_migration_failed_everyone_died_can_unwind.99156.0.log
[8085](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8085) 2025-10-16T17:28:32.612Z note: configured to log to "/var/tmp/omicron_tmp/omicron_nexus-f2fbcdf152a0edc4-test_migration_failed_everyone_died_can_unwind.99156.0.log"
[8086](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8086) 2025-10-16T17:28:32.612Z DB URL: postgresql://root@[::1]:43055/omicron?sslmode=disable
[8087](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8087) 2025-10-16T17:28:32.612Z DB address: [::1]:43055
[8088](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8088) 2025-10-16T17:28:32.612Z log file: /var/tmp/omicron_tmp/omicron_nexus-f2fbcdf152a0edc4-test_migration_failed_everyone_died_can_unwind.99156.2.log
[8089](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8089) 2025-10-16T17:28:32.612Z note: configured to log to "/var/tmp/omicron_tmp/omicron_nexus-f2fbcdf152a0edc4-test_migration_failed_everyone_died_can_unwind.99156.2.log"
[8090](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8090) 2025-10-16T17:28:32.612Z log file: /var/tmp/omicron_tmp/omicron_nexus-f2fbcdf152a0edc4-test_migration_failed_everyone_died_can_unwind.99156.3.log
[8091](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8091) 2025-10-16T17:28:32.612Z note: configured to log to "/var/tmp/omicron_tmp/omicron_nexus-f2fbcdf152a0edc4-test_migration_failed_everyone_died_can_unwind.99156.3.log"
[8092](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8092) 2025-10-16T17:28:32.612Z
[8093](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8093) 2025-10-16T17:28:32.613Z thread 'app::sagas::instance_update::test::test_migration_failed_everyone_died_can_unwind' panicked at sled-agent/src/sim/collection.rs:274:65:
[8094](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8094) 2025-10-16T17:28:32.613Z called `Result::unwrap()` on an `Err` value: InternalError { internal_message: "Communication Error: error sending request for url (http://127.0.0.1:42319/vmms/c7c7857c-9df6-4c7a-9011-b7b3abaac4e2)" }
[8095](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8095) 2025-10-16T17:28:32.613Z stack backtrace:
[8096](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8096) 2025-10-16T17:28:32.613Z 0: __rustc::rust_begin_unwind
[8097](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8097) 2025-10-16T17:28:32.613Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:697:5
[8098](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8098) 2025-10-16T17:28:32.613Z 1: core::panicking::panic_fmt
[8099](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8099) 2025-10-16T17:28:32.613Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:75:14
[8100](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8100) 2025-10-16T17:28:32.613Z 2: core::result::unwrap_failed
[8101](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8101) 2025-10-16T17:28:32.613Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/result.rs:1761:5
[8102](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8102) 2025-10-16T17:28:32.613Z 3: unwrap<(), omicron_common::api::external::error::Error>
[8103](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8103) 2025-10-16T17:28:32.613Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/result.rs:1167:23
[8104](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8104) 2025-10-16T17:28:32.613Z 4: {async_fn#0}<omicron_sled_agent::sim::instance::SimInstance>
[8105](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8105) 2025-10-16T17:28:32.613Z at /work/oxidecomputer/omicron/sled-agent/src/sim/collection.rs:274:65
[8106](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8106) 2025-10-16T17:28:32.613Z 5: {async_fn#0}
[8107](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8107) 2025-10-16T17:28:32.613Z at /work/oxidecomputer/omicron/sled-agent/src/sim/sled_agent.rs:523:58
[8108](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8108) 2025-10-16T17:28:32.613Z 6: {async_fn#0}
[8109](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8109) 2025-10-16T17:28:32.613Z at /work/oxidecomputer/omicron/sled-agent/src/sim/http_entrypoints.rs:836:38
[8110](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8110) 2025-10-16T17:28:32.613Z 7: {async_block#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>, fn(dropshot::handler::RequestContext<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>) -> omicron_sled_agent::sim::http_entrypoints::{impl#1}::from::instance_poke_post::{async_fn_env#0}, omicron_sled_agent::sim::http_entrypoints::{impl#1}::from::instance_poke_post::{async_fn_env#0}, dropshot::handler::HttpResponseUpdatedNoContent, dropshot::error::HttpError, dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>>
[8111](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8111) 2025-10-16T17:28:32.614Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.4/src/handler.rs:664:78
[8112](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8112) 2025-10-16T17:28:32.614Z 8: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<http::response::Response<dropshot::body::Body>, dropshot::handler::HandlerError>> + core::marker::Send), alloc::alloc::Global>>
[8113](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8113) 2025-10-16T17:28:32.614Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
[8114](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8114) 2025-10-16T17:28:32.614Z 9: {async_block#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>, fn(dropshot::handler::RequestContext<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>) -> omicron_sled_agent::sim::http_entrypoints::{impl#1}::from::instance_poke_post::{async_fn_env#0}, (dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>), dropshot::handler::HttpResponseUpdatedNoContent>
[8115](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8115) 2025-10-16T17:28:32.614Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.4/src/handler.rs:793:16
[8116](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8116) 2025-10-16T17:28:32.614Z 10: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<http::response::Response<dropshot::body::Body>, dropshot::handler::HandlerError>> + core::marker::Send), alloc::alloc::Global>>
[8117](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8117) 2025-10-16T17:28:32.614Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
[8118](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8118) 2025-10-16T17:28:32.614Z 11: {async_block#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>
[8119](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8119) 2025-10-16T17:28:32.614Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.4/src/server.rs:937:69
[8120](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8120) 2025-10-16T17:28:32.614Z 12: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
[8121](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8121) 2025-10-16T17:28:32.614Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/core.rs:365:24
[8122](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8122) 2025-10-16T17:28:32.614Z 13: with_mut<tokio::runtime::task::core::Stage<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>>, core::task::poll::Poll<()>, tokio::runtime::task::core::{impl#6}::poll::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>
[8123](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8123) 2025-10-16T17:28:32.614Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/loom/std/unsafe_cell.rs:16:9
[8124](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8124) 2025-10-16T17:28:32.614Z 14: poll<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>
[8125](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8125) 2025-10-16T17:28:32.614Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/core.rs:354:30
[8126](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8126) 2025-10-16T17:28:32.614Z 15: tokio::runtime::task::harness::poll_future::{{closure}}
[8127](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8127) 2025-10-16T17:28:32.614Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/harness.rs:535:30
[8128](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8128) 2025-10-16T17:28:32.614Z 16: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
[8129](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8129) 2025-10-16T17:28:32.614Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panic/unwind_safe.rs:272:9
[8130](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8130) 2025-10-16T17:28:32.614Z 17: do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<()>>
[8131](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8131) 2025-10-16T17:28:32.614Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:589:40
[8132](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8132) 2025-10-16T17:28:32.614Z 18: __rust_try
[8133](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8133) 2025-10-16T17:28:32.614Z 19: catch_unwind<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>>
[8134](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8134) 2025-10-16T17:28:32.614Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:552:19
[8135](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8135) 2025-10-16T17:28:32.614Z 20: catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<()>>
[8136](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8136) 2025-10-16T17:28:32.616Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panic.rs:359:14
[8137](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8137) 2025-10-16T17:28:32.616Z 21: tokio::runtime::task::harness::poll_future
[8138](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8138) 2025-10-16T17:28:32.616Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/harness.rs:523:18
[8139](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8139) 2025-10-16T17:28:32.616Z 22: tokio::runtime::task::harness::Harness<T,S>::poll_inner
[8140](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8140) 2025-10-16T17:28:32.616Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/harness.rs:210:27
[8141](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8141) 2025-10-16T17:28:32.616Z 23: tokio::runtime::task::harness::Harness<T,S>::poll
[8142](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8142) 2025-10-16T17:28:32.616Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/harness.rs:155:20
[8143](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8143) 2025-10-16T17:28:32.616Z 24: tokio::runtime::task::raw::poll
[8144](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8144) 2025-10-16T17:28:32.616Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/raw.rs:325:13
[8145](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8145) 2025-10-16T17:28:32.616Z 25: tokio::runtime::task::raw::RawTask::poll
[8146](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8146) 2025-10-16T17:28:32.616Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/raw.rs:255:18
[8147](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8147) 2025-10-16T17:28:32.616Z 26: tokio::runtime::task::LocalNotified<S>::run
[8148](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8148) 2025-10-16T17:28:32.616Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/task/mod.rs:509:13
[8149](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8149) 2025-10-16T17:28:32.616Z 27: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
[8150](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8150) 2025-10-16T17:28:32.616Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:789:30
[8151](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8151) 2025-10-16T17:28:32.616Z 28: with_budget<(), tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure_env#1}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
[8152](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8152) 2025-10-16T17:28:32.616Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/task/coop/mod.rs:167:5
[8153](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8153) 2025-10-16T17:28:32.617Z 29: budget<(), tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure_env#1}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
[8154](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8154) 2025-10-16T17:28:32.617Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/task/coop/mod.rs:133:5
[8155](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8155) 2025-10-16T17:28:32.617Z 30: {closure#0}<(), tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure_env#1}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
[8156](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8156) 2025-10-16T17:28:32.617Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:368:43
[8157](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8157) 2025-10-16T17:28:32.617Z 31: tokio::runtime::scheduler::current_thread::Context::enter
[8158](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8158) 2025-10-16T17:28:32.617Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:432:19
[8159](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8159) 2025-10-16T17:28:32.617Z 32: tokio::runtime::scheduler::current_thread::Context::run_task
[8160](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8160) 2025-10-16T17:28:32.617Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:368:28
[8161](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8161) 2025-10-16T17:28:32.617Z 33: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
[8162](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8162) 2025-10-16T17:28:32.617Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:785:43
[8163](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8163) 2025-10-16T17:28:32.617Z 34: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
[8164](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8164) 2025-10-16T17:28:32.617Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:829:68
[8165](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8165) 2025-10-16T17:28:32.617Z 35: tokio::runtime::context::scoped::Scoped<T>::set
[8166](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8166) 2025-10-16T17:28:32.617Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context/scoped.rs:40:9
[8167](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8167) 2025-10-16T17:28:32.617Z 36: tokio::runtime::context::set_scheduler::{{closure}}
[8168](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8168) 2025-10-16T17:28:32.617Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context.rs:176:38
[8169](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8169) 2025-10-16T17:28:32.617Z 37: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#8}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
[8170](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8170) 2025-10-16T17:28:32.622Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:315:12
[8171](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8171) 2025-10-16T17:28:32.623Z 38: std::thread::local::LocalKey<T>::with
[8172](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8172) 2025-10-16T17:28:32.623Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:279:20
[8173](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8173) 2025-10-16T17:28:32.623Z 39: tokio::runtime::context::set_scheduler
[8174](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8174) 2025-10-16T17:28:32.623Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context.rs:176:17
[8175](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8175) 2025-10-16T17:28:32.623Z 40: tokio::runtime::scheduler::current_thread::CoreGuard::enter
[8176](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8176) 2025-10-16T17:28:32.623Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:829:27
[8177](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8177) 2025-10-16T17:28:32.623Z 41: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
[8178](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8178) 2025-10-16T17:28:32.623Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:729:24
[8179](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8179) 2025-10-16T17:28:32.623Z 42: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
[8180](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8180) 2025-10-16T17:28:32.623Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:200:33
[8181](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8181) 2025-10-16T17:28:32.623Z 43: tokio::runtime::context::runtime::enter_runtime
[8182](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8182) 2025-10-16T17:28:32.623Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context/runtime.rs:65:16
[8183](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8183) 2025-10-16T17:28:32.623Z 44: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
[8184](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8184) 2025-10-16T17:28:32.623Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:188:9
[8185](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8185) 2025-10-16T17:28:32.623Z 45: tokio::runtime::runtime::Runtime::block_on_inner
[8186](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8186) 2025-10-16T17:28:32.623Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/runtime.rs:356:52
[8187](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8187) 2025-10-16T17:28:32.623Z 46: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
[8188](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8188) 2025-10-16T17:28:32.623Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/runtime.rs:330:18
[8189](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8189) 2025-10-16T17:28:32.623Z 47: test_migration_failed_everyone_died_can_unwind
[8190](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8190) 2025-10-16T17:28:32.623Z at ./src/app/sagas/instance_update/mod.rs:2254:5
[8191](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8191) 2025-10-16T17:28:32.623Z 48: omicron_nexus::app::sagas::instance_update::test::test_migration_failed_everyone_died_can_unwind::{{closure}}
[8192](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8192) 2025-10-16T17:28:32.623Z at ./src/app/sagas/instance_update/mod.rs:2257:6
[8193](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8193) 2025-10-16T17:28:32.623Z 49: core::ops::function::FnOnce::call_once
[8194](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8194) 2025-10-16T17:28:32.623Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
[8195](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8195) 2025-10-16T17:28:32.623Z 50: core::ops::function::FnOnce::call_once
[8196](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8196) 2025-10-16T17:28:32.623Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
[8197](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8197) 2025-10-16T17:28:32.623Z note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[8198](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8198) 2025-10-16T17:28:32.623Z
[8199](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8199) 2025-10-16T17:28:32.623Z thread 'app::sagas::instance_update::test::test_migration_failed_everyone_died_can_unwind' panicked at clients/sled-agent-client/src/lib.rs:378:14:
[8200](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8200) 2025-10-16T17:28:32.623Z instance_finish_transition() failed unexpectedly: reqwest::Error { kind: Request, url: "http://[::1]:43095/vmms/c7c7857c-9df6-4c7a-9011-b7b3abaac4e2/poke", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(IncompleteMessage)) }
[8201](https://buildomat.eng.oxide.computer/wg/0/details/01K7PYZV4M8C3PTYA3E8HWY1NT/cYnizgJDFU6DUy2gVTHyfKk1XCZentz6dQfe14rqpeU5VnfE/01K7PZ0BXR6ABYXRP6EZ37GNJ4#S8201) 2025-10-16T17:28:32.623Z stack backtrace: