panic in crucible::client::DownstairsClient::process_io_completion during rack update #1788
Description
On an upgrade of dogfood, we found a pile of propolis cores left behind as sleds rebooted.
All of the cores had this same stack backtrace:
fffff6ffef08ef62 raise (6) + 22
fffff6ffef068628 abort () + 58
0000000002bf8179 ???????? ()
0000000002bf8169 ???????? ()
0000000002be29dd __rustc::rust_panic () + d
0000000002be2882 std::panicking::rust_panic_with_hook::hb8223ccd7c528a7f () + 22f
0000000002be24a8 std::panicking::begin_panic_handler::{{closure}}::hcec802fa388f0165 () + 98
0000000002bde1f9 ???????? ()
0000000002be213c ???????? ()
0000000002c24bdf ???????? ()
000000000187e55d crucible::client::DownstairsClient::process_io_completion::h774e9f695bd028d9 () + 96d
00000000017919fc crucible::downstairs::Downstairs::process_io_completion::hfd4beb5a9673e22c () + f1c
0000000001838dcf crucible::upstairs::Upstairs::on_client_message::h0d05f243f21facc8 () + bf
0000000001833afd crucible::upstairs::Upstairs::apply::h57a5c23e3793cdae () + 29d
00000000018cfea2 crucible::up_main::_$u7b$$u7b$closure$u7d$$u7d$::h150f4fa6ce5c8862 (.llvm.8401839033468049112) () + 5a2
00000000018c19d5 tokio::runtime::task::core::Core<T,S>::poll::h8198b5a1284cb828 () + 35
00000000018a77eb tokio::runtime::task::harness::Harness<T,S>::poll::h21f4d96260aa377a () + 5b