Interrupt handler does not work when a blocking task is running

[hdevalence]

Scheduling

This usability issue is acceptable for the first stable release, but we should review for lightwalletd.

Version

main

Description

The interrupt handler does not work when the application is busy (i.e., when it is probably needed). This is because the interrupt handler shares priority with the main application future, and futures do cooperative multitasking:

zebra/zebrad/src/components/tokio.rs

Lines 49 to 52 in d4da960

	tokio::select! {
	result = fut => result,
	_ = shutdown() => Ok(()),
	}

Tasks

1. Wait on signals in a separate thread:

• the main application future should be spawned using spawn, and
• the interrupt handler future should use spawn.

2. Then, the select statement should wait on a JoinHandle for the spawned tasks.

We use spawn for the interrupt future so it can be scheduled on any thread. This ensures that the shutdown task will run, even if there are long-running or blocking tasks in other futures.

Alternatives

Get signals as a stream: https://docs.rs/signal-hook-tokio/0.3.0/signal_hook_tokio/

Related Issues

Design and implement graceful shutdown for Zebra #1678

[teor2345]

I've observed a complete hang on Ctrl-C (SIGINT) on Linux, when the sync service was waiting to restart the sync:

Jan 29 13:10:41.867  INFO {zebrad="df399108" net="Test"}:sync: zebrad::components::sync: waiting to restart sync timeout=61s
Jan 29 13:10:43.787  INFO {zebrad="df399108" net="Test"}:peer{addr=127.0.0.1:38233}:msg_as_req{msg=getblocks}:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1263147) response_len=0 chain_tip_height=Height(1263147) stop_height=None intersection_height=Some(Height(1263147))
Jan 29 13:11:42.868  INFO {zebrad="df399108" net="Test"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Jan 29 13:11:42.868  INFO {zebrad="df399108" net="Test"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(1263147) min_locator_height=1263048 locators=[Height(1263147), Height(1263146), Height(1263145), Height(1263143), Height(1263139), Height(1263131), Height(1263115), Height(1263083), Height(1263048)]
Jan 29 13:11:42.868  INFO {zebrad="df399108" net="Test"}:sync:obtain_tips: zebra_network::peer_set::set: network request with no ready peers: finding more peers, waiting for 1 peers to answer requests
Jan 29 13:11:42.869  INFO {zebrad="df399108" net="Test"}:sync: zebrad::components::sync: exhausted prospective tip set
Jan 29 13:11:42.869  INFO {zebrad="df399108" net="Test"}:sync: zebrad::components::sync: waiting to restart sync timeout=61s
Jan 29 13:11:44.788  INFO {zebrad="df399108" net="Test"}:peer{addr=127.0.0.1:38233}:msg_as_req{msg=getblocks}:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1263147) response_len=0 chain_tip_height=Height(1263147) stop_height=None intersection_height=Some(Height(1263147))
^C^C^C

zebrad did not respond within 5 minutes, even though the sync timer was 61 seconds. Perhaps it's blocked on a network task (#1633 or similar).

Backgrounding the task using Ctrl-Z and killing it using kill %1 made it terminate.

Marking this usability issue as medium, because we've seen it in practice, but it seems to be rare.

[teor2345]

We've improved this a lot, and some of the blocking is meant to happen (like database cleanup or ephemeral database deletion).