Network worker not shutting down

[nflaig]

Describe the bug

The network worker thread is not shutting down in some cases.

Force closing (kill) does not help either because worker does receive signal.

Jul-17 17:08:54.000[]                 info: Synced - slot: 6092144 - head: 0xe00e…5e94 - exec-block: valid(9361504 0x8772…) - finalized: 0x7710…765c:190377 - peers: 51
^CJul-17 17:09:01.307[]                 info: Stopping gracefully, use Ctrl+C again to force process exit
Jul-17 17:09:06.176[]                 info: Synced - slot: 6092145 - head: (slot -1) 0xe00e…5e94 - exec-block: valid(9361504 0x8772…) - finalized: 0x7710…765c:190377 - peers: 50
^CJul-17 17:09:10.028[]                 info: Forcing process exit
^C^C^C^C^C./lodestar: line 7: 2357120 Killed                  node --trace-deprecation --max-old-space-size=4096 ./packages/cli/bin/lodestar.js "$@

It is pretty clear based on observed logs that the issue is the network worker.

"terminating network worker" is logged

lodestar/packages/beacon-node/src/network/core/networkCoreWorkerHandler.ts

Line 129 in 5116493

this.modules.logger.debug("terminating network worker");

but "terminated network worker" is not

lodestar/packages/beacon-node/src/network/core/networkCoreWorkerHandler.ts

Line 131 in 5116493

this.modules.logger.debug("terminated network worker");

I checked the log files, there is nothing that peaks out, likely just the same issue we see on the main thread with libp2p

• Beacon node active handles after close sequence is completed #5642

Can likely be resolved by calling process.exit on worker explicitly or force closing it using another approach.

Expected behavior

Network worker should shut down cleanly and not hang the main process.

Steps to reproduce

Run Lodestar beacon node with --network.useWorker true flag.

Additional context

The problem seems to be with libp2p which in the end must take care of closing all connections / removing tcp listeners. There are several closed but also open issues regarding connections not being closed properly.

• [switch] Prevent dials when switch is not in state STARTED libp2p/js-libp2p#433
• [switch] ensure all connections are closed on switch stop libp2p/js-libp2p#436

This comment libp2p/js-libp2p#436 (comment) summarizes open tasks but there was no progress in a while.

Operating system

Linux

Lodestar version or commit hash

ec81531

[nflaig]

Reopening until we can confirm this is fixed by upgrading libp2p, see #5642 (comment).

[nflaig]

This is still an issue in the latest release as confirmed by seamonkey

[nflaig]

Might be resolved in the next node release, see fix nodejs/node#51526 and another related issue tinylibs/tinypool#54

[wemeetagain]

@nflaig I haven't heard of this happening lately, stale?

[nflaig]

This is kinda related to #6053 as well, I haven't seen this myself in a while which is good because hanging work was a pain to deal with.

Should be observed more closely and can probably close it

we also wanna consider removing the timeout if we don't see it anymore

lodestar/packages/beacon-node/src/network/core/networkCore.ts

Lines 269 to 272 in cf72219

	// In some cases, `libp2p.stop` never resolves, it is required
	// to wrap the call with a timeout to allow for a timely shutdown
	// See https://github.com/ChainSafe/lodestar/issues/6053
	await withTimeout(async () => this.libp2p.stop(), 5000);