Node stop mining due to incorrect channel handling

[DarianShawn]

[ibft]

Description

All the validator nodes in devnet stop mining during 2023-03-13T15:50:00 and 2023-03-14T00:11:00. It had not been recovered until restarting.

The validator is yielding lots of channels when notified by blockchain subscription and didn't close it until canceling.
In fact, the context is a different one. So GoRoutine might leaks here, and never closed.

dogechain/consensus/ibft/consensus.go

Line 17 in 6f11a34

func (i *Ibft) runSequence(height uint64) <-chan struct{} {

isValidator = i.isValidSnapshot()

// validator must not be in syncing mode to start a new block
if isValidator && !i.syncer.IsSyncing() {
	sequenceCh = i.runSequence(pending)
}

select {
case <-syncerBlockCh:
	if isValidator {
		i.stopSequence()
		i.logger.Info("canceled sequence", "sequence", pending)
	}

	i.logger.Info("sequence canceled due to new block", "sequence", pending)
case <-sequenceCh:
case <-i.closeCh:
	if isValidator {
		i.stopSequence()
		i.logger.Info("ibft close", "sequence", pending)
	}

	return
}

Your environment

• OS and version
  • CentOS 8
• version of the Dogechain
  • 6f11a34
• branch that causes this issue
  • feat-state-snapshot

Steps to reproduce

• Upgrade the specific version to replace 4 validators and 1 RPC node.
• Send 300-500 multicall contract methods to RPC endpoint.
• After sealing all transactions, within 1-2 hours, the validators will no longer produce blocks.

Logs

Check the reference.
There are repeating logs failed to update submodules in consensus: height=7251395 err="failed to get update lock", then no more synced block triggered, and no block sealed any more.

Proposed sol

not-sealing.log
ution

• Re-combine the IBFT and syncing module, to make it smooth again.

[DarianShawn]

@0xcb9ff9 This is a Top-A issue and we need to fix it before any new releases.

[0xcb9ff9]

The state here is very complicated, first create a temporary patch?

[DarianShawn]

The solution is mainly to preserve the sealed state and prevent re-entering the consensus loop.
Also, we will ignore stale block height notifications since we are already participating in consensus with a higher block height.

It passed system tests and local manually tests with 1-validator and 4-validator network.
It will be tested on DevNet for some corner cases for 1-2 days.

[DarianShawn]

4440033, 766006a attempt to revert to an older available version.

[DarianShawn]

The issue will be closed as it had not be recovered in two weeks.
Please reopen if we find out more. @0xcb9ff9