Beacon Processor's Inbound Event Queue Is Unaffected By a Runtime Argument

[JamesCropcho]

Description

The length of the Beacon Processor's inbound event queue (a.k.a. work queue) is unaffected the by the --beacon-processor-work-queue-len runtime argument.

Version

Multiple versions of Lighthouse are affected.

Lighthouse

Output of lighthouse --version

Lighthouse v5.1.0-10a38a8
BLS library: blst-modern
SHA256 hardware acceleration: true
Allocator: jemalloc
Profile: maxperf
Specs: mainnet (true), minimal (false), gnosis (false)

Output of lighthouse --version

Lighthouse v5.0.0-b5bae6e
BLS library: blst-modern
SHA256 hardware acceleration: true
Allocator: jemalloc
Profile: maxperf
Specs: mainnet (true), minimal (false), gnosis (false)

Command Issued for Building

CARGO_INSTALL_EXTRA_FLAGS="--no-default-features" PROFILE=maxperf RUSTFLAGS='-C target-cpu=native' FEATURES="modern,slasher-lmdb,jemalloc" make

Rust

Output of rustc --version

rustc 1.76.0 (07dca489a 2024-02-04)

Present Behaviour

Runtime Configuration

The Beacon Node service is started with a number of runtime arguments. Here are those which are likely to be relevant:

--beacon-processor-work-queue-len      32768
--beacon-processor-reprocess-queue-len 24576 

The following are the relevant runtime parameters for which an argument is not declared (given here for sake of completeness):

--http-enable-beacon-processor
--beacon-processor-max-workers

Output

Lighthouse's log will include the following:

ERRO Attestation queue full                  queue_len: 16384, msg: the system has insufficient resources for load, service: bproc

The log entry suggests that the queue length is 16384 (suspiciously, this is the default value).

Expected Behaviour

I would expect to see log entries like:

ERRO Attestation queue full                  queue_len: 32768, msg: the system has insufficient resources for load, service: bproc

Note that queue_len above matches the declared --beacon-processor-work-queue-len value.

Online Documentation Referenced

Output of $ lighthouse bn --help | grep -i beacon-processor-work-queue-len -A2:

        --beacon-processor-work-queue-len <INTEGER>
            Specifies the length of the inbound event queue. Higher values may prevent messages from being dropped while
            lower values may help protect the node from becoming overwhelmed. [default: 16384]

Steps To Resolve

I do not know.

[michaelsproul]

You're right that the --beacon-processor-work-queue-len doesn't do anything, but it was never intended to be used by users, and was never intended to modify the length of the attestation queue. There are many many queues in LH, and the attestation queue is just one of them, see:

lighthouse/beacon_node/beacon_processor/src/lib.rs

Lines 74 to 196 in 10a38a8

	/// The maximum size of the channel for work events to the `BeaconProcessor`.
	///
	/// Setting this too low will cause consensus messages to be dropped.
	const DEFAULT_MAX_WORK_EVENT_QUEUE_LEN: usize = 16_384;
	/// The maximum size of the channel for idle events to the `BeaconProcessor`.
	///
	/// Setting this too low will prevent new workers from being spawned. It *should* only need to be
	/// set to the CPU count, but we set it high to be safe.
	const MAX_IDLE_QUEUE_LEN: usize = 16_384;
	/// The maximum size of the channel for re-processing work events.
	const DEFAULT_MAX_SCHEDULED_WORK_QUEUE_LEN: usize = 3 * DEFAULT_MAX_WORK_EVENT_QUEUE_LEN / 4;
	/// The maximum number of queued `Attestation` objects that will be stored before we start dropping
	/// them.
	const MAX_UNAGGREGATED_ATTESTATION_QUEUE_LEN: usize = 16_384;
	/// The maximum number of queued `Attestation` objects that will be stored before we start dropping
	/// them.
	const MAX_UNAGGREGATED_ATTESTATION_REPROCESS_QUEUE_LEN: usize = 8_192;
	/// The maximum number of queued `SignedAggregateAndProof` objects that will be stored before we
	/// start dropping them.
	const MAX_AGGREGATED_ATTESTATION_QUEUE_LEN: usize = 4_096;
	/// The maximum number of queued `SignedAggregateAndProof` objects that will be stored before we
	/// start dropping them.
	const MAX_AGGREGATED_ATTESTATION_REPROCESS_QUEUE_LEN: usize = 1_024;
	/// The maximum number of queued `SignedBeaconBlock` objects received on gossip that will be stored
	/// before we start dropping them.
	const MAX_GOSSIP_BLOCK_QUEUE_LEN: usize = 1_024;
	/// The maximum number of queued `BlobSidecar` objects received on gossip that
	/// will be stored before we start dropping them.
	const MAX_GOSSIP_BLOB_QUEUE_LEN: usize = 1_024;
	/// The maximum number of queued `SignedBeaconBlock` objects received prior to their slot (but
	/// within acceptable clock disparity) that will be queued before we start dropping them.
	const MAX_DELAYED_BLOCK_QUEUE_LEN: usize = 1_024;
	/// The maximum number of queued `SignedVoluntaryExit` objects received on gossip that will be stored
	/// before we start dropping them.
	const MAX_GOSSIP_EXIT_QUEUE_LEN: usize = 4_096;
	/// The maximum number of queued `ProposerSlashing` objects received on gossip that will be stored
	/// before we start dropping them.
	const MAX_GOSSIP_PROPOSER_SLASHING_QUEUE_LEN: usize = 4_096;
	/// The maximum number of queued `AttesterSlashing` objects received on gossip that will be stored
	/// before we start dropping them.
	const MAX_GOSSIP_ATTESTER_SLASHING_QUEUE_LEN: usize = 4_096;
	/// The maximum number of queued `LightClientFinalityUpdate` objects received on gossip that will be stored
	/// before we start dropping them.
	const MAX_GOSSIP_FINALITY_UPDATE_QUEUE_LEN: usize = 1_024;
	/// The maximum number of queued `LightClientOptimisticUpdate` objects received on gossip that will be stored
	/// before we start dropping them.
	const MAX_GOSSIP_OPTIMISTIC_UPDATE_QUEUE_LEN: usize = 1_024;
	/// The maximum number of queued `LightClientOptimisticUpdate` objects received on gossip that will be stored
	/// for reprocessing before we start dropping them.
	const MAX_GOSSIP_OPTIMISTIC_UPDATE_REPROCESS_QUEUE_LEN: usize = 128;
	/// The maximum number of queued `SyncCommitteeMessage` objects that will be stored before we start dropping
	/// them.
	const MAX_SYNC_MESSAGE_QUEUE_LEN: usize = 2048;
	/// The maximum number of queued `SignedContributionAndProof` objects that will be stored before we
	/// start dropping them.
	const MAX_SYNC_CONTRIBUTION_QUEUE_LEN: usize = 1024;
	/// The maximum number of queued `SignedBeaconBlock` objects received from the network RPC that
	/// will be stored before we start dropping them.
	const MAX_RPC_BLOCK_QUEUE_LEN: usize = 1_024;
	/// The maximum number of queued `BlobSidecar` objects received from the network RPC that
	/// will be stored before we start dropping them.
	const MAX_RPC_BLOB_QUEUE_LEN: usize = 1_024;
	/// The maximum number of queued `Vec<SignedBeaconBlock>` objects received during syncing that will
	/// be stored before we start dropping them.
	const MAX_CHAIN_SEGMENT_QUEUE_LEN: usize = 64;
	/// The maximum number of queued `StatusMessage` objects received from the network RPC that will be
	/// stored before we start dropping them.
	const MAX_STATUS_QUEUE_LEN: usize = 1_024;
	/// The maximum number of queued `BlocksByRangeRequest` objects received from the network RPC that
	/// will be stored before we start dropping them.
	const MAX_BLOCKS_BY_RANGE_QUEUE_LEN: usize = 1_024;
	/// The maximum number of queued `BlobsByRangeRequest` objects received from the network RPC that
	/// will be stored before we start dropping them.
	const MAX_BLOBS_BY_RANGE_QUEUE_LEN: usize = 1024;
	/// The maximum number of queued `BlocksByRootRequest` objects received from the network RPC that
	/// will be stored before we start dropping them.
	const MAX_BLOCKS_BY_ROOTS_QUEUE_LEN: usize = 1_024;
	/// The maximum number of queued `BlobsByRootRequest` objects received from the network RPC that
	/// will be stored before we start dropping them.
	const MAX_BLOBS_BY_ROOTS_QUEUE_LEN: usize = 1_024;
	/// Maximum number of `SignedBlsToExecutionChange` messages to queue before dropping them.
	///
	/// This value is set high to accommodate the large spike that is expected immediately after Capella
	/// is activated.
	const MAX_BLS_TO_EXECUTION_CHANGE_QUEUE_LEN: usize = 16_384;
	/// The maximum number of queued `LightClientBootstrapRequest` objects received from the network RPC that
	/// will be stored before we start dropping them.
	const MAX_LIGHT_CLIENT_BOOTSTRAP_QUEUE_LEN: usize = 1_024;
	/// The maximum number of priority-0 (highest priority) messages that will be queued before
	/// they begin to be dropped.
	const MAX_API_REQUEST_P0_QUEUE_LEN: usize = 1_024;
	/// The maximum number of priority-1 (second-highest priority) messages that will be queued before
	/// they begin to be dropped.
	const MAX_API_REQUEST_P1_QUEUE_LEN: usize = 1_024;

We are planning to hide, deprecate and remove these flags.

The solution in your case is to allocate more resources (CPU, RAM, I/O), as queueing more messages is usually not a useful behaviour.

[JamesCropcho]

Okay, noted. Thank you for the explanation as well as the advice!

[JamesCropcho]

It seems like the "Attestation queue full" error message is only appearing immediately after the usual set of "Previous epoch attestation(s)…" info messages.

Is it possible that there is simply a sufficient count of attestation messages broadcast to a sufficient peer count, that a length of 16,384 is guaranteed to be momentarily exceeded?

I'm running a 16-core, 3.5 GHz Intel Xeon Ice Lake CPU (with SHA256 extensions and ADX, and built using "maxperf"). My storage is a Provisioned IOPS network drive. This machine is only running lighthouse bn (the other needed machines are physically clustered near it). RAM allocation is a fraction of the total available.

(I am using --subscribe-all-subnets and --import-all-attestations, just FYI.)

Is it possible that the allocation of more CPU, RAM, or I/O resources is not primarily what is needed? My resource/process monitoring suggests that I am not making use of what is already available.

Sorry if it is a crazy idea, but I've got to ask…

[dapplion]

Ethereum mainnet has 977188 active vals, so that's ~30,000 unaggregated attestations per slot max. Maybe we need to review the queue numbers up @paulhauner ?

[paulhauner]

Ethereum mainnet has 977188 active vals, so that's ~30,000 unaggregated attestations per slot max. Maybe we need to review the queue numbers up @paulhauner ?

Totally! Some sort of mechanism to give us a heads up on this would be handy. I'm not sure of the cleanest way to do this, though. It would be nice if it would just alert us devs rather than every user. Adding something to CI which calls an API (beaconcha.in?) could be an option, but it would also be annoying (some random PR would just start failing).

[dapplion]

Why not make those values dynamic on the state size on the node starting? Should be good enough even for someone running the same process for 1 year un-interrupted

[paulhauner]

Why not make those values dynamic on the state size on the node starting? Should be good enough even for someone running the same process for 1 year un-interrupted

Yep, dynamic is definitely ideal. I think it'll be the first dynamic queue in the BeaconProcessor but it's very achievable.

[michaelsproul]

Beacon processor queue sizes are dynamic as of:

• Make beacon processor queue sizes dynamic #5573

The beacon processor's scheduler is still sub-optimal, and I am planning to write-up an issue for that after some more investigation. Will close this issue for now, as the issue identified has been resolved.