Add more backpressure from full validation to syncing

[teor2345]

Motivation

When Zebra's full validation slows down:

• it uses a lot of RAM and CPU verifying blocks
• any extra verifications are slower because RAM or CPU is fully used
• the syncer keeps downloading and queueing large numbers of blocks for verification, taking more RAM
• when blocks timeout, any uncommitted blocks get cancelled, and they have to be downloaded and verified again

We've partially fixed this issue by decreasing the syncer lookahead in PRs #4662, #4670, #4679.
We also improved halo2 verification speed in PR #4699.

Designs

When we impose backpressure on the syncer, we need to verify the lowest blocks first.
This means that the backpressure has to be implemented in the syncer task.
(After that, the individual block download and verify tasks can run out of order.)

Here is one possible fix:

• when doing checkpoint verification, let the syncer look ahead 400-1200 blocks
• when doing full verification, let the syncer look ahead 1-max_concurrent_block_requests blocks

400 is the minimum for checkpointing, because the maximum checkpoint gap is 400 blocks.
The max_concurrent_block_requests default might need to be tuned for good full verification performance.

Here is where the syncer limit is implemented:

zebra/zebrad/src/components/sync.rs

Lines 425 to 436 in 54efbe9

	while self.downloads.in_flight() > self.lookahead_limit {
	trace!(
	tips.len = self.prospective_tips.len(),
	in_flight = self.downloads.in_flight(),
	lookahead_limit = self.lookahead_limit,
	state_tip = ?self.latest_chain_tip.best_tip_height(),
	"waiting for pending blocks",
	);
	let response = self.downloads.next().await.expect("downloads is nonempty");
	Self::handle_block_response(response)?;

[teor2345]

The halo2 verification performance improvements in PR #4699, and the updated checkpoints in PR #4708 really helped.

CPU and RAM usage are mostly normal, but they occasionally double. There was one block that took a minute to verify, but most of the time verification happens smoothly.

So this change is optional at the moment.

[teor2345]

Actually, I just saw a pretty bad stall.

There were multiple sync failures, with 5 minute gaps between verified blocks, using 10.3 GB RAM and 15 full CPUs.

So I think this might actually be a high priority.

[teor2345]

This bug was blocking my other tasks, so I ended up doing a rough fix.

[conradoplg]

The halo2 verification performance improvements in PR #4699, and the updated checkpoints in PR #4708 really helped.

To be clear AFAIK #4699 doesn't directly speed up halo2 verification, we now need to actually support batched verification

[conradoplg]

The halo2 verification performance improvements in PR #4699, and the updated checkpoints in PR #4708 really helped.

To be clear AFAIK #4699 doesn't directly speed up halo2 verification, we now need to actually support batched verification

Ah nevermind, there was indeed a ~15% improvement in single verification (though we can still improve it a lot by supporting batching)