fix(sync): Pause new downloads when Zebra reaches the lookahead limit

[teor2345]

Motivation

We're seeing a lot of AboveLookaheadLimit errors in full syncs. Instead of dropping those blocks, we should wait for them to verify.

Also we might need more time for a full sync.

Designs

Add a watch channel to the downloader, which gets set when the downloader is a long way away from the state tip. Reset the channel when:

• a downloaded block is much closer to the tip, or
• there are no more blocks pending download or verify.

Solution

Syncer fixes:

• Make the syncer pause new downloads when a downloaded block is 3000 blocks ahead of the state tip
  • If the block is 50,000 blocks ahead, still drop it anyway
• Make sure the default lookahead limit includes 2 full checkpoints (1000 blocks)
  • Add a new test config for this
• Allow 2000 extra blocks in the pre-download queue, or the state commit channel (was 4000 extra blocks)

Related fixes:

• Allow 36 hours for a full sync (was 32 hours)
• Only log lookahead issues once at info level for every pause, rather than for every block
  • Closes ci: logs longer than GitHub's limit do not allow to see the actual error in the console #5518

Related cleanups:

• Fix up a syncer test that didn't actually test what it said it did
• Add a getblocktemplate-rpcs alternative to the missing test config file message

Review

Anyone can review this PR, it's a high priority because it fixes errors on main.

I'm doing a full sync locally to check that it works.

Reviewer Checklist

• Will the PR name make sense to users?
  • Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
• Are the PR labels correct?
• Does the code do what the ticket and PR says?
• How do you know it works? Does it have tests?

[codecov]

Codecov Report

Merging #5561 (4e687d7) into main (75f83fc) will decrease coverage by 0.13%.
The diff coverage is 48.10%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5561      +/-   ##
==========================================
- Coverage   78.84%   78.71%   -0.14%     
==========================================
  Files         305      305              
  Lines       38126    38178      +52     
==========================================
- Hits        30061    30050      -11     
- Misses       8065     8128      +63     

[teor2345]

@Mergifyio update

[mergify]

update

✅ Branch has been successfully updated

[mpguerra]

Do we know why this has started happening now?

[teor2345]

Do we know why this has started happening now?

Probably left over from the changes in #4937. We tried tweaking the lookahead limit a few times, but that doesn't seem to help.

[teor2345]

Syncing worked, but it was pausing for a block timeout every few checkpoints.
So I tweaked the syncer to start again when there were only a few blocks left, rather than waiting for zero blocks.

Apart from that, the logging and sync speed look really good.

[teor2345]

There are still some timeouts in the logs, but I'll deal with them in another PR.

[arya2]

This is wonderful, thank you.

[teor2345]

@Mergifyio refresh

[mergify]

refresh

✅ Pull request refreshed

[teor2345]

@Mergifyio refresh

[mergify]

refresh

✅ Pull request refreshed