fix(sync): Temporarily set full verification concurrency to 30 blocks

[teor2345]

Motivation

We're seeing a lot of sync failures in CI and on developer machines, because some large orchard blocks take a long time to verify.

Depends-On: #4752
Close #4715
Close #4729
Close #4650

Solution

Until we implement orchard batching, temporarily limit full verification to 30 blocks in parallel.

Syncer changes:

• Limit the number of blocks submitted to download and verify
  • By the syncer
  • By the inbound service
• Keep unused extra hashes and submit them to the sync downloader later
• Split the checkpoint and full verify concurrency configs
  • Adjust lookahead limits when transitioning to full verification
  • Use a lower concurrency limit during full verification
  • Decrease full verification concurrency to 5 blocks

Related changes:

• Return the maximum checkpoint height from the chain verifier
• Return the verified block height from the sync downloader (actually not needed, I can remove these changes if we want)
• Remove a verbose log

Review

This bug causes a lot of CI workflows to run slow or fail, so it's a high priority.

I'm still testing it locally, it also needs a full sync test.

Reviewer Checklist

• Code implements Specs and Designs
• Existing CI sync tests pass

[codecov]

Codecov Report

Merging #4726 (88e2517) into main (42ef884) will decrease coverage by 0.11%.
The diff coverage is 58.55%.

@@            Coverage Diff             @@
##             main    #4726      +/-   ##
==========================================
- Coverage   78.87%   78.76%   -0.12%     
==========================================
  Files         306      306              
  Lines       37557    37671     +114     
==========================================
+ Hits        29624    29670      +46     
- Misses       7933     8001      +68     

[conradoplg]

FYI I tried this out and my node still gets stuck at height 1719629

[oxarbitrage]

The code looks good to me however i am not sure if it brings too much practical improvements. Teor reported some improvements in the sync using this PR but further (unrelated?) problems a bit further. Conrado reported no benefits.

With that said, feel free to merge.

[teor2345]

I'm running a full sync test here, to check if this solves the remaining syncer stalls:
https://github.com/ZcashFoundation/zebra/actions/runs/2619418857

It should be faster than PR #4752.

[teor2345]

(I removed the batch verifier changes from this PR, I'll submit them in another PR.)

[teor2345]

Using a partly-synced local Zebra instance, I can get Zebra to sync to the tip with:

• fix(verify): Only verify halo2 proofs once per transaction #4752
• fix(sync): Temporarily set full verification concurrency to 30 blocks #4726

I'm running full syncs to confirm, but we might want to merge them anyway, so we can move on with other work.

[gustavovalverde]

Admin merged, as the full sync got stuck and we need the fix anyways.