Fix syncer download order and add sync tests

[teor2345]

Motivation

We want the syncer to download blocks in chain order, as documented.

We want to make sure that the syncer correctly drops blocks that are a long way ahead of the state tip.
This is a follow-up test for PR #3167 for ticket #1389.

Solution

ChainSync bug fixes:

• Download synced blocks in chain order, not arbitrary HashSet order
  • This adds a direct dependency on indexmap (multiple Zebra dependencies already use indexmap)
• Increase the syncer lookahead height limit, because in-order syncs are more efficient
• Spawn the syncer into its own task - closes Spawn the syncer as an async task #3114

Extra Tests:

• Add ChainSync tests for obtain_tips, extend_tips, and wrong block hashes
• Add block too high tests for obtain_tips and extend_tips
• Add ChainSync tests for duplicate FindBlocks response hashes

Make some futures std::marker::Send or std::marker::Sync, so they can be spawned:

• Refactor so that RetryLimit::Future is std::marker::Sync
• Make the ChainSync::sync future std::marker::Send by spawning tips futures

Improve test harnesses:

• Improve MockService failure messages
• Add closure-based responses to the MockService API
• Move MockChainTip to zebra-chain
• Add a MockChainTipSender type alias
• Support MockChainTip in ChainSync and its downloader

Review

I'm not sure who is around to review this PR, maybe @upbqdn?

Reviewer Checklist

• Code implements Specs and Designs
• Tests for Expected Behaviour
• Tests for Errors

[codecov]

Codecov Report

Merging #3168 (508b3ea) into main (4c310c0) will increase coverage by 0.81%.
The diff coverage is 59.63%.

@@            Coverage Diff             @@
##             main    #3168      +/-   ##
==========================================
+ Coverage   77.30%   78.11%   +0.81%     
==========================================
  Files         265      266       +1     
  Lines       31415    31455      +40     
==========================================
+ Hits        24284    24572     +288     
+ Misses       7131     6883     -248     

[teor2345]

+ Coverage   77.18%   77.90%   +0.72%     

This is a really nice test coverage increase.

[conradoplg]

Looks good! The tests are great.

There are two muts that I don't understand why they're needed, but it's not a big deal. I'm approving this and we can take a look later if needed.

[conradoplg]

Why is this needed?

[upbqdn]

If I understand it correctly, those muts make the compiler aware that update_metrics has the sole access to self. This is needed in a multi-threaded environment, but I can't remember why. I had a discussion about this with Teor.

[conradoplg]

Why is this needed?