Improve the reliability of the zebrad testnet sync tests

[teor2345]

Version

zebra main as of 26 October 2020.

Platform

ZcashFoundation/zebra CI tests.

Description

The zebrad sync tests frequently fail on testnet.

I tried this:

Submitting PR #1206.

I expected to see this happen:

The tests succeeded, like they did on my development machine.

Instead, this happened:

The testnet sync test failed, which happens frequently, and more often than mainnet.

TODO

• Activate Hedge as soon as Zebra starts (from the genesis block download), rather than waiting until the histogram has 20 entries
  • this change improves request latency at genesis, immediately after a restart, and after long network delays
• Improve genesis block download diagnostics at info level
• Check genesis block retry interval / behaviour
  • A new genesis Download Set request is launched if the old request fails, times out, or the syncer genesis depth request happens before the state commits the genesis block
• Check genesis block download timeout
  • It's the same as every other block

Alternatives

• Increase the quick sync test timeout
  • It's currently 30 seconds, which is already a bit too much for a quick test

Follow-Up

• Disable testnet sync tests, unless an env var is specified (open a new ticket, if there is a testnet sync failure after this fix)
• Deploy more zcashd instances on testnet (Deploy more Zebra or zcashd instances on testnet #1222)