Fix: reduce chain-liveness poll frequency

[kantai]

Description

This PR reduces the chain-liveness poll frequency. This was driving the issues facing Hiro's testnet miner yesterday.

[codecov]

Codecov Report

Merging #3610 (6ccadda) into master (753aa1a) will increase coverage by 0.46%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #3610      +/-   ##
==========================================
+ Coverage   29.86%   30.33%   +0.46%     
==========================================
  Files         298      298              
  Lines      275064   275069       +5     
==========================================
+ Hits        82143    83435    +1292     
+ Misses     192921   191634    -1287     

Impacted Files	Coverage Δ
testnet/stacks-node/src/config.rs	48.37% <100.00%> (+0.24%)	⬆️
testnet/stacks-node/src/run_loop/neon.rs	83.14% <100.00%> (-0.56%)	⬇️

... and 75 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[jcnelson]

LGTM. I'm curious how this lead to the livelock conditions you were seeing?

[jcnelson]

Also, does this supersede #3609?

[kantai]

Yes, I think #3609 isn't necessary with this patch.

Our testnet miner was configured with:

[miner]
    first_attempt_time_ms = 5000
    subsequent_attempt_time_ms = 15000
    microblock_attempt_time_ms = 5000

This caused the chain-liveness thread to submit a new stack block event or new burn block event to the chains coordinator every ~15 seconds. When the coordinator gets those events, it interrupts the miner regardless of whether or not a new burn or stacks block actually arrived. This was just not giving the miner enough time to mine a block.

I think this PR is still a bandaid for this behavior. I think there's perhaps two other approaches that should be explored:

1. Refactor the miner blocking signal in the chains coordinator to wait until it knows that a block will be processed before interrupting the miner. That has its own kind of concurrency difficulty: the chains coordinator and the miner contend for a lock on the chainstate db (the chains coordinator tries to begin a tx before it checks whether or not there's a block to process).
2. Refactor the RunTenure directive handler to wait_on() the miner blocking signal. If the miner is in the blocked state, then RunTenure should just wait until it unblocks (up to some timeout). Simply losing a RunTenure directive is costly to active mining.

[obycode]

lgtm

[igorsyl]

FYI @jferrant @stjepangolemac

[blockstack-devops]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.