voting: Optimize poll locks

[jamescowens]

This PR implements a new locking scheme for the poll registry and related structures/methods. Closes #2618.

Voting in Gridcoin is a high reliability, high integrity subsystem that is built up from polls that are published by wallet holders in accordance with protocol rules for polls, and votes by walletholders. Polls and voting were completely redone in Fern to address integrity issues and further enhanced in major milestones since then.

One thing that hasn't changed since Fern... due to the complexity of tallying the votes and the desire to conserve memory footprint, the results for polls are built dynamically on demand by either the GUI or rpc. The GUI offers a (re)fresh of the entire polling/results based on a single button push, while the rpc splits up the poll listing and poll results into separate rpc functions, as this is more amenable to scripting. During the GUI poll update, if there are a number of polls that have high participation, the GUI can experience lockups for minutes at a time on slow computers (i.e. a slow processor and/or especially a slow disk, such as a traditional spinning HDD, especially if it is connected via USB). This is because while the poll GUI refresh was implemented on a separate thread, a lock was taken on cs_main for the entire update. This lock eventually becomes blocking to the cs_main taken at regular intervals by the core (driven by the net code), and therefore the core and GUI both block until the poll refresh is done and the lock on cs_main by the poll refresh is released. While this is a simple approach to insuring absolute integrity to the poll refresh, because it locks the entire wallet state, it suffers from poor usability.

There are a number of things that can be done to address these issues and have already been enumerated in #2618. This PR represents the implementation of the 1st improvement.

This PR removes the cs_main lock that was being taken in VotingModel::buildPollTable and instead inverts the locking scheme. A new lock to protect the poll registry and related structures has been implemented, cs_poll_registry. This lock is used for critical structures in the poll registry for polls and voting and also the GUI code that is executed by the refresh thread. This ensures protection of the registry state.

In general it is dangerous to access transaction state from the core without cs_main taken, unless you really know what you are doing. In this instance I have implemented a reorg/fork detection scheme that occurs between the contract handlers, the registry itself, and the buildPollTable GUI method. Note that during the registry traversal for the poll refresh, the cs_poll_registry lock is only taken to generate the sequence for the range loop. The BuildPollItem and associated machinery, instead of taking and maintaining the cs_poll_registry lock, or the cs_main lock, deep within the call structure in the ProcessVoteCandidate, a check of the atomic boolean reorg_occurred_during_reg_traversal is made for each vote before it is processed. This reorg_occurred_during_reg_traversal is in the poll registry, but does not require the cs_poll_registry lock, because it is atomic. It is set by the contract handler virtual function Add and Delete for polls/votes as part of a reorg/fork detection routine. Note that when the ProcessVoteCandidate method encounters that flag as set, it immediately throws InvalidDuetoReorgFork(), a new error, which bubbles up to the try around the BuildPollItem() and immediately aborts the tally. Then a waiting loop is entered that checks at one second intervals to see if the reorg/fork has cleared.

Note that this PR does not address performance issues involving the tallying of votes on slow machines, but instead makes sure that the GUI and core are responsive during this process, which may take minutes on a slow computer, and that the final tally is accurate at least at the height incurred at the start of the poll refresh or higher. For polls and voting to scale to much larger participants we will have to implement a different approach, but this will involve a tradeoff in more memory usage.

Note that static thread safety analysis keywords were implemented here which allow clang to detect thread safety issues. On a related note, a full deadlock analysis due to possible lock order issues has not been conducted yet

[jamescowens]

I am going to have to go back to the drawing board on the locking, because I got a deadlock to occur in testing.

[jamescowens]

I am getting very close on this. I have been testing on my Odroid, doing the poll refresh, which nominally takes about 5 mins. During the most stressful test, I start the poll refresh, then about 150 seconds in start a 1100 block forced reorg while the refresh is going on. The poll traversal correctly detects the reorg, aborts the run, and then waits (with 1 sec intervals) for the reorg to be finished and then restarts. The poll results displayed agree with current production.

[jamescowens]

I am putting this back in draft mode until I have a chance to retest the changes to the reorg detection.

[jamescowens]

Final PR testing

Pressure testing this PR was quite difficult and deserves some documentation in its own right.

Note that the conditions you need here are polls with a lot of votes and a slow computer with a relatively slow disk to ensure that the poll vote tally takes a long time. Given that recently we had a number of polls on mainnet with relatively large vote counts (over 600 for the just the Folding@home poll), this provides an ideal set of conditions for the test when using my Odroid XU4, which is relatively slow, with limited memory and a relatively slow disk. (I am using an NFS mount for the wallet.) Testnet does not have enough votes to make a meaningful test for this PR. Note that mainnet testing for this is safe, because no votes are being cast or polls being created.

The goal of the test is to force a deep reorg at the same time a poll tally refresh is going on in the GUI to force the activation of the abort and wait code and see if it resumes properly and the final displayed results are correct. Note that a force reorganization to a hash that is much lower in the chain will result in less trust than before, and the wallet will then almost immediately self reorganize back to the original height. The issue with this method of testing a reorganization is that it is in two segments. The initial reorg is considered ended when the force reorganization gets to the provided block hash. This resets the global atomic g_reorg_in_progress boolean and ends the GUI wait and the tally proceeds. This is different than an automatic reorg which allows disconnects and reconnects to switch to a block of higher trust in one action. So in an automatic reorg, the GUI poll tally code will wait until the entire reorg is finished and block trust is higher than before. The reason I mention this is that the method of testing here with the reorg in two distinct segments (one manual, the other automatic), will allow the tally to resume earlier than in a totally automatic reorg, and since the tally will "outstrip" the readdition of the contracts via the block connection during the automatic reorg, some votes will be missed as a result. This should be seen as a slightly lower than normal tally result for that first test.

This PR also adds quite a bit of LogPrint based instrumentation to facilitate testing, which is only turned on when the VOTE logging category is enabled.

Testing Procedure

1. Remove the blockchain database and sync to 2795228, which is the end of the Folding@home project listing poll.
2. Initiate a poll refresh in the GUI while the wallet is syncing to head. Verify neither blocks each other.
3. Reorg to 2794220 ceb2e1cd47400f299a403d71f465226984f47cfdbd77b4f886004cca5d47471e during poll refresh (approximately 1000 blocks). This is a much higher than normal reorg depth. (The usual depth of reorgs is 2 blocks, and the biggest we have seen in two years is about 50.)
4. Verify the poll tally aborts and waits for reorg completion. (Remember unlike a totally automatic reorg, this will be when the rollback successfully reaches ceb2e1cd47400f299a403d71f465226984f47cfdbd77b4f886004cca5d47471e .)
5. Verify the poll tally resumes when the reorg rollback is finished. Verity poll tally is not blocked by the core or blocking the core.
6. Check the poll tally results in the GUI. They should be lower than what is reported in a wallet at the current head, since under this scenario some votes will not have been counted.
7. Refresh again after the wallet completes the phase 2 reorg (automatic portion) and finishes syncing to the head of the chain without restarting the wallet. Check the poll tally results are in agreement with other mainnet wallets that did not have a reorg during the GUI poll refresh.
8. Reboot and refresh again and verify poll tally results match

[jamescowens]

The wallet passed the test above.

[div72]

ACK 8441bda.