Optimize on-disk deterministic masternode storage to reduce size of e…

…vodb (#3017)

* Implement CompactFull() in CDBWrapper

This allows to compact the whole DB in one go.

* Implement more compact version of CDeterministicMNListDiff

This introduces CDeterministicMNStateDiff which requires to only store
fields on-disk which actually changed.

* Avoid writing mnUniquePropertyMap to disk when storing snapshots

This map can be rebuilt by simply using AddMN for each deserialized MN.

* Implement Serialize/Unserialize in CScript

This allows us to directly use READWRITE() on scripts and removes the need
for the ugly cast to CScriptBase. This commit also changes all Dash specific
uses of CScript to not use the cast.

* Keep track of registeration counts and introduce internalID for masternodes

The "internalId" is simply the number of MNs registered so far when the
new MN is added. It is deterministic and stays the same forever.

* Use internalId as keys in MN list diffs

This reduces the used size on-disk.

* Two simple speedups in MN list diff handling

1. Avoid full compare if dmn or state pointers match in BuildDiff
2. Use std::move when adding diff to listDiff in GetListForBlock

* Implement upgrade code for old CDeterministicMNListDiff format to new format

* Track tipIndex instead of tipHeight/tipBlockHash

* Store and pass around CBlockIndex* instead of block hash and height

This allows us to switch CDeterministicMNManager::GetListForBlock to work
with CBlockIndex.

* Refactor CDeterministicMNManager::GetListForBlock to require CBlockIndex*

Instead of requiring a block hash. This allows us to remove blockHash and
prevBlockHash from CDeterministicMNListDiff without the use of cs_main
locks in GetListForBlock.

* Remove prevBlockHash, blockHash and nHeight from CDeterministicMNListDiff

* Remove access to determinisitcMNManager in CMasternodeMetaMan::ToString()

The deterministic MN manager is not fully initialized yet at the time this
is called, which results in an empty list being returned everytime.

* Better logic to determine if an upgrade is needed

Reuse the "best block" logic to figure out if an upgrade is needed. Also
use it to ensure that older nodes are unable to start after the upgrade
was performed.

* Return null block hash if it was requested with getmnlistdiff

* bump CGovernanceManager::SERIALIZATION_VERSION_STRING

* Check SERIALIZATION_VERSION_STRING before deserializing anything else

* Invoke Clear() before deserializing just to be sure

src/activemasternode.cpp

@@ -134,7 +134,7 @@ void CActiveMasternodeManager::UpdatedBlockTip(const CBlockIndex* pindexNew, con
  if (!deterministicMNManager->IsDIP3Enforced(pindexNew->nHeight)) return;
 
  if (state == MASTERNODE_READY) {
- auto mnList = deterministicMNManager->GetListForBlock(pindexNew->GetBlockHash());
+ auto mnList = deterministicMNManager->GetListForBlock(pindexNew);
  if (!mnList.IsMNValid(mnListEntry->proTxHash)) {
  // MN disappeared from MN list
  state = MASTERNODE_REMOVED;

src/dbwrapper.h

@@ -391,6 +391,11 @@ class CDBWrapper
  pdb->CompactRange(&slKey1, &slKey2);
  }
 
+ void CompactFull() const
+ {
+ pdb->CompactRange(nullptr, nullptr);
+ }
+
 };
 
 template<typename CDBTransaction>