state sync proof

[turuslan]

Referenced issues

• /state/2 proof=true #1693

Description of the Change

• state proof=true client and server
• decode and encode compact trie proof
• write complete subtrees to db
• ChildPrefix will be reused by TriePruner
• fix PolkadotTrieCursorImpl::seek
• fix MerkleValue (allow empty)
• fix trie encoding (remove wrong Type::ReservedForCompactEncoding)
• add hash arg to OnNodeLoaded
• OnLoad to save nodes/values (state api, light client, simple proof=true server)

Benefits

• verify state hash early
• reduce memory usage
• can sync inconsistent state during trie v1 migration

Possible Drawbacks

• StateSyncRequestFlow must not skip branches if hash exists in db because that hash can come from value (or store nodes and values in db separately)
• removes proof=false client code
• can merge traversal StateProtocolObserverImpl::prove and encoding compactEncode
• maybe StateSyncRequestFlow::known_ cache is useless, if one block state has no (too few) duplicated trie nodes
  • 87198 (3%) of 2845204 nodes are duplicates on polkadot 16435607
    {1: 2718316, 2: 34263, 3: 1941, 4: 1109, 5: 329, 6: 367, 7: 139, 8: 187, 9: 60, 10: 128, 11: 71, 12: 177, 13: 35, 14: 76, 15: 29, 16: 44, 17: 24, 18: 49, 19: 32, 20: 77, 21: 14, 22: 36, 23: 8, 24: 37, 25: 6, 26: 10, 27: 3, 28: 11, 29: 5, 30: 7, 31: 3, 32: 9, 33: 12, 34: 18, 35: 3, 36: 6, 37: 3, 38: 6, 39: 7, 40: 6, 41: 3, 42: 6, 44: 9, 45: 5, 46: 6, 47: 4, 48: 8, 49: 2, 50: 4, 51: 4, 52: 11, 53: 6, 54: 4, 56: 4, 57: 1, 58: 3, 59: 3, 60: 3, 61: 7, 62: 7, 63: 1, 64: 4, 65: 5, 66: 5, 67: 6, 68: 2, 69: 3, 70: 5, 71: 2, 72: 6, 73: 5, 74: 18, 75: 4, 76: 6, 77: 4, 78: 3, 79: 3, 80: 1, 81: 3, 82: 4, 83: 1, 84: 7, 85: 52, 86: 87, 106: 1, 142: 1, 144: 1, 146: 1, 162: 1, 168: 1}
  • 559061 (26%) of 2106932 nodes are duplicates on westend 16775728
    {1: 1539498, 2: 6234, 3: 686, 4: 285, 5: 94, 6: 235, 7: 31, 8: 23, 9: 23, 10: 15, 11: 18, 12: 16, 13: 13, 14: 5, 15: 4, 16: 11, 17: 10, 18: 8, 19: 5, 20: 4, 21: 8, 22: 2, 23: 4, 24: 4, 25: 2, 26: 3, 27: 1, 28: 3, 29: 3, 30: 3, 31: 1, 32: 2, 33: 3, 34: 5, 35: 2, 37: 4, 38: 4, 39: 2, 41: 1, 42: 4, 43: 8, 44: 4, 45: 3, 46: 3, 47: 1, 48: 5, 49: 3, 50: 2, 52: 3, 53: 1, 55: 1, 56: 1, 59: 2, 60: 1, 61: 2, 63: 2, 64: 2, 65: 3, 66: 1, 69: 2, 74: 1, 76: 1, 81: 1, 85: 12, 86: 10, 89: 2, 96: 1, 98: 2, 99: 1, 102: 2, 103: 1, 104: 2, 112: 1, 115: 1, 127: 1, 129: 1, 132: 1, 133: 2, 135: 2, 139: 1, 140: 1, 142: 1, 143: 2, 144: 1, 145: 1, 147: 2, 148: 2, 150: 2, 151: 1, 152: 1, 153: 2, 154: 2, 155: 6, 156: 1, 157: 3, 158: 2, 159: 2, 160: 3, 161: 1, 162: 5, 163: 1, 164: 7, 165: 1, 166: 2, 167: 1, 168: 2, 169: 5, 170: 2, 171: 1, 174: 6, 176: 1, 177: 1, 181: 2, 183: 1, 184: 1, 188: 1, 189: 1, 190: 1, 195: 1, 219: 1, 235: 1, 264: 1, 277: 1, 295: 1, 371: 1, 398: 1, 432: 1, 500: 1, 501: 1, 556: 1, 558: 1, 570: 1, 614: 1, 644: 1, 729: 1, 764: 1, 780: 1, 838: 1, 895: 1, 900: 1, 919: 1, 994: 1, 1014: 1, 1386: 1, 1397: 1, 1542: 1, 1883: 1, 2442: 1, 2486: 1, 2641: 1, 2654: 1, 2833: 1, 4515: 1, 4796: 1, 8541: 1, 10980: 1, 21269: 1, 33639: 1, 408345: 1}

[igor-egorov]

Built & tested 👍

Probably we have to mention --sync Warp options in Kagome's --help output.