les: historical data garbage collection

[rjl493456442]

This PR introduces garbage collection feature for light client. If you want to disable the GC, please add flag --light.nopruning

Now we have several types of data in light client which can be pruned:

• block headers
• bloom bits
• CHT trie
• Bloom trie

Since light client can generate the new cht root and bloom trie root in runtime, so all historical chain data is unnecessary for light client.

If we need to prove something with a pruned headers, light client will fetch it again with latest CHT root which covers all historical headers.

Here are some GC results:

Original client:

+-----------------+------------------------------+------------+
|    DATABASE     |           CATEGORY           |    SIZE    |
+-----------------+------------------------------+------------+
| Key-Value store | Headers                      | 164.75 MiB |
| Key-Value store | Bodies                       | 44.00 B    |
| Key-Value store | Receipts                     | 42.00 B    |
| Key-Value store | Total Difficulty             | 12.45 MiB  |
| Key-Value store | Block <number-hash> pairings | 11.40 MiB  |
| Key-Value store | Block <hash-number> pairings | 10.51 MiB  |
| Key-Value store | Trie nodes                   | 50.88 KiB  |
| Key-Value store | Transaction indexes          | 0.00 B     |
| Key-Value store | Preimages                    | 15.81 KiB  |
| Key-Value store | BloomBits                    | 12.46 MiB  |
| Ancient store   | Headers                      | 6.00 B     |
| Ancient store   | Bodies                       | 6.00 B     |
| Ancient store   | Receipts                     | 6.00 B     |
| Ancient store   | Total Difficulty             | 6.00 B     |
| Ancient store   | Block <number-hash> pairings | 6.00 B     |
| Light client    | CHT trie nodes               | 25.13 MiB  |
| Light client    | Bloom trie nodes             | 25.51 MiB  |
+-----------------+------------------------------+------------+
|                              TOTAL             | 262.46 MIB |
+-----------------+------------------------------+------------+

GCed client

+-----------------+------------------------------+-----------+
|    DATABASE     |           CATEGORY           |   SIZE    |
+-----------------+------------------------------+-----------+
| Key-Value store | Headers                      | 44.19 MiB |
| Key-Value store | Bodies                       | 44.00 B   |
| Key-Value store | Receipts                     | 42.00 B   |
| Key-Value store | Total Difficulty             | 3.34 MiB  |
| Key-Value store | Block <number-hash> pairings | 3.08 MiB  |
| Key-Value store | Block <hash-number> pairings | 2.82 MiB  |
| Key-Value store | Trie nodes                   | 51.51 KiB |
| Key-Value store | Transaction indexes          | 0.00 B    |
| Key-Value store | Preimages                    | 15.81 KiB |
| Key-Value store | BloomBits                    | 3.04 MiB  |
| Ancient store   | Headers                      | 6.00 B    |
| Ancient store   | Bodies                       | 6.00 B    |
| Ancient store   | Receipts                     | 6.00 B    |
| Ancient store   | Total Difficulty             | 6.00 B    |
| Ancient store   | Block <number-hash> pairings | 6.00 B    |
| Light client    | CHT trie nodes               | 3.59 MiB  |
| Light client    | Bloom trie nodes             | 2.57 MiB  |
+-----------------+------------------------------+-----------+
|                              TOTAL             | 62.88 MIB |
+-----------------+------------------------------+-----------+

With this feature, we can control the storage size of light client into a fixed value.

[holiman]

Not sure if I know the codebase enough to review, generally looks good to me

[zsfelfoldi]

Does this mean that every time the client is started you begin deleting old chain data starting from the checkpoint? This feels a bit wasteful.

[zsfelfoldi]

Just an idea: you could change Prune to use a db iterator to find and delete everything in the db from the beginning to the given section instead of iterating with a for loop on every block number of the specific section. This would have multiple advantages:

• you do not need to remember what was pruned already
• it is efficient because chain data db keys are prefixed with block number (this was one of the reasons I did it like this)
• this way you can also prune cached chain data that was ODRed after pruning that section. For example log searching can download many old receipts and it is nice to cache them for a while but it would also be good to throw them away eventually. If you just clean everything before the new section every time a new section is processed then I think it is good enough.

[zsfelfoldi]

This logic does not handle indexer rollbacks. While this is not a frequent case we do handle it in the indexers (might happen on testnets/private nets). Rolling back while pruning is a bit tricky though (you would have to restore the headers of the current unfinished section) so I think it is fine to not be able to roll back properly if pruning is enabled. In this case an error message would be nice at least. And/or automatically reverting to the last stable checkpoint and resyncing from there.

[rjl493456442]

Now I'll keep the latest section in db. It means at least 32768+2048 headers are kept.
Do we really have that deep reorg even in the testnet? Maybe it's possible in the ropsten...

Now we don't have a very good solution to restore all pruned chain data. Rewind HEAD to checkpoint(or geneisis) seems feasible. I may try this approach.

[rjl493456442]

Find a critial issue.
If a light client is pruned, then it will lose all hash->number mapping of headers.
So these backend APIs won't work since the parameters of these APIs are blockhash. We can't retrieve relative headers back via ODR.

• GetReceipts
• GetLogs

[rjl493456442]

@zsfelfoldi I change the code a bit. One important thing is I reserve all hash->number mappings in the database since they are necessary for hash based APIs. From the storage size, the size of 1 section mappings is about 1.4MB. So I think it's totally acceptable.

Please take another look :).

[zsfelfoldi]

See the comments about the iterators (the rest is fine now).

[karalabe]

This iterator implementation seems a bit suboptimal to me. If I call this for mainnet with all blocks present, for from = 8000000 and to = from + 1, you will iterate over 8M entries until you find the starting item. I guess if the use case is light client pruning you can expect all initial keys to be missing, but it still seems suboptimal.

Wouldn't a better solution be to use: start = headerHashKey(from); end = headerHashKey(to)and then simply iterate withNewIteratorWithStart(start)and terminate whenbytes.Compare(key, end) >= 0? We could also swap out NewIteratorWithStarttoNewIteratorWithRange` to make this code even simpler and the iterator even more flexible.

[karalabe]

What's the purpose of this callback?

[rjl493456442]

Because we want to prune CHT and Bloom Trie. Since the trie nodes of the latest section are enough for generating next Merkle root. So I use the callback to collect all trie nodes of the current section and delete all other.

[rjl493456442]

@karalabe Fixed, ptal

[fjl]

Please remove the flag. This feature can just be on by default.