State tree format conversion with the tree overlay method

[gballet]

This EIP describes the tree overlay method to change the state trie format from hexary to binary, as presented in https://ethresear.ch/t/overlay-method-for-hex-bin-tree-conversion/7104

[holiman]

Clarification: does this EIP only concern the account trie, and not storage tries?

[gballet]

That is voluntarily left vague at the moment, because I don't know at this time if both trees are going to be merged (it has been discussed at the 1x workshop and there's an EIP by Vitalik that I haven't found yet).

I currently assume that it will be both the account and storage tries, but the exact shape of things depend on how and when other teams work this out.

[holiman]

Is there really any particular point in having the state writes go to an overlay binary trie? Would it make any difference if we made writes to e.g. an overlay hexary trie?

[gballet]

If the overlay is hexary, then it would have to be converted during the phase 3 merge. Having an overlay binary trie allows the clients to directly switch to binary tries. Then it gives clients time to perform the conversion in the background.

[holiman]

Please flesh this out a bit. binary root hash equals hexary root hash? I understand what you mean, but technically, the roots won't be the same, so it looks odd to me.

[gballet]

What I mean is that the new Header root is replaced with the binary root. I'm trying to follow the yp notation, in spite of unicode missing some of the symbols. Gotta rethink this then 👍

[axic]

I'm trying to follow the yp notation

Ah that clarifies my (not yet asked) question. Can you perhaps make this clear in the beginning of the specification section?

[holiman]

Will N be specified later? Or would it happen implicitly via the next bullet-item?

[gballet]

N still has to be specified. The way I see it, there are two approaches, and I'd like to get some feedback on that:

Approach 1: N is set to roughly 700. This is because a quick estimate based on my tests say it represents about 1 second. This number would be bigger if we use extensions in binary trees, which I will try when I complete the code that does this.

Approach 2: Dynamically calculate N based on the difficulty, to ensure it takes around 1 second.

Any thoughts?

[axic]

Shouldn't this also clarify that:

1. Before every write a read also happens (needed to determine costs).
2. Upon writing into the overlay, it is deleted from the base trie?

I'm not sure if 2) is proposed here or whether it is a good idea or not, but clarity would be nice.

[gballet]

During phase 1, values are not deleted from the base trie, as the base trie is read-only. I'll make it clear 👍

[holiman]

what happens on state deletions, when an account is deleted from the state trie. Do we insert some "temporary" null-value into the trie or do we need to maintain an explicit "deletion" thing?

[holiman]

Alternative variant:
During phase 1 , we don't stop the world, but we stop transactions. The side-effect would be, that the only things that goes into the overlay trie is the changes to the miner coinbases. Thus, the overlay trie will be very small (realistically a handful, theoretically the same as the number of blocks in phase 1).
The result will be that the merging of overlay trie into base trie can happen in one go, and we can skip the move-700-accounts phase altogether.

I think phase 2 is complex, as we need to check conversion-pointer, read from overlay trie, and then read from main trie if the account was deleted from the bintrie due to an earlier write.

This is not 'stop the world', just means that we won't process transactions for (a couple of days? one day? a few hours?), but we still pay miners for work performed.

[holiman]

One more upside with no-tx-in-phase-1 is that we won't have to worry about deletions.
So here's the problem with deletions:

• The hexary trie is 'frozen', so if an account is deleted, we somehow have to place a deleted-marker into the "ephemeral" overlay binary trie.
• However, if an account is created during this phase, and then deleted, it must not get that same deleted-marker.
• There's no precedent about a deleted-marker in tries, and there's no definitions on how to hash it.

So you might run into the solution that we'll have to maintain an additional "deletion-trie", or "deletion-list", which also must be in the blockhash. IT becomes a pretty hairy problem to solve.

So if the only effect after a block is the addition of mining rewards to the miner and any ommer-miners, then that problem goes away.

TLDR; - there are two reasons for not allowing txs during phase 1:

• Phase 2 can be done instantly, in one block
• No deletions need to be managed,

[gballet]

There's no precedent about a deleted-marker in tries, and there's no definitions on how to hash it.

You can use the empty root hash as a deleted marker. You walk the down the trie, and instead of encountering an account, you encounter the empty hash.

So you might run into the solution that we'll have to maintain an additional "deletion-trie", or "deletion-list", which also must be in the blockhash. IT becomes a pretty hairy problem to solve.

I don't believe that a deletion tree is needed, since the deletion is already recorded inside the trie.

(...) it must not get that same deleted-marker.

Could you elaborate on why a different marker is necessary? If you create an account in phase 1, and then delete it in that same phase, the overlay tree would end up with a deleted marker and the behavior is exactly the same: when encountering an empty hash in the overlay trie, don't search the base trie.

I get the rationale for a quasi-instant phase 2. I would still prefer an approach in which you increase the cost of CREATE and SSTORE during phase 1 - you can still execute transactions if you really need to, and you don't have to disable writing.

[holiman]

Could you elaborate on why a different marker is necessary?

No, actually, you're probably right, those could be treated the same way. But what we wind up with is a third trie variant, which functions slightly differently than the other tries.

This third trie variant is IMO kind of hacky: there is no unambiguous representation of an empty trie. It gets a sort of 'memory'. I would probably prefer some extra data structure instead of "abusing" the trie this way, if we were to go with your approach.

[axic]

@gballet when should this be merged as a draft?

[gballet]

@axic it could be merged now.