[Neo3] Distribute state hash via p2p messages

[igormcoelho]

I think that, since it may not very good to store state hash on block header (#385), we should distribute it via p2p messages (signed by each CN), as soon as consensus nodes process it.
This way, nodes can keep track of current "official" states, while these still can effectively be updated, if bugs are fixed on code (bugs that harm the protocol).

An extra protection on balances can be simply done by wallets adopting this process: neo-project/proposals#97 (Solid State Transfers). It's compatible to both Neo2 and Neo3, easy and secure.

[lock9]

@igormcoelho but where to store this state? If this is not stored in the blockchain, can this state "be trusted"?
Do you think this can be some sort of extra field/messages during the consensus?

[igormcoelho]

This could be used in Alert messages, perhaps, that already exist.

[lock9]

But if we are not replacing the header, are we going to have any improvements with this? Isn't it going to be a "duplicated" feature?

[igormcoelho]

No, not duplicated.. state will only be distributed via p2p messages, in this proposal. I took some time to understand this, but Erik clarified this to me, it's a big advantage to have the bugfix capability on a blockchain. As long as we have a clearly defined protocol, immutability is not affected. But not having any state message is bad for cross-chains, and also user wallets, that have no other ways of knowing RPC is telling the truth. Using this, RPC could redistribute latest state messages, and user could locally re-compute whatever states it wants, with the guarantee given by ellected consensus nodes.

[lock9]

@igormcoelho is there impact in the overall blockchain size? If not, I think this is good to be implemented.

[igormcoelho]

No impact, just p2p distribution. We must see if this impact p2p with too much extra messages... right now it's few consensus, but if we put more and more, perhaps it would be nice to agregate messages as long as they are sent and received.
Example:
Node receives from CN A: state 0x00..05
=>
Node relays: from CN A state 0x00..05

Node receives from CN B: state 0x00..05
=>
Node relays: from CN A,B state 0x00..05 (instead of relaying just CN B)

Since every CN must sign its message, best approach here would be to have aggregated Schnorr sigs, so space is not affected by increasing list sizes. This aggregation is good, because it allows other nodes to receive full packages with a least 2f+1 confirmations, proving that state is correct.

Right now we still have not dozens of nodes, so a simple approach would work already... let's discuss this before release Neo 3 (perhaps with Schnorr already, I don't know). But perhaps, direct retransmission is good already too, we must experiment that in some months.

[lock9]

Seems interesting.
What tasks do we need to make this work? We don't need the Patricia tree for this, do we?
Also, what is this "state" that you are sharing? It is the current state + verified transactions that are waiting to be added to the next block? Can't the blockchain state be defined by it's latest block hash?

[igormcoelho]

For this one we need MPT or any other state trie. In fact, it would be better to put a HashType on protocol, so that we can upgrade this in the future (today is MPT, tomorrow SMT, etc). Since it's not recorded on the blockchain, it's easy to adjust and improve.

[jsolman]

As long as the CN nodes sign the MPT root hashes that end up being distributed over P2P layer by the nodes, this seems like a good idea.

[igormcoelho]

@jsolman just gave a very nice improvement here, on a private talk:

Idea is simple and quite efficient: let's redistribute MPT hashes on next consensus p2p messages, since it must have been processed in at most one block.

There's an open issue on requiring at most one block difference, but I believe this is de fato standard.

[jsolman]

I talked more with @igormcoelho; we could include the MPT hash of the previous block along with the client version in certain types of the existing consensus messages that a node broadcasts. This will allow the MPT hashes to be distributed through the network, where they can be verified and stored in order to support queries through RPC. Yet they won’t actually be included in the blockchain data.

If an updated version of Neo needs to fix some issue that will change MPT hashes, an offline MPT hash file can be distributed that will contain updated MPT hashes signed by CN nodes. Clients can load this when resyncing the chain from an offline chain file.

[jsolman]

Also, some P2P messages will still be needed to request MPT hashes if a node misses them due to being offline for a while or while syncing the chain, but an offline MPT hash sync package can be provided for new nodes coming online to join the network.

[igormcoelho]

Also, some P2P messages will still be needed to request MPT hashes if a node misses them due to being offline for a while or while syncing the chain, but an offline MPT hash sync package can be provided for new nodes coming online to join the network.

In this sense, a regular full-node will use these hashes just for checking... so even if they lose some messages, this is not supposed to change unless a "catastrophic" event has happened. You see, idea is to almost never change past states, only when fork iminent situation would happen due to the lack of a bugfix, this is my view on that.

By the way @jsolman, you will love this one ;) neo-project/proposals#102

[Lichen9618]

Hi @igormcoelho
Any update for state hash p2p messages?
We need it to implement some projects depend on it.
If there is any problems, we can also help on it.

[shargon]

I prefer to store it in the block, (previous state) what do you think about it? we only need to create one message for getting the current state, not all of them