"Severe Protocol Error" reported by nodes running older substrate/libp2p after portion of network updates

[drewstone]

We recently began updating nodes on our testnet to our new Edgeware client. For some time, the network was partitioned into two groups that still remained in sync, though peers could be identified as running "old" or "new" software. Once we began updating the remaining old nodes under our management, the remaining older nodes all dropped from their peer lists. The culprit seems to be a Severe Protocol Error like the following reported by an old node:

Handler(PeerId("12D3KooWMVj5bwGuB8e3BB7ikvmJGXEEC8MvXWffPeC7NCHpwkRH")) => Severe protocol error: Upgrade(Select(ProtocolError(InvalidMessage)))

Our libp2p version of that old node is:

3.0.8 (old):
name = "libp2p"
version = "0.19.1"
name = "libp2p-core"
version = "0.19.2"

3.1.0 (old):
name = "libp2p"
version = "0.28.1"
name = "libp2p-core"
version = "0.22.1"

3.2.0 (new):
name = "libp2p"
version = "0.33.0"
name = "libp2p-core"
version = "0.26.0"

It seems that new client software is not backwards compatible and we're looking for insight to help our investigation as well as poll for any ideas around mitigation. We are looking to upgrade our network soon but don't want to cause a large amount of slashable events in the case where validators fail to update in a timely manner and are subsequently dropped from connecting to other peers due to a protocol mismatch.

Linking @jnaviask.

[drewstone]

This is our new node:

• https://github.com/hicommonwealth/edgeware-node/tree/develop

These are our old nodes

• https://github.com/hicommonwealth/edgeware-node/tree/v3.1.0
• https://github.com/hicommonwealth/edgeware-node/tree/v3.0.8

Substrate commits

• 3.0.8 https://github.com/hicommonwealth/substrate/tree/57bfa83d2e5d8af08f1034ad065602552a9d6209
• 3.1.0 hicommonwealth/substrate@86bb8c1
• 3.2.0 paritytech/substrate@b2a5019

[drewstone]

In previous versions we had carried over the legacy protocol commits that have since exited Substrate since mid last year. We are not carrying it over in this new update and are trying to get off of any legacy protocol. Is there a way this was handled smoothly for Kusama/Polkadot or recommendations on how we should handle this for Edgeware?

[drewstone]

One of our validators on the testnet who had fallen behind running 3.1.0, and then subsequently updated the node to 3.2.0 shared this set of logs AFTER updating: https://pastebin.com/F3Xc3RMA. Sharing in case this is also helpful.

[tomaka]

One of our validators on the testnet who had fallen behind running 3.1.0, and then subsequently updated the node to 3.2.0 shared this set of logs AFTER updating: https://pastebin.com/F3Xc3RMA. Sharing in case this is also helpful.

You need to add this line when initializing the service: https://github.com/paritytech/polkadot/blob/44b756b0323759e426a793ce18c9d6fe2d1f576b/node/service/src/lib.rs#L573
It is now needed to "connect" the networking to the grandpa voter.

[tomaka]

The so-called legacy protocol is still in the Substrate code base, and is still supposed to work. In theory, even a Substrate from mid-2018 should be able to connect to a recent Substrate. In practice, however, we don't test it, and it might not actually work.

I see that your "old" version is from August. That should in theory make it possible for old nodes to connect to new nodes, but not the other way around.

[drewstone]

@tomaka is there a flag like there used to be that is accessible to the node operator? What we were seeing is that old nodes were rejecting new nodes due to protocol error, right @jnaviask ?