Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Invalid MAC during peer handshake when using static nodes #3323

Closed
siladu opened this issue Jan 25, 2022 · 3 comments
Closed

Invalid MAC during peer handshake when using static nodes #3323

siladu opened this issue Jan 25, 2022 · 3 comments
Labels
bug Something isn't working non mainnet (private networks) not related to mainnet features - covers privacy, permissioning, IBFT2, QBFT P4 Low (ex: Node doesn't start up when the configuration file has unexpected "end-of-line" character) peering Stale

Comments

@siladu
Copy link
Contributor

siladu commented Jan 25, 2022 •
edited
Loading

Starting a single bootnode on a network with some static nodes configured causes Invalid MAC errors.

The errors go away once all the nodes in the static-nodes file are up and running.

What is weird is that the error seems to indicate that the node is receiving a message, even though it's the only node running. This may be because it's trying to peer with itself. This may be related to #3322 and possibly #2689

The same issue occurs whether p2p discovery is enabled or disabled.

Steps to Reproduce with minimal local setup

  1. https://github.com/siladu/besu-duplicate-peers/tree/invalid-mac

Steps to Reproduce Locally

  1. npx quorum-dev-quickstart
  2. Enable DEBUG logging and update .env with BESU_VERSION (I used latest but it's failed on develop as well)
  3. Enable dns in config/besu/config.toml:
...
...
Xdns-enabled=true
Xdns-update-enabled=true
  1. In static-nodes.json, replace IPs with dns hostnames (matching the docker service names), i.e.
[
"enode://8208a3f344695d44e9cf2c023683cbea7b9343e2f70a5e804bd2c93858e945f8f91439eef96a4ab6c47ff06637d6fbe6472f96de1655a1bee57ea896654f3a22@validator1:30303",
"enode://b9050e002aa42464e6b07c811a1f9dfec01249af03f67b753e8415420649b184447bb2a784863ccbf327ad9e31aaba803464979dfe6a7facc669151a5fa6ad1b@validator2:30303",
"enode://59cf0c623c582fa9b19bdf70fb6bade07f4ae32218dd4d1c7e2c7e65acf87da45cf2ab55d16d27360aafef17622c37c09db60d7680ebcc17b78867f4c05bcaa4@validator3:30303",
"enode://2fd5b5b6ad529f55b71602026d1849d0036f06482368b5812fa793014195d3571b0840dbc4175617de2a12db8f1222c012420d471ae5c0d982118625cae58868@validator4:30303",
"enode://86fcc16f4730fbfd238dc17ea552854c0943923bb1d5e886e5601b8d884fb0519060e0023f495dd24ffe60a65660fb7fdcdebfceedd2b3673dfa63658825924b@rpcnode:30303"
]
  1. In docker-compose.yml comment out all services except rpcnode
  2. ./run.sh should bring up rpcnode only and you'll see Invalid MAC in the logs

Steps to Reproduce on AKS (DNS enabled by default using quorum-kubernetes)

  1. Use an existing AKS cluster or provision a new one as per: https://besu.hyperledger.org/en/stable/Tutorials/Kubernetes/Create-Cluster/#azure-aks
  2. Enable DEBUG logging: https://github.com/ConsenSys/quorum-kubernetes/blob/186bef8ffa7331bbdabeb54c25653499b6fb30ba/dev/helm/charts/besu-node/values.yaml#L48
  3. Following: https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/README.md deploy the genesis and a single bootnode:
cd dev/helm/
helm install genesis ./charts/besu-genesis --namespace quorum --create-namespace --values ./values/genesis-besu.yml
helm install bootnode-1 ./charts/besu-node --namespace quorum --values ./values/bootnode.yml
  1. See Invalid MAC in the logs

Versions (Add all that apply)

  • Software version: besu/v21.10.8/linux-x86_64/openjdk-java-11

Additional Information

Static nodes:

[
"enode://0cb3359528b3a5b0282fc5d3d64aa0dd2f8a5807bd932ba9b505ad59ec2d78b3c156caf7e540e7a6cff82b0acdd798d0c30f527ee0c009375ed0f317826c1e00@besu-node-validator-1-0.besu-node-validator-1.quorum.svc.cluster.local:30303?discport=0"
,"enode://054821448263dd09e4107c2eaea45a51c6afb6828899085446ebc210ef17fde33d1506e0157c2ccd07d574895d19761c5b6b6b5293de2f40ce08fa7f031fa971@besu-node-validator-2-0.besu-node-validator-2.quorum.svc.cluster.local:30303?discport=0"
,"enode://be9acba007582b98b6f4f647998c17d4789dfded084d4332d1ac6b081d7a5aa5262586f587be7daae8f239eda47ce9226268e63a98e8b8a5e6f2e513ba351df3@besu-node-validator-3-0.besu-node-validator-3.quorum.svc.cluster.local:30303?discport=0"
,"enode://d5a0676d9ac38c8a2f0ceee5a8747fa319ed868d9d181c23241b4c33fd2bd36f66214226b5f2a056b194732affaaac1cfa5d348706d5df94b7107e03e6991eb5@besu-node-validator-4-0.besu-node-validator-4.quorum.svc.cluster.local:30303?discport=0"
,"enode://5faa950b09016d68d35649627028045379c7cc692c964b89da2840b3a171b552de105c2144b102f33f1fd629fa38a7ee4ded2c73b5b2adfe16da1d56ec70c68f@besu-node-bootnode-1-0.besu-node-bootnode-1.quorum.svc.cluster.local:30303?discport=0"
,"enode://40b8f4cc5ab5b8c28c7248af48e01321b538da79e76aa8ad8320f3fb7f40c4240f79dcf570cf7f41f8c3fc93eac2c144919f97f9b7e4b82fc63d28c6d87a6124@besu-node-bootnode-2-0.besu-node-bootnode-2.quorum.svc.cluster.local:30303?discport=0"
]

Sample log:

2022-01-25 05:56:42.185+00:00 | nioEventLoopGroup-3-10 | DEBUG | HandshakeHandlerOutbound | Wrote initial crypto handshake message to besu-node-bootnode-1-0.besu-node-bootnode-1.quorum.svc.cluster.local/10.0.0.111:30303.
2022-01-25 05:56:42.185+00:00 | nioEventLoopGroup-3-9 | DEBUG | HandshakeHandlerOutbound | Wrote initial crypto handshake message to /127.0.0.1:30303.
2022-01-25 05:56:42.188+00:00 | nioEventLoopGroup-3-2 | DEBUG | AbstractHandshakeHandler | Handshake error:
org.hyperledger.besu.ethereum.p2p.rlpx.handshake.HandshakeException: Decrypting an incoming handshake message failed
	at org.hyperledger.besu.ethereum.p2p.rlpx.handshake.ecies.ECIESHandshaker.handleMessage(ECIESHandshaker.java:211)
	at org.hyperledger.besu.ethereum.p2p.rlpx.connections.netty.HandshakeHandlerInbound.nextHandshakeMessage(HandshakeHandlerInbound.java:60)
	at org.hyperledger.besu.ethereum.p2p.rlpx.connections.netty.AbstractHandshakeHandler.channelRead0(AbstractHandshakeHandler.java:92)
	at org.hyperledger.besu.ethereum.p2p.rlpx.connections.netty.AbstractHandshakeHandler.channelRead0(AbstractHandshakeHandler.java:44)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.bouncycastle.crypto.InvalidCipherTextException: Invalid MAC.
	at org.hyperledger.besu.ethereum.p2p.rlpx.handshake.ecies.ECIESEncryptionEngine.decrypt(ECIESEncryptionEngine.java:277)
	at org.hyperledger.besu.ethereum.p2p.rlpx.handshake.ecies.ECIESEncryptionEngine.decrypt(ECIESEncryptionEngine.java:215)
	at org.hyperledger.besu.ethereum.p2p.rlpx.handshake.ecies.EncryptedMessage.decryptMsgEIP8(EncryptedMessage.java:94)
	at org.hyperledger.besu.ethereum.p2p.rlpx.handshake.ecies.ECIESHandshaker.handleMessage(ECIESHandshaker.java:203)
	... 20 more

besu-node-bootnode-1-0.log
@siladu siladu added the bug Something isn't working label Jan 25, 2022
@siladu
Copy link
Contributor Author

siladu commented Jan 25, 2022 •
edited
Loading

Recreated this locally with quorum-dev-quickstart with only change being DNS enabled.

i.e. update the quickstart static-node.json to be:

[
"enode://8208a3f344695d44e9cf2c023683cbea7b9343e2f70a5e804bd2c93858e945f8f91439eef96a4ab6c47ff06637d6fbe6472f96de1655a1bee57ea896654f3a22@validator1:30303",
"enode://b9050e002aa42464e6b07c811a1f9dfec01249af03f67b753e8415420649b184447bb2a784863ccbf327ad9e31aaba803464979dfe6a7facc669151a5fa6ad1b@validator2:30303",
"enode://59cf0c623c582fa9b19bdf70fb6bade07f4ae32218dd4d1c7e2c7e65acf87da45cf2ab55d16d27360aafef17622c37c09db60d7680ebcc17b78867f4c05bcaa4@validator3:30303",
"enode://2fd5b5b6ad529f55b71602026d1849d0036f06482368b5812fa793014195d3571b0840dbc4175617de2a12db8f1222c012420d471ae5c0d982118625cae58868@validator4:30303",
"enode://86fcc16f4730fbfd238dc17ea552854c0943923bb1d5e886e5601b8d884fb0519060e0023f495dd24ffe60a65660fb7fdcdebfceedd2b3673dfa63658825924b@rpcnode:30303"
]

and add this to ./config/besu/config.toml

Xdns-enabled=true
Xdns-update-enabled=true

then comment out all services in docker-compose except rpcnode and docker-compose up

@siladu siladu changed the title Invalid MAC during peer handshake Invalid MAC during peer handshake when using DNS Jan 27, 2022
@siladu siladu added the P4 Low (ex: Node doesn't start up when the configuration file has unexpected "end-of-line" character) label Jan 27, 2022
@siladu
Copy link
Contributor Author

siladu commented Jan 27, 2022

Have now recreated this with a local setup without DNS so probably not DNS related afterall!

Follow the README on this repo to recreate locally: https://github.com/siladu/besu-duplicate-peers/tree/invalid-mac

@siladu siladu changed the title Invalid MAC during peer handshake when using DNS Invalid MAC during peer handshake when using static nodes Jan 27, 2022
@poupas poupas mentioned this issue Jun 6, 2022
Closed
@ibhagwan ibhagwan mentioned this issue Sep 18, 2022
Closed
@macfarla macfarla added the non mainnet (private networks) not related to mainnet features - covers privacy, permissioning, IBFT2, QBFT label Mar 20, 2023
@macfarla macfarla added the Stale label Dec 6, 2024
@github-actions GitHub Actions
Copy link

This issue was closed because it has been inactive for 14 days since being marked as stale.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working non mainnet (private networks) not related to mainnet features - covers privacy, permissioning, IBFT2, QBFT P4 Low (ex: Node doesn't start up when the configuration file has unexpected "end-of-line" character) peering Stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@macfarla @siladu @MadelineMurray