PoA nodes are no longer transmitting transactions to other (mining)-nodes #9329
Comments
Tbaut
added
F2-bug 🐞
|
Do you have any way to reproduce on a minimal setup for us to reproduce quickly? This would tremendously help :) |
|
Use this docker-compose file https://github.com/settlemint/mintnet/blob/master/docker-compose.yml and a client node will spin up and start syncing. After the sync, you can send a tx to it. The easiest way to send a TX is by running Adding more logging to the container can be done by cloning https://github.com/settlemint/mintnet , changing either the toml or the entrypoint.sh file and modifying the docker-compose file as follows then run |
This is the open web, anything is possible... Your configuration is very open. Be sure to not use things like
I can't think of anything that was changed between 2.0 and 2.0.1 except maybe #9148 which should have a minimal impact.
|
Good point, just making sure it is not setup issue in that respect.
0 transactions, and after adding one, 1 transaction
Blocks usually have 0 or 1 tx per block. |
|
I couldn't reproduce. I've deployed a dummy contract at The logs showed the following: Transaction receipt shows a status of 0x1 and my contract was deployed. |
|
To what rpc endpoint? A locally running node or https://mintnet.settlemint.com? That last one points to a mining node at the moment. |
|
I launched a local node using your Docker and deployed a contract with the above RPC pointed to this local node. |
|
Sorry for the silence, i was on holiday. I'm fairly sure that did not work. The transaction from your log is not found in the chain: https://explorer.mintnet.settlemint.com/tx/0x9dbfcf62c29e48fc086851dcf77b73880d946e11423a470fb17da6baf49b1b30 I tried it myself again (deploying Migrations.sol from Truffle) Doing the same migration on the mining node shows me this on my local docker node So the migration works for sure, the nodes are in connection with each other since it receives both transactions from the mining nodes pool and new blocks. From the node to the mining nodes pool, however, that is a mystery... Nothing appears in the mining node log when adding a tx in the local nodes txqueue. |
|
It feels like it has to be a networking thing, can you advise what logging and level I could use to sho what parity does after receiving the tx and wants to send it on? |
|
We can also just drop the local docker nodes from the discussion for the sake of simplicity, since deploying directly to the mining nodes has the following effect:
The mining nodes only process the transactions sent to themselves. That is why you see a TX to amsterdam, nothing for frankfurt, and again a tx to amsterdam. And the same but reversed if I use frankfurt. |
roderik
changed the title
|
I downgraded everything to v2.0.0, and it did not change the difference. So the exact version is not the issue |
|
It seems weird, i made sure all mining settings were out of the non mining nodes, and as soon as i push in a TX, it starts up the internal miner. It seems like each node wants to mine its own transactions. FYI, the configs: Config non-mining node: https://gist.github.com/roderik/7b4f9ca2d819da534af231268e8750bb Config mining node: https://gist.github.com/roderik/e75b62fdbf6e4f06bbada1ca7d5143fa |
|
After a full day of banging my head reverted everything to v1.11.6 and it is working again. Would love to get to the bottom of this together, so ping me when anyone has the time to go through everything for a few hours. I can provide access to the servers, repos, etc. |
|
Summarizing our conversation on Gitter: The node shows: 2018-08-30 12:52:00 UTC Imported #2059429 0x3bb7…33fe (0 txs, 0.00 Mgas, 3 ms, 0.57 KiB)
ethereum_1 | 2018-08-30 12:52:05 UTC Transaction mined (hash 0x1a91e05ae2397532d96d0d7b50f2ff86fc1e81544a7c78a54f6dbcf03fd33e58)
ethereum_1 | 2018-08-30 12:52:05 UTC Imported #2059430 0xd103…ae9a (1 txs, 0.02 Mgas, 5 ms, 0.67 KiB)I made the Tx manually, so the problem could come from truffle. I suggest you try out manually also. e.g. |
|
Upgraded all the nodes again, and now everything works. 🤯 |
|
If you read the thread, I have never been able to replicate the problem, there is no evidence that the problem indeed comes from Parity Ethereum. @roderik has closed the issue, we're always happy to reopen if you or anyone encounters the problem. In that case adding anything constructive to the discussion is welcome :). |
Actual and Expected behavior with v2.0.0 and below
This setup ran smoothly. Transactions were sent to either the client or edge nodes and processed by the mining nodes every 5 seconds.
Actual behavior after upgrading everything to v2.0.1
Sending a transaction to a client node, will import the TX nicely but does not transmit it to the rest of the nodes. So the miners never pick up on them.
Sending a transaction to the load balancer or an edge node directly, same behavior
Sending a transaction to a mining node, will mine a block with the transaction in it.
Changes made: https://github.com/settlemint/mintnet/compare/2.0.0...master (lots of debugging tries so do not mind the commit mess)
Edge node logs with a lot of tracing enabled: https://gist.github.com/roderik/17897483ca68be04f659e97e224a63c6 (look for tx id 0x9e792fd35d277627c268119113d3168be5eb599417d821e28012fbb2467af594)
Client node logs with less tracing enabled: https://gist.github.com/roderik/ba6078f14d0f412ca478ea0159e0d4cb (look for tx id 0xe48675d399ac2b5d3d9e05ea0f326a3187da13ccf6041a03dc4a4edce7b16f98)
What is weird:
I've got a workaround, I pointed my load balancer to the mining node, but that is not in the least decentralized nor resilient. I've got no further ideas on what it could be, the only thing I have not tried is rolling everything back to 2.0.0, but since I've found no issues relating to the same problem, I guess it's more a config issue on my end than a software issue.
The text was updated successfully, but these errors were encountered: