Skip to content
This repository has been archived by the owner on Nov 6, 2020. It is now read-only.

Sync gets stuck in parity private network (connectivity problems) #10778

Closed
miguelmtzinf opened this issue Jun 25, 2019 · 8 comments
Closed

Sync gets stuck in parity private network (connectivity problems) #10778

miguelmtzinf opened this issue Jun 25, 2019 · 8 comments
Labels
F2-bug 🐞 The client fails to follow expected behavior. M4-core ⛓ Core client code / Rust. Z0-unconfirmed 🤔 Issue might be valid, but it’s not yet known.

Comments

@miguelmtzinf
Copy link

miguelmtzinf commented Jun 25, 2019 •
edited
Loading

  • Parity Ethereum version: v2.4.7-stable
  • Operating system: Linux
  • Installation: Docker official image parity/parity:stable
  • Fully synchronized: no
  • Network: private network
  • Restarted: yes

I am running a parity private network using docker-compose locally. The network is compound by two nodes (node0 and node1) and using aura consensus with 1 signer (from node0). When node1 looses connectivity with node0 or is restarted, is not able to reconnect with node0. WHen node1 is restarted is not able to resync the new produced blocks (I guess because the connectivity stuff).

Here is the reserved_peers: (both enodes match)

enode://7a859c5d4294136b1a01b604477ac28a5643ee39a59ce44a9588001883bf16aa262f6fcf56f0c527c536484ebf6a6a28b98a2faa2027764ad9949152ac2ef9ec@172.16.0.11:30303
enode://6355b8589ae36fbfdc5f2a3acd1ec6a1ae5841299a9a0132906e73ac3c587bdab5ad6f0353112a3e58f85b91839f14840456345b492dc374f3d6cd92a0361fa2@172.16.0.10:30303

Here is the authority.toml:

[parity]
chain = "/parity/config/chain.json"
mode = "active"
auto_update = "none"
base_path = "/parity/.local/share/io.parity.ethereum"
keys_path = "/parity/keys"
no_persistent_txqueue = true

[rpc]
port = 8545
interface = "0.0.0.0"
cors = ["all"]
hosts = ["all"]
apis = ["web3", "eth", "net", "parity", "traces", "rpc", "personal", "parity_accounts", "signer", "parity_set"]

[network]
reserved_peers = "/parity/config/reserved_peers"
reserved_only = true

[account]
password = ["/parity/authority.pwd"]

[mining]
usd_per_tx = "0"
force_sealing = true
reseal_on_txs = "all"
gas_floor_target = "0"
reseal_min_period = 0

[misc]
logging = "miner=info,own_tx=info,sync=trace,net=trace"

Log with network trace level:

parity_authority1 | Loading config file from /parity/config/authority.toml
parity_authority0 | Loading config file from /parity/config/authority.toml
parity_authority0 | 2019-06-25 10:28:22 UTC main INFO parity_ethereum::run  Starting Parity-Ethereum/v2.4.6-stable-94164e1-20190514/x86_64-linux-gnu/rustc1.34.1
parity_authority0 | 2019-06-25 10:28:22 UTC main INFO parity_ethereum::run  Keys path /parity/keys/parityDockerized
parity_authority0 | 2019-06-25 10:28:22 UTC main INFO parity_ethereum::run  DB path /parity/.local/share/io.parity.ethereum/chains/parityDockerized/db/54db8f336512a42e
parity_authority0 | 2019-06-25 10:28:22 UTC main INFO parity_ethereum::run  State DB configuration: fast
parity_authority0 | 2019-06-25 10:28:22 UTC main INFO parity_ethereum::run  Operating mode: active
parity_authority0 | 2019-06-25 10:28:22 UTC main INFO ethcore_service::service  Configured for parityDockerized using AuthorityRound engine
parity_authority0 | 2019-06-25 10:28:23 UTC main INFO parity_ethereum::run  Running without a persistent transaction queue.
parity_authority0 | 2019-06-25 10:28:23 UTC main DEBUG network  Listening at V4(0.0.0.0:30303)
parity_authority1 | 2019-06-25 10:28:23 UTC main INFO parity_ethereum::run  Starting Parity-Ethereum/v2.4.6-stable-94164e1-20190514/x86_64-linux-gnu/rustc1.34.1
parity_authority1 | 2019-06-25 10:28:23 UTC main INFO parity_ethereum::run  Keys path /parity/keys/parityDockerized
parity_authority1 | 2019-06-25 10:28:23 UTC main INFO parity_ethereum::run  DB path /parity/.local/share/io.parity.ethereum/chains/parityDockerized/db/54db8f336512a42e
parity_authority1 | 2019-06-25 10:28:23 UTC main INFO parity_ethereum::run  State DB configuration: fast
parity_authority1 | 2019-06-25 10:28:23 UTC main INFO parity_ethereum::run  Operating mode: active
parity_authority1 | 2019-06-25 10:28:23 UTC main INFO ethcore_service::service  Configured for parityDockerized using AuthorityRound engine
parity_authority1 | 2019-06-25 10:28:24 UTC main INFO parity_ethereum::run  Running without a persistent transaction queue.
parity_authority1 | 2019-06-25 10:28:24 UTC main DEBUG network  Listening at V4(0.0.0.0:30303)
parity_authority0 | 2019-06-25 10:28:24 UTC IO Worker #2 TRACE network  0x7a85…f9ec: Connecting to V4(172.16.0.11:30303)
parity_authority0 | 2019-06-25 10:28:24 UTC IO Worker #2 TRACE network  0: Initiating session Some(0x7a859c5d4294136b1a01b604477ac28a5643ee39a59ce44a9588001883bf16aa262f6fcf56f0c527c536484ebf6a6a28b98a2faa2027764ad9949152ac2ef9ec)
parity_authority0 | 2019-06-25 10:28:24 UTC IO Worker #2 TRACE network  Sending handshake auth to "172.16.0.11:30303"
parity_authority0 | 2019-06-25 10:28:24 UTC IO Worker #2 TRACE network  0: Sending 307 bytes
parity_authority0 | 2019-06-25 10:28:24 UTC IO Worker #2 TRACE network  Expect to read 210 bytes
parity_authority0 | 2019-06-25 10:28:24 UTC IO Worker #2 DEBUG network  Connecting peers: 0 sessions, 0 pending + 1 started
parity_authority0 | 2019-06-25 10:28:24 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority0 | 2019-06-25 10:28:24 UTC  TRACE network  connection register; token=Token(0)
parity_authority0 | 2019-06-25 10:28:24 UTC IO Worker #1 TRACE network  0: Wrote 307 bytes
parity_authority0 | 2019-06-25 10:28:24 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority0 | 2019-06-25 10:28:25 UTC IO Worker #0 INFO import  Imported #84 0x6139…fadd (0 txs, 0.00 Mgas, 2 ms, 0.57 KiB)
parity_authority1 | 2019-06-25 10:28:25 UTC IO Worker #2 TRACE network  0x6355…1fa2: Connecting to V4(172.16.0.10:30303)
parity_authority1 | 2019-06-25 10:28:25 UTC IO Worker #2 TRACE network  0: Initiating session Some(0x6355b8589ae36fbfdc5f2a3acd1ec6a1ae5841299a9a0132906e73ac3c587bdab5ad6f0353112a3e58f85b91839f14840456345b492dc374f3d6cd92a0361fa2)
parity_authority1 | 2019-06-25 10:28:25 UTC IO Worker #2 TRACE network  Sending handshake auth to "172.16.0.10:30303"
parity_authority1 | 2019-06-25 10:28:25 UTC IO Worker #2 TRACE network  0: Sending 307 bytes
parity_authority1 | 2019-06-25 10:28:25 UTC IO Worker #2 TRACE network  Expect to read 210 bytes
parity_authority1 | 2019-06-25 10:28:25 UTC IO Worker #2 DEBUG network  Connecting peers: 0 sessions, 0 pending + 1 started
parity_authority1 | 2019-06-25 10:28:25 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority1 | 2019-06-25 10:28:25 UTC  TRACE network  connection register; token=Token(0)
parity_authority1 | 2019-06-25 10:28:25 UTC IO Worker #0 TRACE network  0: Wrote 307 bytes
parity_authority1 | 2019-06-25 10:28:25 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority0 | 2019-06-25 10:28:25 UTC IO Worker #2 DEBUG network  Connecting peers: 0 sessions, 1 pending + 0 started
parity_authority1 | 2019-06-25 10:28:26 UTC IO Worker #0 DEBUG network  Connecting peers: 0 sessions, 1 pending + 0 started
parity_authority0 | 2019-06-25 10:28:26 UTC IO Worker #1 DEBUG network  Connecting peers: 0 sessions, 1 pending + 0 started
parity_authority1 | 2019-06-25 10:28:27 UTC IO Worker #0 DEBUG network  Connecting peers: 0 sessions, 1 pending + 0 started
parity_authority0 | 2019-06-25 10:28:27 UTC IO Worker #3 DEBUG network  Connecting peers: 0 sessions, 1 pending + 0 started
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #3 DEBUG network  Connecting peers: 0 sessions, 1 pending + 0 started
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 INFO network  Public node URL: enode://6355b8589ae36fbfdc5f2a3acd1ec6a1ae5841299a9a0132906e73ac3c587bdab5ad6f0353112a3e58f85b91839f14840456345b492dc374f3d6cd92a0361fa2@172.16.0.10:30303
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  Accepting incoming connection
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  1: Initiating session None
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  Expect to read 307 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC  TRACE network  connection register; token=Token(1)
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #0 TRACE network  1: Read 64 of 307 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #0 TRACE network  1: Read 128 of 307 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #0 TRACE network  1: Read 256 of 307 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #0 TRACE network  1: Read 307 of 307 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #0 TRACE network  Received handshake auth from "172.16.0.11:55648"
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #0 TRACE network  Sending handshake ack to "172.16.0.11:55648"
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #0 TRACE network  1: Sending 210 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #0 TRACE network  Expect to read 32 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #0 TRACE network  1: Sending 240 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority0 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority0 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  0: Read 64 of 210 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  0: Read 128 of 210 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  0: Read 210 of 210 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  Received handshake ack from "172.16.0.10:30303"
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  Expect to read 32 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  1: Wrote 210 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  0: Sending 240 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  1: Wrote 240 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  0: Read 32 of 32 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  Expect to read 208 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  0: Read 64 of 208 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #0 DEBUG network  Connecting peers: 0 sessions, 2 pending + 0 started
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  1: Read 32 of 32 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  Expect to read 208 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  1: Read 64 of 208 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  1: Read 128 of 208 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  1: Read 208 of 208 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  Expect to read 32 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 DEBUG network  Hello: Parity-Ethereum/v2.4.6-stable-94164e1-20190514/x86_64-linux-gnu/rustc1.34.1 v5 0x7a85…f9ec [SessionCapabilityInfo { protocol: [101, 116, 104], version: 63, packet_count: 17, id_offset: 16 }, SessionCapabilityInfo { protocol: [112, 97, 114], version: 3, packet_count: 24, id_offset: 33 }, SessionCapabilityInfo { protocol: [112, 105, 112], version: 1, packet_count: 9, id_offset: 57 }]
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  compressed 1 to 3
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  1: Sending 64 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  1: Read 32 of 32 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  Expect to read 32 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  1: Read 32 of 32 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  Expect to read 32 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  compressed 1 to 3
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  1: Sending 64 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  Rejected duplicate connection: 1
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  compressed 2 to 4
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  1: Sending 64 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority0 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority0 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  1: Wrote 64 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  0: Read 128 of 208 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #0 TRACE network  1: Wrote 64 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  0: Read 208 of 208 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  Expect to read 32 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority0 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  1: Wrote 64 bytes
parity_authority0 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority0 | 2019-06-25 10:28:28 UTC  TRACE network  connection deregister; token=1
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #3 DEBUG network  Hello: Parity-Ethereum/v2.4.6-stable-94164e1-20190514/x86_64-linux-gnu/rustc1.34.1 v5 0x6355…1fa2 [SessionCapabilityInfo { protocol: [101, 116, 104], version: 63, packet_count: 17, id_offset: 16 }, SessionCapabilityInfo { protocol: [112, 97, 114], version: 3, packet_count: 24, id_offset: 33 }, SessionCapabilityInfo { protocol: [112, 105, 112], version: 1, packet_count: 9, id_offset: 57 }]
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  compressed 1 to 3
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  0: Sending 64 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority1 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority1 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  compressed 125 to 95
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  0: Sending 144 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  0: Wrote 240 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority1 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  compressed 313 to 291
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  0: Sending 352 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  0: Wrote 64 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority1 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  0: Wrote 144 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  0: Wrote 352 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  0: Read 32 of 32 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  Expect to read 32 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  0: Read 32 of 32 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  Expect to read 32 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  compressed 1 to 3
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  0: Sending 64 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  0: Wrote 64 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  0: Read 32 of 32 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  Expect to read 32 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  0: Read 32 of 32 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #2 TRACE network  Expect to read 32 bytes
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  Hup: 0
parity_authority1 | 2019-06-25 10:28:28 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority1 | 2019-06-25 10:28:28 UTC IO Worker #3 TRACE network  Connection closed: 0
parity_authority1 | 2019-06-25 10:28:28 UTC  TRACE network  connection deregister; token=0
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  0x6355…1fa2: Connecting to V4(172.16.0.10:30303)
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  0: Initiating session Some(0x6355b8589ae36fbfdc5f2a3acd1ec6a1ae5841299a9a0132906e73ac3c587bdab5ad6f0353112a3e58f85b91839f14840456345b492dc374f3d6cd92a0361fa2)
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  Sending handshake auth to "172.16.0.10:30303"
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  Accepting incoming connection
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  1: Initiating session None
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  Expect to read 307 bytes
parity_authority0 | 2019-06-25 10:28:29 UTC  TRACE network  connection register; token=Token(1)
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  1: Read 64 of 307 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  0: Sending 307 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  Expect to read 210 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #3 DEBUG network  Connecting peers: 0 sessions, 0 pending + 1 started
parity_authority1 | 2019-06-25 10:28:29 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority1 | 2019-06-25 10:28:29 UTC  TRACE network  connection register; token=Token(0)
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  0: Wrote 307 bytes
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  1: Read 128 of 307 bytes
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  1: Read 256 of 307 bytes
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  1: Read 307 of 307 bytes
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  Received handshake auth from "172.16.0.11:55650"
parity_authority1 | 2019-06-25 10:28:29 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  Sending handshake ack to "172.16.0.11:55650"
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  1: Sending 210 bytes
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  Expect to read 32 bytes
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  1: Sending 240 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  0: Read 64 of 210 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  0: Read 128 of 210 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  0: Read 210 of 210 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  Received handshake ack from "172.16.0.10:30303"
parity_authority0 | 2019-06-25 10:28:29 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority0 | 2019-06-25 10:28:29 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority0 | 2019-06-25 10:28:29 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  1: Wrote 210 bytes
parity_authority0 | 2019-06-25 10:28:29 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  Expect to read 32 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  0: Sending 240 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  0: Read 32 of 32 bytes
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #2 TRACE network  1: Wrote 240 bytes
parity_authority0 | 2019-06-25 10:28:29 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  Expect to read 208 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  0: Read 64 of 208 bytes
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  1: Read 32 of 32 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  0: Read 128 of 208 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  0: Read 208 of 208 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  Expect to read 32 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 DEBUG network  Hello: Parity-Ethereum/v2.4.6-stable-94164e1-20190514/x86_64-linux-gnu/rustc1.34.1 v5 0x6355…1fa2 [SessionCapabilityInfo { protocol: [101, 116, 104], version: 63, packet_count: 17, id_offset: 16 }, SessionCapabilityInfo { protocol: [112, 97, 114], version: 3, packet_count: 24, id_offset: 33 }, SessionCapabilityInfo { protocol: [112, 105, 112], version: 1, packet_count: 9, id_offset: 57 }]
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  compressed 1 to 3
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  Expect to read 208 bytes
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  1: Read 64 of 208 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  0: Sending 64 bytes
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  1: Read 128 of 208 bytes
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  1: Read 208 of 208 bytes
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  Expect to read 32 bytes
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 DEBUG network  Hello: Parity-Ethereum/v2.4.6-stable-94164e1-20190514/x86_64-linux-gnu/rustc1.34.1 v5 0x7a85…f9ec [SessionCapabilityInfo { protocol: [101, 116, 104], version: 63, packet_count: 17, id_offset: 16 }, SessionCapabilityInfo { protocol: [112, 97, 114], version: 3, packet_count: 24, id_offset: 33 }, SessionCapabilityInfo { protocol: [112, 105, 112], version: 1, packet_count: 9, id_offset: 57 }]
parity_authority1 | 2019-06-25 10:28:29 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  compressed 1 to 3
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  1: Sending 64 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority1 | 2019-06-25 10:28:29 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  Rejected duplicate connection: 1
parity_authority0 | 2019-06-25 10:28:29 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  compressed 2 to 4
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  1: Sending 64 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  0: Wrote 240 bytes
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #2 TRACE network  1: Wrote 64 bytes
parity_authority0 | 2019-06-25 10:28:29 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority0 | 2019-06-25 10:28:29 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  1: Wrote 64 bytes
parity_authority0 | 2019-06-25 10:28:29 UTC  TRACE network  connection reregister; token=Token(1)
parity_authority0 | 2019-06-25 10:28:29 UTC  TRACE network  connection deregister; token=1
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  compressed 125 to 95
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  0: Sending 144 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  compressed 313 to 291
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  0: Sending 352 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority1 | 2019-06-25 10:28:29 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority1 | 2019-06-25 10:28:29 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  0: Wrote 64 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  0: Read 32 of 32 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  Expect to read 32 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  0: Read 32 of 32 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  Expect to read 32 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  compressed 1 to 3
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  0: Sending 64 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  Session write error: 0: Error(Io(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }), State { next_error: None, backtrace: InternalBacktrace { backtrace: Some(stack backtrace:
parity_authority1 |    0:     0x55df5b00119d - <no info>
parity_authority1 |    1:     0x55df5b000562 - <no info>
parity_authority1 |    2:     0x55df5ad59ec8 - <no info>
parity_authority1 |    3:     0x55df5ad5a1f5 - <no info>
parity_authority1 |    4:     0x55df5a5cb763 - <no info>
parity_authority1 |    5:     0x55df5a3e0ce9 - <no info>
parity_authority1 |    6:     0x55df5a3be9e9 - <no info>
parity_authority1 |    7:     0x55df5a3a7958 - <no info>
parity_authority1 |    8:     0x55df5a3d4c31 - <no info>
parity_authority1 |    9:     0x55df5a3ddbfc - <no info>
parity_authority1 |   10:     0x55df5a3a0b6c - <no info>
parity_authority1 |   11:     0x55df5a39c712 - <no info>
parity_authority1 |   12:     0x55df5a3ef6b6 - <no info>
parity_authority1 |   13:     0x55df5a39c18a - <no info>
parity_authority1 |   14:     0x55df5a39cdd3 - <no info>
parity_authority1 |   15:     0x55df5a39caaa - <no info>
parity_authority1 |   16:     0x55df5a39c4c6 - <no info>
parity_authority1 |   17:     0x55df5a3d56d3 - <no info>
parity_authority1 |   18:     0x55df5a3a8b26 - <no info>
parity_authority1 |   19:     0x55df5b079ed9 - <no info>
parity_authority1 |   20:     0x55df5a3dba27 - <no info>
parity_authority1 |   21:     0x55df5b07921d - <no info>
parity_authority1 |   22:     0x7f44108f66b9 - <no info>
parity_authority1 |   23:     0x7f441041641c - <no info>
parity_authority1 |   24:                0x0 - <no info>) } })
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  0: Read 32 of 32 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  Expect to read 32 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  0: Read 32 of 32 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  Expect to read 32 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 DEBUG network  Disconnected: 0: DuplicatePeer
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 TRACE network  Session read error: 0:Some(0x6355b8589ae36fbfdc5f2a3acd1ec6a1ae5841299a9a0132906e73ac3c587bdab5ad6f0353112a3e58f85b91839f14840456345b492dc374f3d6cd92a0361fa2) (Err(Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" })) Error(Disconnect(DuplicatePeer), State { next_error: None, backtrace: InternalBacktrace { backtrace: Some(stack backtrace:
parity_authority1 |    0:     0x55df5b00119d - <no info>
parity_authority1 |    1:     0x55df5b000562 - <no info>
parity_authority1 |    2:     0x55df5ad59ec8 - <no info>
parity_authority1 |    3:     0x55df5ad5a1f5 - <no info>
parity_authority1 |    4:     0x55df5a5cbd72 - <no info>
parity_authority1 |    5:     0x55df5a3f2311 - <no info>
parity_authority1 |    6:     0x55df5a3bae67 - <no info>
parity_authority1 |    7:     0x55df5a3a79b2 - <no info>
parity_authority1 |    8:     0x55df5a3d4c31 - <no info>
parity_authority1 |    9:     0x55df5a3ddbfc - <no info>
parity_authority1 |   10:     0x55df5a3a0b6c - <no info>
parity_authority1 |   11:     0x55df5a39c712 - <no info>
parity_authority1 |   12:     0x55df5a3ef6b6 - <no info>
parity_authority1 |   13:     0x55df5a39c18a - <no info>
parity_authority1 |   14:     0x55df5a39cdd3 - <no info>
parity_authority1 |   15:     0x55df5a39caaa - <no info>
parity_authority1 |   16:     0x55df5a39c4c6 - <no info>
parity_authority1 |   17:     0x55df5a3d56d3 - <no info>
parity_authority1 |   18:     0x55df5a3a8b26 - <no info>
parity_authority1 |   19:     0x55df5b079ed9 - <no info>
parity_authority1 |   20:     0x55df5a3dba27 - <no info>
parity_authority1 |   21:     0x55df5b07921d - <no info>
parity_authority1 |   22:     0x7f44108f66b9 - <no info>
parity_authority1 |   23:     0x7f441041641c - <no info>
parity_authority1 |   24:                0x0 - <no info>) } })
parity_authority1 | 2019-06-25 10:28:29 UTC  TRACE network  connection reregister; token=Token(0)
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  Session write error: 0: Error(Io(Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }), State { next_error: None, backtrace: InternalBacktrace { backtrace: Some(stack backtrace:
parity_authority1 |    0:     0x55df5b00119d - <no info>
parity_authority1 |    1:     0x55df5b000562 - <no info>
parity_authority1 |    2:     0x55df5ad59ec8 - <no info>
parity_authority1 |    3:     0x55df5ad5a1f5 - <no info>
parity_authority1 |    4:     0x55df5a5cb763 - <no info>
parity_authority1 |    5:     0x55df5a3e0ce9 - <no info>
parity_authority1 |    6:     0x55df5a3be9e9 - <no info>
parity_authority1 |    7:     0x55df5a3a7958 - <no info>
parity_authority1 |    8:     0x55df5a3d4c31 - <no info>
parity_authority1 |    9:     0x55df5a3ddbfc - <no info>
parity_authority1 |   10:     0x55df5a3a0b6c - <no info>
parity_authority1 |   11:     0x55df5a39c712 - <no info>
parity_authority1 |   12:     0x55df5a3ef6b6 - <no info>
parity_authority1 |   13:     0x55df5a39c18a - <no info>
parity_authority1 |   14:     0x55df5a39cdd3 - <no info>
parity_authority1 |   15:     0x55df5a39caaa - <no info>
parity_authority1 |   16:     0x55df5a39c4c6 - <no info>
parity_authority1 |   17:     0x55df5a3d56d3 - <no info>
parity_authority1 |   18:     0x55df5a3a8b26 - <no info>
parity_authority1 |   19:     0x55df5b079ed9 - <no info>
parity_authority1 |   20:     0x55df5a3dba27 - <no info>
parity_authority1 |   21:     0x55df5b07921d - <no info>
parity_authority1 |   22:     0x7f44108f66b9 - <no info>
parity_authority1 |   23:     0x7f441041641c - <no info>
parity_authority1 |   24:                0x0 - <no info>) } })
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  Hup: 0
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  Connection closed: 0
parity_authority1 | 2019-06-25 10:28:29 UTC  TRACE network  connection deregister; token=0
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #2 DEBUG network  Connecting peers: 0 sessions, 1 pending + 0 started
parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network  Connection timeout: 0
parity_authority0 | 2019-06-25 10:28:29 UTC  TRACE network  connection deregister; token=0
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #0 INFO network  Public node URL: enode://7a859c5d4294136b1a01b604477ac28a5643ee39a59ce44a9588001883bf16aa262f6fcf56f0c527c536484ebf6a6a28b98a2faa2027764ad9949152ac2ef9ec@172.16.0.11:30303
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #2 TRACE network  Accepting incoming connection
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #2 TRACE network  0: Initiating session None
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #2 TRACE network  Expect to read 307 bytes
parity_authority1 | 2019-06-25 10:28:29 UTC  TRACE network  connection register; token=Token(0)
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #1 TRACE network  Hup: 0
parity_authority1 | 2019-06-25 10:28:29 UTC IO Worker #1 TRACE network  Connection closed: 0
parity_authority1 | 2019-06-25 10:28:29 UTC  TRACE network  connection deregister; token=0

Logs with sync trace level:

parity_authority1 | Loading config file from /parity/config/authority.toml
parity_authority0 | Loading config file from /parity/config/authority.toml
parity_authority1 | 2019-06-25 10:29:56 UTC main INFO parity_ethereum::run  Starting Parity-Ethereum/v2.4.6-stable-94164e1-20190514/x86_64-linux-gnu/rustc1.34.1
parity_authority1 | 2019-06-25 10:29:56 UTC main INFO parity_ethereum::run  Keys path /parity/keys/parityDockerized
parity_authority1 | 2019-06-25 10:29:56 UTC main INFO parity_ethereum::run  DB path /parity/.local/share/io.parity.ethereum/chains/parityDockerized/db/54db8f336512a42e
parity_authority1 | 2019-06-25 10:29:56 UTC main INFO parity_ethereum::run  State DB configuration: fast
parity_authority1 | 2019-06-25 10:29:56 UTC main INFO parity_ethereum::run  Operating mode: active
parity_authority1 | 2019-06-25 10:29:56 UTC main INFO ethcore_service::service  Configured for parityDockerized using AuthorityRound engine
parity_authority1 | 2019-06-25 10:29:57 UTC main INFO parity_ethereum::run  Running without a persistent transaction queue.
parity_authority0 | 2019-06-25 10:29:57 UTC main INFO parity_ethereum::run  Starting Parity-Ethereum/v2.4.6-stable-94164e1-20190514/x86_64-linux-gnu/rustc1.34.1
parity_authority0 | 2019-06-25 10:29:57 UTC main INFO parity_ethereum::run  Keys path /parity/keys/parityDockerized
parity_authority0 | 2019-06-25 10:29:57 UTC main INFO parity_ethereum::run  DB path /parity/.local/share/io.parity.ethereum/chains/parityDockerized/db/54db8f336512a42e
parity_authority0 | 2019-06-25 10:29:57 UTC main INFO parity_ethereum::run  State DB configuration: fast
parity_authority0 | 2019-06-25 10:29:57 UTC main INFO parity_ethereum::run  Operating mode: active
parity_authority0 | 2019-06-25 10:29:57 UTC main INFO ethcore_service::service  Configured for parityDockerized using AuthorityRound engine
parity_authority0 | 2019-06-25 10:29:58 UTC main INFO parity_ethereum::run  Running without a persistent transaction queue.
parity_authority0 | 2019-06-25 10:30:00 UTC IO Worker #3 DEBUG sync  Finished block propagation, took 0ms
parity_authority0 | 2019-06-25 10:30:00 UTC IO Worker #0 TRACE sync  Propagating blocks, state=WaitingPeers
parity_authority0 | 2019-06-25 10:30:00 UTC IO Worker #0 TRACE sync  Sending proposed blocks to []
parity_authority0 | 2019-06-25 10:30:00 UTC IO Worker #0 INFO import  Imported #86 0x99c8…c006 (0 txs, 0.00 Mgas, 2 ms, 0.57 KiB)
parity_authority1 | 2019-06-25 10:30:02 UTC IO Worker #3 INFO network  Public node URL: enode://7a859c5d4294136b1a01b604477ac28a5643ee39a59ce44a9588001883bf16aa262f6fcf56f0c527c536484ebf6a6a28b98a2faa2027764ad9949152ac2ef9ec@172.16.0.11:30303
parity_authority0 | 2019-06-25 10:30:02 UTC IO Worker #1 TRACE sync  == Connected 0: Parity-Ethereum/v2.4.6-stable-94164e1-20190514/x86_64-linux-gnu/rustc1.34.1
parity_authority0 | 2019-06-25 10:30:02 UTC IO Worker #1 TRACE sync  Sending status to 0, protocol version 3
parity_authority1 | 2019-06-25 10:30:02 UTC IO Worker #0 TRACE sync  == Disconnecting 1: Parity-Ethereum/v2.4.6-stable-94164e1-20190514/x86_64-linux-gnu/rustc1.34.1
parity_authority0 | 2019-06-25 10:30:02 UTC IO Worker #1 TRACE sync  == Disconnecting 0: Parity-Ethereum/v2.4.6-stable-94164e1-20190514/x86_64-linux-gnu/rustc1.34.1
parity_authority0 | 2019-06-25 10:30:03 UTC IO Worker #2 TRACE sync  == Connected 0: Parity-Ethereum/v2.4.6-stable-94164e1-20190514/x86_64-linux-gnu/rustc1.34.1
parity_authority0 | 2019-06-25 10:30:03 UTC IO Worker #2 TRACE sync  Sending status to 0, protocol version 3
parity_authority1 | 2019-06-25 10:30:03 UTC IO Worker #3 TRACE sync  == Connected 1: Parity-Ethereum/v2.4.6-stable-94164e1-20190514/x86_64-linux-gnu/rustc1.34.1
parity_authority1 | 2019-06-25 10:30:03 UTC IO Worker #3 TRACE sync  Sending status to 1, protocol version 3
parity_authority0 | 2019-06-25 10:30:03 UTC IO Worker #1 TRACE sync  New peer 0 (protocol: 3, network: 8995, difficulty: Some(20416942015256307807802476445905780526929), latest:0x6824…ea3e, genesis:0x4845…0f25, snapshot:Some(0), private_tx_enabled:false)
parity_authority0 | 2019-06-25 10:30:03 UTC IO Worker #1 DEBUG sync  Connected 0:Parity-Ethereum/v2.4.6-stable-94164e1-20190514/x86_64-linux-gnu/rustc1.34.1
parity_authority1 | 2019-06-25 10:30:03 UTC IO Worker #1 TRACE sync  New peer 1 (protocol: 3, network: 8995, difficulty: Some(29264283555200707857850216239131754024482), latest:0x99c8…c006, genesis:0x4845…0f25, snapshot:Some(0), private_tx_enabled:false)
parity_authority1 | 2019-06-25 10:30:03 UTC IO Worker #1 DEBUG sync  Connected 1:Parity-Ethereum/v2.4.6-stable-94164e1-20190514/x86_64-linux-gnu/rustc1.34.1
parity_authority1 | 2019-06-25 10:30:03 UTC IO Worker #1 TRACE sync  Checking snapshot sync: 0 vs 60 (peer: 1)
parity_authority0 | 2019-06-25 10:30:03 UTC IO Worker #1 TRACE sync  Skipping peer 0, force=false, td=Some(20416942015256307807802476445905780526929), our td=29264283555200707857850216239131754024482, state=WaitingPeers
parity_authority0 | 2019-06-25 10:30:03 UTC IO Worker #1 INFO network  Public node URL: enode://6355b8589ae36fbfdc5f2a3acd1ec6a1ae5841299a9a0132906e73ac3c587bdab5ad6f0353112a3e58f85b91839f14840456345b492dc374f3d6cd92a0361fa2@172.16.0.10:30303
parity_authority1 | 2019-06-25 10:30:04 UTC IO Worker #3 TRACE sync  Syncing with peers: 1 active, 1 available, 1 total
parity_authority1 | 2019-06-25 10:30:04 UTC IO Worker #3 TRACE sync  Checking snapshot sync: 0 vs 60 (peer: 1)
parity_authority0 | 2019-06-25 10:30:05 UTC IO Worker #3 TRACE sync  Checking peer our best 0x99c8…c006 their best 0x6824…ea3e
parity_authority0 | 2019-06-25 10:30:05 UTC IO Worker #3 DEBUG sync  Finished block propagation, took 0ms
parity_authority0 | 2019-06-25 10:30:05 UTC IO Worker #2 TRACE sync  Propagating blocks, state=WaitingPeers
parity_authority0 | 2019-06-25 10:30:05 UTC IO Worker #2 TRACE sync  Checking peer our best 0x10b1…02ef their best 0x6824…ea3e
parity_authority0 | 2019-06-25 10:30:05 UTC IO Worker #2 TRACE sync  Sending NewBlocks to [0]
parity_authority0 | 2019-06-25 10:30:05 UTC IO Worker #2 TRACE sync  Sending NewHashes to [0]
parity_authority1 | 2019-06-25 10:30:05 UTC IO Worker #1 DEBUG sync  1 -> Dispatching packet: 7
parity_authority1 | 2019-06-25 10:30:05 UTC IO Worker #1 TRACE sync  1 -> NewBlock (0x10b1…02ef)
parity_authority0 | 2019-06-25 10:30:05 UTC IO Worker #2 TRACE sync  Sent sealed block to all peers
parity_authority0 | 2019-06-25 10:30:05 UTC IO Worker #2 TRACE sync  Sending proposed blocks to [0]
parity_authority1 | 2019-06-25 10:30:05 UTC IO Worker #1 TRACE sync  New block with unknown parent (0x99c8b85a74af5dfacddbb8158ec0a1cce6298b570c43a53bdeae2066abb7c006) 0x10b19c1014764e8a498c66624f44e304f664ab570066cfd640e72f795fa902ef
parity_authority1 | 2019-06-25 10:30:05 UTC IO Worker #1 TRACE sync  NewBlock ignored while seeking
parity_authority1 | 2019-06-25 10:30:05 UTC IO Worker #1 TRACE sync  Checking snapshot sync: 0 vs 60 (peer: 1)
parity_authority0 | 2019-06-25 10:30:05 UTC IO Worker #2 INFO import  Imported #87 0x10b1…02ef (0 txs, 0.00 Mgas, 0 ms, 0.57 KiB)
parity_authority1 | 2019-06-25 10:30:05 UTC IO Worker #1 DEBUG sync  1 -> Dispatching packet: 1
parity_authority1 | 2019-06-25 10:30:05 UTC IO Worker #1 TRACE sync  Ignoring new hashes since we're already downloading.
parity_authority1 | 2019-06-25 10:30:05 UTC IO Worker #1 TRACE sync  Checking snapshot sync: 0 vs 60 (peer: 1)
parity_authority0 | 2019-06-25 10:30:05 UTC IO Worker #0 TRACE sync  Syncing with peers: 1 active, 1 available, 1 total
parity_authority0 | 2019-06-25 10:30:05 UTC IO Worker #0 TRACE sync  Skipping peer 0, force=false, td=Some(20416942015256307807802476445905780526929), our td=29604565922121646321313590846563522235936, state=WaitingPeers

Any idea? Thanks in advance
@jam10o-new jam10o-new added F2-bug 🐞 The client fails to follow expected behavior. M4-core ⛓ Core client code / Rust. Z0-unconfirmed 🤔 Issue might be valid, but it’s not yet known. labels Jun 28, 2019
@grbIzl
Copy link
Collaborator

grbIzl commented Jul 1, 2019

Hey @miguelmartinezinf thanks for the issue and logs! Can you please clarify more details? Are your nodes not able to connect totally or is it just a stuck at the nodes' startup and after some time connectivity backs to normal? I'm asking this, because the reason of the observed behavior in network logs is hanging on the handshake session, that prevents other sessions to be created. And your log finishes exactly on the place, where this hanging session killed with timeout (5 sec):

parity_authority0 | 2019-06-25 10:28:29 UTC IO Worker #3 TRACE network Connection timeout: 0
parity_authority0 | 2019-06-25 10:28:29 UTC TRACE network connection deregister; token=0

So the most important question for me, what was happening after this exact moment, was Parity able to restore connectivity or not. If not, can you please collect logs with network trace enabled for about 3 minutes after the startup? That would be very helpful for the investigation.

@miguelmtzinf
Copy link
Author

Hey @grbIzl , thanks for your comment. Nodes are not able to connect anymore after an "InternalBacktrace" error appears... If I re-launch each node individually or both nodes at the same time, they are working but not able to inter-connect, they get stuck in the handshake (as you can see in logs).
So, if I re-launch both nodes, node0 (which has a signer) goes on building blocks and node1 (just a replicate peer) tries to sync (unsuccessfully because it cannot connect to node0).

I think I know what exactly the problem is. This connectivity problem started to happen after an "InternalBacktrace" error appeared with the name of "InvalidChainId". This error appeared in logs just after send some transactions indicating a wrong "chainId" as transaction options (my networkId is 0x2323 and chainId was 4 idk why). The transactions got mined and block height increased. My theory is that after this thing, parity node is not able to keep stable, because starts to get connectivity and sync problems. Should parity node rejects the transaction if there is no chain working with the supplied id?

I saw is a good idea to add chainId on tx options when submitting txs for replay protection or choosing between ETH or ETC chains, but should be use also with private chains? Do you think this config option is needed? why?

Thanks in advance

@grbIzl
Copy link
Collaborator

grbIzl commented Jul 2, 2019

@miguelmartinezinf Can you please just make a little bit longer log for both nodes with the same network trace level? 2-3 minutes will be enough. Simply launch nodes and wait a little bit, don't restart or do anything. If you're saying, that the nodes cannot connect to each other, it is the real problem already, and anything else (like issues with chainid etc) can be just a side effect.

@miguelmtzinf
Copy link
Author

I think that chainId issue is not a side effect, but the main problem here...

This time I work with 3 nodes: node0 is the signer one, node1 and node2 are just replicas.
I followed this workflow:

  1. launch 3 nodes
  2. send my transactions with wrong chainId parameter
  3. "Backtrace error" appears on node1 and node2.
  4. I shut down node1 and node2.
  5. I relaunch both node1 and node2 and they cannot connect with node0 to sync. Node1 and node2 are interconnected, but not able to connect with node0. I let them work for more than 5 minutes (and more) and no success.

Here you have logs: node0.log node1.log node1.log

@jam10o-new
Copy link
Contributor

Hey @miguelmartinezinf while the chainId issue is certainly a valid issue (and a known one at that, discussion in #10411), it's a different issue from what @grbIzl has been looking into recently and trying to fix.

While your logs are certainly helpful, what would be most helpful to resolve this issue is network level logs, from two nodes running/starting up in the same time period so that we can see their interactions with each other as the disconnect happens, if you are able to share that? As mentioned above we only need a couple of minutes to figure out what's going on here - the chain ID error causes this issue, but others do too as we have seen this with other errors, we need to understand why peers are dropped after that

@miguelmtzinf
Copy link
Author

Thanks, as @grbIzl is looking into the 2 nodes network I explained at the initial issue description, I decided to take that sceneario (2 nodes network). Here you have the network trace level logs for a 2 node network: node0.log node1.log

Workflow followed:

  1. launch 2 nodes (node0 is signer, node1 is 1 replica)
  2. send my transactions with wrong chainId parameter
  3. "Backtrace error" appears on node1. node0 goes on building blocks, node1 goes on importing blocks.
  4. I shut down node0 and node1.
  5. I relaunch both node0 and node1 and I see sync and connectivity problems... I let them work for more than 5 minutes (and more) and no success.

@grbIzl
Copy link
Collaborator

grbIzl commented Jul 3, 2019

@miguelmartinezinf thanks for the logs! After looking into them I can confirm, that the issue is in the incorrect chainId on node0. Node1 disconnects on every BlockHeader packet, code is here:
https://github.com/paritytech/parity-ethereum/blob/master/ethcore/sync/src/block_sync.rs#L326
I didn't find any other reasons for connectivity losses. And this reason seems to be valid (disconnect peer with such a fork). So I would close this issue, if you don't mind, meanwhile #10411 seems major for me, such transactions should not be added into the block. I think, I will proceed with #10411 further.

@miguelmtzinf
Copy link
Author

Great, thanks a lot @grbIzl @joshua-mir

@grbIzl grbIzl closed this as completed Jul 3, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
F2-bug 🐞 The client fails to follow expected behavior. M4-core ⛓ Core client code / Rust. Z0-unconfirmed 🤔 Issue might be valid, but it’s not yet known.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@grbIzl @miguelmtzinf @jam10o-new