re-sync crashed - 3030000 block_validate_exception: Block exception

[EOS9Cat]

Hi there,

I'm re-sync'ing one of our nodes on the main-net, but when we sync'ed at about block number - #5826476, it crashed.

eos v1.1.1

config.ini

get-transactions-time-limit = 3
block-dir = "/opt/eos/blocks"

http-server-address = 0.0.0.0:8888
p2p-listen-endpoint = 0.0.0.0:9876
p2p-server-address = :9876

chain-state-db-size-mb = 16384

p2p-max-nodes-per-host = 100

http-validate-host = false
verbose-http-errors = true

access-control-allow-origin = *
# access-control-allow-headers =
# access-control-max-age =
# access-control-allow-credentials = false

# actor-whitelist =
# actor-blacklist =
# contract-whitelist =
# contract-blacklist =
# filter-on = *

# SSL
# Filename with https private key in PEM format. Required for https (eosio::http_plugin)
# https-server-address =
# Filename with the certificate chain to present on https connections. PEM format. Required for https. (eosio::http_plugin)
# https-certificate-chain-file =
# Filename with https private key in PEM format. Required for https (eosio::http_plugin)
# https-private-key-file =

allowed-connection = any

log-level-net-plugin = info
max-clients = 200
connection-cleanup-period = 60
network-version-match = 1
sync-fetch-span = 2000
enable-stale-production = false

max-implicit-request = 1500
pause-on-startup = false
max-transaction-time = 60
max-irreversible-block-age = -1
txn-reference-block-lag = 0
unlock-timeout = 90000

mongodb-queue-size = 256
# mongodb-uri =

# peer-key =
# peer-private-key =

#plugin = eosio::producer_plugin
plugin = eosio::chain_api_plugin
plugin = eosio::history_plugin
plugin = eosio::history_api_plugin
plugin = eosio::chain_plugin

#plugin = net_plugin
#plugin = net_api_plugin

#BNET
#plugin = eosio::bnet_plugin

agent-name = 

p2p-peer-address = 106.10.42.238:9876
p2p-peer-address = 123.59.116.52:49876
p2p-peer-address = 130.211.59.178:9876
p2p-peer-address = 159.65.214.150:9876

error:

2018-07-25T06:52:41.961 thread-0   net_plugin.cpp:1571           recv_block           ]  got block 5826475 from nodeosd:443 - 6b35004
2018-07-25T06:52:41.961 thread-0   net_plugin.cpp:1609           recv_block           ] calling sync_wait on connection nodeosd:443 - 6b35004
2018-07-25T06:52:41.961 thread-0   net_plugin.cpp:2539           handle_message       ] canceling wait on nodeosd:443 - 6b35004
2018-07-25T06:52:41.962 thread-0   net_plugin.cpp:1674           recv_block           ] canceling wait on nodeosd:443 - 6b35004
2018-07-25T06:52:41.962 thread-0   net_plugin.cpp:2555           handle_message       ] ["nodeosd:443 - 6b35004" 170.33.8.16:443]received signed_block : #5826476 block age in secs = 936531
2018-07-25T06:52:41.963 thread-0   net_plugin.cpp:2718           irreversible_block   ] signaled, id = 0058dc62330a65a14f2670380a87e7d4c339d634fc7f855578ca834de4286765
2018-07-25T06:52:41.963 thread-0   net_plugin.cpp:2709           accepted_block_heade ] signaled, id = 0058e7ac2d0bff79b430d3bd7f3cf301b867523d80fbab5c41be02f359875688
2018-07-25T06:52:41.963 thread-0   controller.cpp:940            maybe_switch_forks   ] switching forks from 0058dc63c436e2a56a387d63b2854c0e8c487fe4d6c3a2ada6bfc62ce6b239c6 (block number 5823587) to 0058e7ac2d0bff79b430d3bd7f3cf301b867523d80fbab5c41be02f359875688 (block number 5826476)
2018-07-25T06:52:41.964 thread-0   controller.cpp:913            push_block           ] 3030000 block_validate_exception: Block exception
attempt to pop beyond last irreversible block
    {}
    thread-0  controller.cpp:76 pop_block
2018-07-25T06:52:41.964 thread-0   producer_plugin.cpp:306       on_incoming_block    ] 3030000 block_validate_exception: Block exception
attempt to pop beyond last irreversible block
    {}
    thread-0  controller.cpp:76 pop_block
rethrow
    {}
    thread-0  controller.cpp:913 push_block
2018-07-25T06:52:41.965 thread-0   net_plugin.cpp:1571           recv_block           ]  got block 5826476 from nodeosd:443 - 6b35004
2018-07-25T06:52:41.965 thread-0   net_plugin.cpp:1609           recv_block           ] calling sync_wait on connection nodeosd:443 - 6b35004
2018-07-25T06:52:41.965 thread-0   net_plugin.cpp:2539           handle_message       ] canceling wait on nodeosd:443 - 6b35004
2018-07-25T06:52:41.965 thread-0   net_plugin.cpp:1674           recv_block           ] canceling wait on nodeosd:443 - 6b35004
2018-07-25T06:52:41.965 thread-0   net_plugin.cpp:2555           handle_message       ] ["nodeosd:443 - 6b35004" 170.33.8.16:443]received signed_block : #5826477 block age in secs = 936531
2018-07-25T06:52:41.966 thread-0   net_plugin.cpp:2718           irreversible_block   ] signaled, id = 0058dc63c436e2a56a387d63b2854c0e8c487fe4d6c3a2ada6bfc62ce6b239c6
2018-07-25T06:52:41.966 thread-0   net_plugin.cpp:2709           accepted_block_heade ] signaled, id = 0058e7ad1fe2aaab19b1ea033c6324a597d7de29b5aa849efc83672a00eff92c
2018-07-25T06:52:41.966 thread-0   controller.cpp:940            maybe_switch_forks   ] switching forks from 0058dc63c436e2a56a387d63b2854c0e8c487fe4d6c3a2ada6bfc62ce6b239c6 (block number 5823587) to 0058e7ad1fe2aaab19b1ea033c6324a597d7de29b5aa849efc83672a00eff92c (block number 5826477)

I reproduced many times that it crashed every time.

Please help.

thanks.

[jgiszczak]

Neither of those are crashes. Those are caught exceptions, handled successfully. From the look of it, you have a bad peer. Remove all but one of them and allow your node to synchronize completely, then add them back one by one until you determine which peer is persistently sending you bad blocks.

[EOS9Cat]

Thanks. Why did one of the peers make the bad blocks?

[jgiszczak]

The blocks themselves may not be bad. On closer examination, it looks like your node is suffering frequent forks during synching, which is not expected to happen with 1.1.1. Verify you're actually running 1.1.1 with cleos get info to your node. The reported version should be 40a2076.

You didn't include anything in the pasted output indicating an actual crash, but there is another very similar issue, filed today, that did. You may be experiencing #4884.

[heifner]

Very old and no update from reporter, closing.