nodeos crash in ubuntu 16.04 after running several days as block producer

[liangcd]

Build : dawn-v3.0.0
Environment : Ubuntu 16.04.3 LTS
Linux kernal: Linux i-hmidbtjm 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Issue: nodeos crash after running several days as block producer.

More information:

1. output from dmesg command after crash.

[808583.358958] traps: nodeos[25675] general protection ip:7f9b0aedc478 sp:7ffe6b2f09e0 error:0 in libc-2.23.so[7f9b0ae5e000+1c0000]
[852538.838959] nodeos[8772]: segfault at 7fb4cd719010 ip 00000000006297f5 sp 00007ffcf6b4d860 error 6 in nodeos[400000+20b1000]
[912444.313550] ip_tables: (C) 2000-2006 Netfilter Core Team
[1352932.903115] nodeos[8301]: segfault at 0 ip 0000000000629fb5 sp 00007ffc08447960 error 6 in nodeos[400000+20b1000]

2. last stderr.txt before crash:
   push block #1321906 from bat 2018-04-14T21:05:28.000 00142bb2d89c58997771c116a0f678891e2594193ff67c620dc285f2c4bdd076 lib: 1310190 success
   push block #1321907 from bat 2018-04-14T21:05:28.500 00142bb348d94b0cf0c8775891398c3193fd30285227a3043c400f98b4ca01e0 lib: 1310190 success
   350671ms thread-0 fork_database.cpp:78 _push_block ] Number of missed blocks: 12
   350673ms thread-0 chain_controller.cpp:1406 validate_block_heade ] head_block_time 2018-04-14T21:05:28.500, next_block 2018-04-14T21:05:35.000, block_interval 500
   350673ms thread-0 chain_controller.cpp:1408 validate_block_heade ] Did not produce block within block_interval 500ms, took 6500ms)
   push block #1321908 from dragonfly 2018-04-14T21:05:35.000 00142bb449668c7fda66f6dda1f34754ac168ff6961f8db55addc3ea818956cc lib: 1310190 success
   350674ms thread-0 net_plugin.cpp:1385 recv_notice ] sync_manager got catch up block notice
   350674ms thread-0 net_plugin.cpp:1385 recv_notice ] sync_manager got catch up block notice
   350674ms thread-0 net_plugin.cpp:1385 recv_notice ] sync_manager got catch up block notice
   350674ms thread-0 net_plugin.cpp:1385 recv_notice ] sync_manager got catch up block notice
   350674ms thread-0 net_plugin.cpp:1385 recv_notice ] sync_manager got catch up block notice
   350674ms thread-0 net_plugin.cpp:1385 recv_notice ] sync_manager got catch up block notice
   350674ms thread-0 net_plugin.cpp:1385 recv_notice ] sync_manager got catch up block notice
   350674ms thread-0 net_plugin.cpp:1385 recv_notice ] sync_manager got catch up block notice
   350674ms thread-0 net_plugin.cpp:1385 recv_notice ] sync_manager got catch up block notice
   350674ms thread-0 net_plugin.cpp:1385 recv_notice ] sync_manager got catch up block notice
   350674ms thread-0 net_plugin.cpp:1385 recv_notice ] sync_manager got catch up block notice
   350674ms thread-0 net_plugin.cpp:1385 recv_notice ] sync_manager got catch up block notice
   350674ms thread-0 net_plugin.cpp:1385 recv_notice ] sync_manager got catch up block notice
   350675ms thread-0 net_plugin.cpp:1385 recv_notice ] sync_manager got catch up block notice
   350675ms thread-0 net_plugin.cpp:1385 recv_notice ] sync_manager got catch up block notice
   350675ms thread-0 net_plugin.cpp:1385 recv_notice ] sync_manager got catch up block notice
   350675ms thread-0 net_plugin.cpp:1385 recv_notice ] sync_manager got catch up block notice
   350675ms thread-0 net_plugin.cpp:1385 recv_notice ] sync_manager got catch up block notice

3. after crash, restarting the nodeos can turn the node back to normal.

[liangcd]

It happen again today

dmesg:
[852538.838959] nodeos[8772]: segfault at 7fb4cd719010 ip 00000000006297f5 sp 00007ffcf6b4d860 error 6 in nodeos[400000+20b1000]
[912444.313550] ip_tables: (C) 2000-2006 Netfilter Core Team
[1352932.903115] nodeos[8301]: segfault at 0 ip 0000000000629fb5 sp 00007ffc08447960 error 6 in nodeos[400000+20b1000]
[1496711.262204] nodeos[23747]: segfault at f ip 0000000000a68d0e sp 00007fffe4f584e0 error 4 in nodeos[400000+20b1000]

tail -f stderr.txt
123722ms thread-0 net_plugin.cpp:1755 operator() ] connection failed to 149.202.201.27:9876: Connection refused
123743ms thread-0 net_plugin.cpp:1755 operator() ] connection failed to jungle.cryptolions.io:9890: Connection refused
123842ms thread-0 net_plugin.cpp:1755 operator() ] connection failed to ctestnet.eosio.se:6602: Connection refused
push block #1539511 from spider 2018-04-16T13:02:04.000 00177db730701a7d0d71e394f8493dbca8d985af6992f35a19ce09777c90967f lib: 1539044 success
push block #1539512 from spider 2018-04-16T13:02:04.500 00177db8a4dae07ff84ba68532adaf46bf6f124e179cbf9e78ecc52d320cf2fe lib: 1539044 success

[liangcd]

After that starting nodeos gets following errors:

3028278ms thread-0 block_log.cpp:120 open ] Log is nonempty
3028281ms thread-0 chain_plugin.cpp:217 plugin_startup ] 13 NSt8ios_base7failureB5cxx11E: basic_ios::clear: iostream error
rethrow basic_ios::clear: iostream error:
{"my->genesis_file.generic_string()":"/data/JungleTestnet/genesis.json","what":"basic_ios::clear: iostream error"}
thread-0 chain_plugin.cpp:217 plugin_startup
3028281ms thread-0 chain_plugin.cpp:217 plugin_startup ] my->genesis_file.generic_string(): /data/JungleTestnet/genesis.json
3028282ms thread-0 main.cpp:95 main ] 13 NSt8ios_base7failureB5cxx11E: basic_ios::clear: iostream error
rethrow basic_ios::clear: iostream error:
{"my->genesis_file.generic_string()":"/data/JungleTestnet/genesis.json","what":"basic_ios::clear: iostream error"}
thread-0 chain_plugin.cpp:217 plugin_startup

[kyzitemelos]

I keep getting the same errors:
387353ms thread-0 main.cpp:95 main ] 13 St13__ios_failure: basic_ios::clear: iostream error
rethrow basic_ios::clear: iostream error
{"my>genesis_file.generic_string()":"/home/user/.local/share/eosio/nodeos/config/genesis.json" ,"what":"basic_ios::clear: iostream error"} thread-0 chain_plugin.cpp:217 plugin_startup

Any idea what exactly it is?
I don't know where to check the error numbers at.

[chris-allnutt]

I'm seeing a similar error when stopping and restating the node, happens when I get an exit code of 137 while stopping

./nodeosd.sh
2018-04-20T15:04:27.209029956Z 
2018-04-20T15:04:27.209450498Z 
267222ms thread-0   chain_plugin.cpp:99           plugin_initialize    ] initializing chain plugin
267222ms thread-0   http_plugin.cpp:141           plugin_initialize    ] host: 0.0.0.0 port: 8888 
267222ms thread-0   http_plugin.cpp:144           plugin_initialize    ] configured http to listen on 0.0.0.0:8888
267222ms thread-0   wallet_plugin.cpp:41          plugin_initialize    ] initializing wallet plugin
267222ms thread-0   wallet_api_plugin.cpp:112     plugin_initialize    ] 
*************************************
*                                   *
*  --   Wallet NOT on localhost  -- *
*  - Password and/or Private Keys - *
*  - are transferred unencrypted. - *
*                                   *
*************************************
2018-04-20T15:04:27.222707004Z 
267222ms thread-0   net_plugin.cpp:2631           plugin_initialize    ] Initialize net plugin
267222ms thread-0   net_plugin.cpp:2647           plugin_initialize    ] Setting net_plugin logging level to info
267222ms thread-0   net_plugin.cpp:2672           plugin_initialize    ] host: 0.0.0.0 port: 9876 
267223ms thread-0   net_plugin.cpp:2748           plugin_initialize    ] my node_id is 6217230e72b17d80373947860fb2018bd21196207de7181921c71fce05030dfa
267224ms thread-0   main.cpp:90                   main                 ] nodeos version 367ada2d
267224ms thread-0   main.cpp:91                   main                 ] eosio root is /root/.local/share
267235ms thread-0   block_log.cpp:120             open                 ] Log is nonempty
267238ms thread-0   chain_plugin.cpp:217          plugin_startup       ] 13 NSt8ios_base7failureB5cxx11E: basic_ios::clear: iostream error
basic_ios::clear: iostream error: error unpacking fc::sha256
    {"type":"fc::sha256","what":"basic_ios::clear: iostream error"}
    thread-0  raw.hpp:568 unpack
Error unpacking field previous
    {"field":"previous"}
    thread-0  raw.hpp:327 operator()
error unpacking eosio::chain::signed_block
    {"type":"eosio::chain::signed_block"}
    thread-0  raw.hpp:568 unpack
267238ms thread-0   chain_plugin.cpp:217          plugin_startup       ] my->genesis_file.generic_string(): /opt/eosio/bin/data-dir/genesis.json 
267238ms thread-0   main.cpp:95                   main                 ] 13 NSt8ios_base7failureB5cxx11E: basic_ios::clear: iostream error
basic_ios::clear: iostream error: error unpacking fc::sha256
    {"type":"fc::sha256","what":"basic_ios::clear: iostream error"}
    thread-0  raw.hpp:568 unpack
Error unpacking field previous
    {"field":"previous"}
    thread-0  raw.hpp:327 operator()
error unpacking eosio::chain::signed_block
    {"type":"eosio::chain::signed_block"}
    thread-0  raw.hpp:568 unpack
rethrow
    {"my->genesis_file.generic_string()":"/opt/eosio/bin/data-dir/genesis.json"}
    thread-0  chain_plugin.cpp:217 plugin_startup

[liangcd]

I caught a core dump in a crash while syncing blocks.

(lldb) bt all
Core file '/data/JungleTestnet/core' (x86_64) was loaded.

(lldb) * thread #1, name = 'nodeos', stop reason = signal SIGSEGV

• frame #0: nodeosboost::simple_segregated_storage<unsigned long>::free(this=0x000000000314dee0, chunk=0x0000000000000000) at simple_segregated_storage.hpp:165 frame #1: nodeosboost::simple_segregated_storage::ordered_free(this=0x000000000314dee0, chunk=0x0000000000000000) at simple_segregated_storage.hpp:182
  frame Transactions should Explicitly Declare Required Authority Level #2: nodeosboost::pool<boost::default_user_allocator_new_delete>::ordered_free(this=0x000000000314dee0, chunk=0x0000000000000000) at pool.hpp:471 frame #3: nodeosboost::object_pool<std::array<char, 1048576ul>, boost::default_user_allocator_new_delete>::free(this=0x000000000314dee0, chunk=0x0000000000000000) at object_pool.hpp:126
  frame Implement Last Irreversible Block, et al #4: nodeosboost::object_pool<std::array<char, 1048576ul>, boost::default_user_allocator_new_delete>::destroy(this=0x000000000314dee0, chunk=0x0000000000000000) at object_pool.hpp:205 frame #5: nodeoseosio::message_buffer<1048576u>::reset(this=0x000000000484f410) at message_buffer.hpp:110
  frame Port to AppBase #6: nodeoseosio::message_buffer<1048576u>::advance_read_ptr(this=0x000000000484f410, bytes=32) at message_buffer.hpp:154 frame #7: nodeoseosio::message_buffer<1048576u>::read(this=0x000000000484f410, s=0x00007fa74802dbd0, size=32) at message_buffer.hpp:205
  frame Continuous Growing Merkel of All Block IDs  #8: nodeoseosio::mb_datastream<1048576u>::read(this=0x00007ffd5d1fdd98, d="\\\xa9m\xac\xb1\x80P>\xaa\x93?????\x1cA?b?q}L6\xb3??\x17m\x9b@", s=32) at message_buffer.hpp:284 frame #9: nodeoseosio::mb_datastream<1048576u>& fc::operator>><eosio::mb_datastream<1048576u> >(ds=0x00007ffd5d1fdd98, ep=0x00007fa74802dbd0) at sha256.hpp:59
  frame Fix unit tests #10: nodeosvoid fc::raw::detail::if_class<fc::true_type>::unpack<eosio::mb_datastream<1048576u>, fc::sha256>(s=0x00007ffd5d1fdd98, v=0x00007fa74802dbd0) at raw.hpp:338 frame #11: nodeosvoid fc::raw::detail::if_reflectedfc::false_type::unpack<eosio::mb_datastream<1048576u>, fc::sha256>(s=0x00007ffd5d1fdd98, v=0x00007fa74802dbd0) at raw.hpp:386
  frame Chain API #12: nodeosvoid fc::raw::unpack<eosio::mb_datastream<1048576u>, fc::sha256>(s=0x00007ffd5d1fdd98, v=0x00007fa74802dbd0) at raw.hpp:567 frame #13: nodeosvoid fc::raw::detail::unpack_object_visitor<eosio::mb_datastream<1048576u>, eosio::chain::transaction_receipt>::operator(this=0x00007ffd5d1fbb20, name="")<fc::sha256, eosio::chain::transaction_receipt, &(eosio::chain::transaction_receipt::id)>(char const*) const at raw.hpp:326
  frame API Theory #14: nodeosvoid fc::reflector<eosio::chain::transaction_receipt>::visit<fc::raw::detail::unpack_object_visitor<eosio::mb_datastream<1048576u>, eosio::chain::transaction_receipt> >(v=0x00007ffd5d1fbb20) at transaction.hpp:259 frame #15: nodeosvoid fc::raw::detail::if_enumfc::false_type::unpack<eosio::mb_datastream<1048576u>, eosio::chain::transaction_receipt>(s=0x00007ffd5d1fdd98, v=0x00007fa74802dbc0) at raw.hpp:361
  frame Atomically create and use account? #16: nodeosvoid fc::raw::detail::if_reflected<fc::true_type>::unpack<eosio::mb_datastream<1048576u>, eosio::chain::transaction_receipt>(s=0x00007ffd5d1fdd98, v=0x00007fa74802dbc0) at raw.hpp:397 frame #17: nodeosvoid fc::raw::unpack<eosio::mb_datastream<1048576u>, eosio::chain::transaction_receipt>(s=0x00007ffd5d1fdd98, v=0x00007fa74802dbc0) at raw.hpp:567
  frame Unify types #18: nodeosvoid fc::raw::unpack<eosio::mb_datastream<1048576u>, eosio::chain::transaction_receipt>(s=0x00007ffd5d1fdd98, value=size=1) at raw.hpp:531 frame #19: nodeosvoid fc::raw::detail::unpack_object_visitor<eosio::mb_datastream<1048576u>, eosio::chain::shard_summary>::operator(this=0x00007ffd5d1fc340, name="")<std::vector<eosio::chain::transaction_receipt, std::allocatoreosio::chain::transaction_receipt >, eosio::chain::shard_summary, &(eosio::chain::shard_summary::transactions)>(char const*) const at raw.hpp:326
  frame Please add Dockerfile #20: nodeosvoid fc::reflector<eosio::chain::shard_summary>::visit<fc::raw::detail::unpack_object_visitor<eosio::mb_datastream<1048576u>, eosio::chain::shard_summary> >(v=0x00007ffd5d1fc340) at block.hpp:131 frame #21: nodeosvoid fc::raw::detail::if_enumfc::false_type::unpack<eosio::mb_datastream<1048576u>, eosio::chain::shard_summary>(s=0x00007ffd5d1fdd98, v=0x00007fa74801be30) at raw.hpp:361
  frame appbase @ 024ec0d not properly linked in master #22: nodeosvoid fc::raw::detail::if_reflected<fc::true_type>::unpack<eosio::mb_datastream<1048576u>, eosio::chain::shard_summary>(s=0x00007ffd5d1fdd98, v=0x00007fa74801be30) at raw.hpp:397 frame #23: nodeosvoid fc::raw::unpack<eosio::mb_datastream<1048576u>, eosio::chain::shard_summary>(s=0x00007ffd5d1fdd98, v=0x00007fa74801be30) at raw.hpp:567
  frame CMake on the mac #24: nodeosvoid fc::raw::unpack<eosio::mb_datastream<1048576u>, eosio::chain::shard_summary>(s=0x00007ffd5d1fdd98, value=size=1) at raw.hpp:531 frame #25: nodeosvoid fc::raw::unpack<eosio::mb_datastream<1048576u>, std::vector<eosio::chain::shard_summary, std::allocatoreosio::chain::shard_summary > >(s=0x00007ffd5d1fdd98, value=size=1) at raw.hpp:531
  frame fc::variant deserialization fails for boost::multiprecision::number<T> #26: nodeosvoid fc::raw::detail::unpack_object_visitor<eosio::mb_datastream<1048576u>, eosio::chain::region_summary>::operator(this=0x00007ffd5d1fcc40, name="")<std::vector<std::vector<eosio::chain::shard_summary, std::allocator<eosio::chain::shard_summary> >, std::allocator<std::vector<eosio::chain::shard_summary, std::allocator<eosio::chain::shard_summary> > > >, eosio::chain::region_summary, &(eosio::chain::region_summary::cycles_summary)>(char const*) const at raw.hpp:326 frame #27: nodeosvoid fc::reflectoreosio::chain::region_summary::visit<fc::raw::detail::unpack_object_visitor<eosio::mb_datastream<1048576u>, eosio::chain::region_summary> >(v=0x00007ffd5d1fcc40) at block.hpp:132
  frame Proposed API JSON response format #28: nodeosvoid fc::raw::detail::if_enum<fc::false_type>::unpack<eosio::mb_datastream<1048576u>, eosio::chain::region_summary>(s=0x00007ffd5d1fdd98, v=0x00007fa748007810) at raw.hpp:361 frame #29: nodeosvoid fc::raw::detail::if_reflectedfc::true_type::unpack<eosio::mb_datastream<1048576u>, eosio::chain::region_summary>(s=0x00007ffd5d1fdd98, v=0x00007fa748007810) at raw.hpp:397
  frame Add get types api #30: nodeosvoid fc::raw::unpack<eosio::mb_datastream<1048576u>, eosio::chain::region_summary>(s=0x00007ffd5d1fdd98, v=0x00007fa748007810) at raw.hpp:567 frame #31: nodeosvoid fc::raw::unpack<eosio::mb_datastream<1048576u>, eosio::chain::region_summary>(s=0x00007ffd5d1fdd98, value=size=1) at raw.hpp:531
  frame Building latest Web Assemble changes #32: nodeosvoid fc::raw::detail::unpack_object_visitor<eosio::mb_datastream<1048576u>, eosio::chain::signed_block_summary>::operator(this=0x00007ffd5d1fd460, name="")<std::vector<eosio::chain::region_summary, std::allocator<eosio::chain::region_summary> >, eosio::chain::signed_block_summary, &(eosio::chain::signed_block_summary::regions)>(char const*) const at raw.hpp:326 frame #33: nodeosvoid fc::reflectoreosio::chain::signed_block_summary::visit<fc::raw::detail::unpack_object_visitor<eosio::mb_datastream<1048576u>, eosio::chain::signed_block_summary> >(v=0x00007ffd5d1fd460) at block.hpp:133
  frame deterministic_randomness test failures #34: nodeosvoid fc::raw::detail::if_enum<fc::false_type>::unpack<eosio::mb_datastream<1048576u>, eosio::chain::signed_block_summary>(s=0x00007ffd5d1fdd98, v=0x00007ffd5d1fdbf8) at raw.hpp:361 frame #35: nodeosvoid fc::raw::detail::if_reflectedfc::true_type::unpack<eosio::mb_datastream<1048576u>, eosio::chain::signed_block_summary>(s=0x00007ffd5d1fdd98, v=0x00007ffd5d1fdbf8) at raw.hpp:397
  frame Failing test: block_tests/create_script #36: nodeosvoid fc::raw::unpack<eosio::mb_datastream<1048576u>, eosio::chain::signed_block_summary>(s=0x00007ffd5d1fdd98, v=0x00007ffd5d1fdbf8) at raw.hpp:567 frame #37: nodeosvoid fc::raw::unpack_static_variant<eosio::mb_datastream<1048576u> >::operator(this=0x00007ffd5d1fd9d0, v=0x00007ffd5d1fdbf8)eosio::chain::signed_block_summary(eosio::chain::signed_block_summary&) const at raw.hpp:669
  frame up the minimum required boost level to 1.64 to fix problem with strin… #38: nodeosfc::raw::unpack_static_variant<eosio::mb_datastream<1048576u> >::result_type fc::impl::storage_ops<6, eosio::chain::signed_block_summary, eosio::chain::signed_block, eosio::chain::signed_transaction, eosio::chain::packed_transaction>::apply<fc::raw::unpack_static_variant<eosio::mb_datastream<1048576u> > >(n=6, data=0x00007ffd5d1fdbf8, v=0x00007ffd5d1fd9d0) at static_variant.hpp:98 frame #39: nodeosfc::raw::unpack_static_variant<eosio::mb_datastream<1048576u> >::result_type fc::impl::storage_ops<5, eosio::sync_request_message, eosio::chain::signed_block_summary, eosio::chain::signed_block, eosio::chain::signed_transaction, eosio::chain::packed_transaction>::apply<fc::raw::unpack_static_variant<eosio::mb_datastream<1048576u> > >(n=6, data=0x00007ffd5d1fdbf8, v=0x00007ffd5d1fd9d0) at static_variant.hpp:99
  frame Fix #34 - deterministic_randomness test failures #40: nodeosfc::raw::unpack_static_variant<eosio::mb_datastream<1048576u> >::result_type fc::impl::storage_ops<4, eosio::request_message, eosio::sync_request_message, eosio::chain::signed_block_summary, eosio::chain::signed_block, eosio::chain::signed_transaction, eosio::chain::packed_transaction>::apply<fc::raw::unpack_static_variant<eosio::mb_datastream<1048576u> > >(n=6, data=0x00007ffd5d1fdbf8, v=0x00007ffd5d1fd9d0) at static_variant.hpp:99 frame #41: nodeosfc::raw::unpack_static_variant<eosio::mb_datastream<1048576u> >::result_type fc::impl::storage_ops<3, eosio::notice_message, eosio::request_message, eosio::sync_request_message, eosio::chain::signed_block_summary, eosio::chain::signed_block, eosio::chain::signed_transaction, eosio::chain::packed_transaction>::apply<fc::raw::unpack_static_variant<eosio::mb_datastream<1048576u> > >(n=6, data=0x00007ffd5d1fdbf8, v=0x00007ffd5d1fd9d0) at static_variant.hpp:99
  frame Building error. Linking CXX executable chainbase_test #42: nodeosfc::raw::unpack_static_variant<eosio::mb_datastream<1048576u> >::result_type fc::impl::storage_ops<2, eosio::time_message, eosio::notice_message, eosio::request_message, eosio::sync_request_message, eosio::chain::signed_block_summary, eosio::chain::signed_block, eosio::chain::signed_transaction, eosio::chain::packed_transaction>::apply<fc::raw::unpack_static_variant<eosio::mb_datastream<1048576u> > >(n=6, data=0x00007ffd5d1fdbf8, v=0x00007ffd5d1fd9d0) at static_variant.hpp:99 frame #43: nodeosfc::raw::unpack_static_variant<eosio::mb_datastream<1048576u> >::result_type fc::impl::storage_ops<1, eosio::go_away_message, eosio::time_message, eosio::notice_message, eosio::request_message, eosio::sync_request_message, eosio::chain::signed_block_summary, eosio::chain::signed_block, eosio::chain::signed_transaction, eosio::chain::packed_transaction>::apply<fc::raw::unpack_static_variant<eosio::mb_datastream<1048576u> > >(n=6, data=0x00007ffd5d1fdbf8, v=0x00007ffd5d1fd9d0) at static_variant.hpp:99
  frame Net_plugin first feature #44: nodeosfc::raw::unpack_static_variant<eosio::mb_datastream<1048576u> >::result_type fc::impl::storage_ops<0, eosio::handshake_message, eosio::go_away_message, eosio::time_message, eosio::notice_message, eosio::request_message, eosio::sync_request_message, eosio::chain::signed_block_summary, eosio::chain::signed_block, eosio::chain::signed_transaction, eosio::chain::packed_transaction>::apply<fc::raw::unpack_static_variant<eosio::mb_datastream<1048576u> > >(n=6, data=0x00007ffd5d1fdbf8, v=0x00007ffd5d1fd9d0) at static_variant.hpp:99 frame #45: nodeosfc::raw::unpack_static_variant<eosio::mb_datastream<1048576u> >::result_type fc::static_variant<eosio::handshake_message, eosio::go_away_message, eosio::time_message, eosio::notice_message, eosio::request_message, eosio::sync_request_message, eosio::chain::signed_block_summary, eosio::chain::signed_block, eosio::chain::signed_transaction, eosio::chain::packed_transaction>::visit<fc::raw::unpack_static_variant<eosio::mb_datastream<1048576u> > >(this=0x00007ffd5d1fdbf8, v=0x00007ffd5d1fd9d0) at static_variant.hpp:329
  frame Docker compile and docker-compose start eos #46: nodeosvoid fc::raw::unpack<eosio::mb_datastream<1048576u>, eosio::handshake_message, eosio::go_away_message, eosio::time_message, eosio::notice_message, eosio::request_message, eosio::sync_request_message, eosio::chain::signed_block_summary, eosio::chain::signed_block, eosio::chain::signed_transaction, eosio::chain::packed_transaction>(s=0x00007ffd5d1fdd98, sv=0x00007ffd5d1fdbf8) at raw.hpp:686 frame #47: nodeoseosio::connection::process_next_message(this=0x000000000484f390, impl=0x00000000047b63f0, message_length=271) at net_plugin.cpp:1099
  frame PLEase help - EOS ICO #48: nodeoseosio::net_plugin_impl::start_read_message(this=0x00007ffd5d1feae8, ec=error_code @ 0x00007ffd5d1fea00, bytes_transferred=13354)::$_15::operator()(boost::system::error_code, unsigned long) const at net_plugin.cpp:1839 frame #49: nodeosboost::asio::detail::binder2<eosio::net_plugin_impl::start_read_message(std::shared_ptreosio::connection)::$_15, boost::system::error_code, unsigned long>::operator(this=0x00007ffd5d1feae8)() at bind_handler.hpp:164
  frame misc_tests/deterministic_randomness is failing #50: nodeosvoid boost::asio::asio_handler_invoke<boost::asio::detail::binder2<eosio::net_plugin_impl::start_read_message(std::shared_ptr<eosio::connection>)::$_15, boost::system::error_code, unsigned long> >(function=0x00007ffd5d1feae8)::$_15, boost::system::error_code, unsigned long>&, ...) at handler_invoke_hook.hpp:69 frame #51: nodeosvoid boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<eosio::net_plugin_impl::start_read_message(std::shared_ptreosio::connection)::$_15, boost::system::error_code, unsigned long>, eosio::net_plugin_impl::start_read_message(std::shared_ptreosio::connection)::$_15>(function=0x00007ffd5d1feae8, context=0x00007ffd5d1feae8)::$_15, boost::system::error_code, unsigned long>&, eosio::net_plugin_impl::start_read_message(std::shared_ptreosio::connection)::$_15&) at handler_invoke_helpers.hpp:37
  frame Removing float opcodes from non-float macro #52: nodeosvoid boost::asio::detail::handler_work<eosio::net_plugin_impl::start_read_message(std::shared_ptr<eosio::connection>)::$_15, boost::asio::system_executor>::complete<boost::asio::detail::binder2<eosio::net_plugin_impl::start_read_message(this=0x00007ffd5d1feb18, function=0x00007ffd5d1feae8, handler=0x00007ffd5d1feae8)::$_15, boost::system::error_code, unsigned long> >(boost::asio::detail::binder2<eosio::net_plugin_impl::start_read_message(std::shared_ptr<eosio::connection>)::$_15, boost::system::error_code, unsigned long>&, eosio::net_plugin_impl::start_read_message(std::shared_ptr<eosio::connection>)::$_15&) at handler_work.hpp:82 frame #53: nodeosboost::asio::detail::reactive_socket_recv_op<std::vector<boost::asio::mutable_buffer, std::allocatorboost::asio::mutable_buffer >, eosio::net_plugin_impl::start_read_message(std::shared_ptreosio::connection)::$_15>::do_complete(owner=0x00000000047b4f20, base=0x00007fa748031fb0, (null)=0x00007ffd5d1fedb8, (null)=0) at reactive_socket_recv_op.hpp:122
  frame Supplementary docker operation detailed documentation #54: nodeosboost::asio::detail::scheduler_operation::complete(this=0x00007fa748031fb0, owner=0x00000000047b4f20, ec=0x00007ffd5d1fedb8, bytes_transferred=0) at scheduler_operation.hpp:40 frame #55: nodeosboost::asio::detail::epoll_reactor::descriptor_state::do_complete(owner=0x00000000047b4f20, base=0x0000000004e18ec0, ec=0x00007ffd5d1fedb8, bytes_transferred=5) at epoll_reactor.ipp:776
  frame Change Account Name & Message Types #56: nodeosboost::asio::detail::scheduler_operation::complete(this=0x0000000004e18ec0, owner=0x00000000047b4f20, ec=0x00007ffd5d1fedb8, bytes_transferred=5) at scheduler_operation.hpp:40 frame #57: nodeosboost::asio::detail::scheduler::do_run_one(this=0x00000000047b4f20, lock=0x00007ffd5d1fed10, this_thread=0x00007ffd5d1fed48, ec=0x00007ffd5d1fedb8) at scheduler.ipp:401
  frame could not gain write access to the shared memory file #58: nodeosboost::asio::detail::scheduler::run(this=0x00000000047b4f20, ec=0x00007ffd5d1fedb8) at scheduler.ipp:154 frame #59: nodeosboost::asio::io_context::run(this=0x00000000047b4eb0) at io_context.ipp:62
  frame Unable to build docker image #60: nodeosappbase::application::exec(this=0x000000000314da50) at application.cpp:209 frame #61: nodeosmain(argc=5, argv=0x00007ffd5d1ff858) at main.cpp:93
  frame [dontmerge] Hack to fix link error #62: 0x00007fa997b0a830 libc.so.6__libc_start_main + 240 frame #63: 0x00007ffd5d399669 nodeos_start + 41
  thread Transactions should Explicitly Declare Required Authority Level #2, stop reason = signal 0
  frame #0: 0x00007fa99945b360 libpthread.so.0__pthread_cond_wait + 192 frame #1: nodeosvoid boost::asio::detail::posix_event::waitboost::asio::detail::conditionally_enabled_mutex::scoped_lock(this=0x0000000004847670, lock=0x00007fa9539e5c60) at posix_event.hpp:106
  frame Transactions should Explicitly Declare Required Authority Level #2: nodeosboost::asio::detail::conditionally_enabled_event::wait(this=0x0000000004847668, lock=0x00007fa9539e5c60) at conditionally_enabled_event.hpp:89 frame #3: nodeosboost::asio::detail::scheduler::do_run_one(this=0x0000000004847600, lock=0x00007fa9539e5c60, this_thread=0x00007fa9539e5c98, ec=0x00007fa9539e5d08) at scheduler.ipp:409
  frame Implement Last Irreversible Block, et al #4: nodeosboost::asio::detail::scheduler::run(this=0x0000000004847600, ec=0x00007fa9539e5d08) at scheduler.ipp:154 frame #5: nodeosboost::asio::io_context::run(this=0x0000000004845eb0) at io_context.ipp:62
  frame Port to AppBase #6: nodeosboost::asio::detail::resolver_service_base::work_io_context_runner::operator(this=0x00000000048477c8)() at resolver_service_base.ipp:32 frame #7: nodeosboost::asio::detail::posix_thread::funcboost::asio::detail::resolver_service_base::work_io_context_runner::run(this=0x00000000048477c0) at posix_thread.hpp:86
  frame Continuous Growing Merkel of All Block IDs  #8: nodeosboost_asio_detail_posix_thread_function(arg=0x00000000048477c0) at posix_thread.ipp:74 frame #9: 0x00007fa9994556ba libpthread.so.0start_thread + 202
  frame Fix unit tests #10: 0x00007fa997bf141d libc.so.6`clone + 109

[jgiszczak]

This appears to be two different issues.

Startup after a crash or ungraceful shutdown nearly always fails due to corruption of the boost shared memory cache of the in-memory database. --resync is required to clean up the mess.

For normal shutdown, never kill with -9. Always use either the default (no argument) signal (which is SIGTERM) or SIGINT. Numerically, those are 15 and 2, respectively.

pkill nodeos | Safe
pkill -15 nodeos | Safe
pkill -2 nodeos | Safe
pkill -TERM nodeos | Safe
pkill -SIGTERM nodeos | Safe
pkill -INT nodeos | Safe
pkill -SIGINT nodeos | Safe
pkill -9 nodeos | Not Safe
pkill -KILL nodeos | Not Safe
pkill -SIGKILL nodeos | Not Safe

The core dump is a different problem. That looks like a corrupted network packet, specifically a signed_block_summary. Summary messages are being eliminated from the protocol, so this particular error will no longer be possible soon.

[RayMetz100]

Update: I fixed my issue with:
rm -r ~/.local/share/eosio/nodeos/data

I'm using Docker on Win10. The first time I ran Dawn 4.2, I ran: docker run --name nodeos -p 8888:8888 -p 9876:9876 -t eosio/eos nodeosd.sh arg1 arg2
I noticed my windows firewall pop up and hit ok. Then I saw blocks logging every half-second which seemed good.

After that worked, I did a docker system prune -a and tried version 1.0.1. It fails with:
2094856ms thread-0 http_plugin.cpp:369 add_handler ] add api url: /v1/history/get_transaction

2094856ms thread-0 net_plugin.cpp:2920 plugin_startup ] starting listener, max clients is 25

2094856ms thread-0 producer_plugin.cpp:577 plugin_startup ] producer plugin: plugin_startup() begin

2094856ms thread-0 producer_plugin.cpp:592 plugin_startup ] Launching block production for 1 producers at 2018-06-02T22:34:54.856.

2094856ms thread-0 producer_plugin.cpp:604 plugin_startup ] producer plugin: plugin_startup() end

It doesn't show the blocks every half second now. I pruned and retried a few times and even reverted back to Dawn 4.2, but no luck yet. Do I need to reset something on my network or firewall in addition to docker system prune -a ? This is my first time trying Docker or nodeos.

Also, assuming the mainnet arrives, how do I connect to it from my own Docker nodeos?