AddressSanitizer stack-buffer-overflow in fc::time_point::now()

[abitmore]

Bug Description

witness_node fails to start if built with AddressSanitizer.

False positive error reports may follow
For details see http://code.google.com/p/address-sanitizer/issues/detail?id=189
=================================================================
==14469==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f42cf1b9eb0 at pc 0x7f42dc776860 bp 0x7f42cf1b9ea0 sp 0x7f42cf1b9648
WRITE of size 16 at 0x7f42cf1b9eb0 thread T0
    #0 0x7f42dc77685f  (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x4785f)
    #1 0x1fdb1fd in boost::chrono::system_clock::now() (/usr/btspp/dbg331b/bitshares-core/build/programs/witness_node/witness_node+0x1fdb1fd)
    #2 0x1632d08 in fc::time_point::now() (/usr/btspp/dbg331b/bitshares-core/build/programs/witness_node/witness_node+0x1632d08)
    #3 0x194a4e4 in graphene::net::detail::statistics_gathering_node_delegate_wrapper::call_statistics_collector::actual_execution_measurement_helper::~actual_execution_measurement_helper() (/usr/btspp/dbg331b/bitshares-core/build/programs/witness_node/witness_node+0x194a4e4)
    #4 0x189347c in fc::detail::functor_run<graphene::net::detail::statistics_gathering_node_delegate_wrapper::get_block_ids(std::vector<fc::ripemd160, std::allocator<fc::ripemd160> > const&, unsigned int&, unsigned int)::{lambda()#1}>::run(void*, fc::detail::functor_run<graphene::net::detail::statistics_gathering_node_delegate_wrapper::get_block_ids(std::vector<fc::ripemd160, std::allocator<fc::ripemd160> > const&, unsigned int&, unsigned int)::{lambda()#1}>) (/usr/btspp/dbg331b/bitshares-core/build/programs/witness_node/witness_node+0x189347c)
    #5 0x1600e75 in fc::task_base::run_impl() (/usr/btspp/dbg331b/bitshares-core/build/programs/witness_node/witness_node+0x1600e75)
    #6 0x15fc36c in fc::thread_d::process_tasks() (/usr/btspp/dbg331b/bitshares-core/build/programs/witness_node/witness_node+0x15fc36c)
    #7 0x15fd4c9 in fc::thread_d::start_process_tasks(long) (/usr/btspp/dbg331b/bitshares-core/build/programs/witness_node/witness_node+0x15fd4c9)
    #8 0x1fdb5f0 in make_fcontext (/usr/btspp/dbg331b/bitshares-core/build/programs/witness_node/witness_node+0x1fdb5f0)

0x7f42cf1b9eb0 is located 2094768 bytes inside of 2097152-byte region [0x7f42cefba800,0x7f42cf1ba800)
allocated by thread T0 here:
    #0 0x7f42dc7c7602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
    #1 0x15f8a53 in fc::thread_d::start_next_fiber(bool) (/usr/btspp/dbg331b/bitshares-core/build/programs/witness_node/witness_node+0x15f8a53)

SUMMARY: AddressSanitizer: stack-buffer-overflow ??:0 ??
Shadow bytes around the buggy address:
  0x0fe8d9e2f380: 04 f4 f2 f2 f2 f2 00 00 04 f4 f2 f2 f2 f2 00 00
  0x0fe8d9e2f390: 04 f4 f2 f2 f2 f2 00 00 00 00 f2 f2 f2 f2 00 00
  0x0fe8d9e2f3a0: 00 00 f2 f2 f2 f2 00 00 00 00 f2 f2 f2 f2 00 00
  0x0fe8d9e2f3b0: 00 00 f2 f2 f2 f2 00 00 00 00 f2 f2 f2 f2 00 00
  0x0fe8d9e2f3c0: 00 00 f2 f2 f2 f2 00 00 00 00 f2 f2 f2 f2 00 00
=>0x0fe8d9e2f3d0: 00 00 f3 f3 f3 f3[f3]f3 f3 f3 00 00 00 00 00 00
  0x0fe8d9e2f3e0: 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1 00 00
  0x0fe8d9e2f3f0: f4 f4 f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00
  0x0fe8d9e2f400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe8d9e2f410: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1
  0x0fe8d9e2f420: 00 00 00 00 f2 f2 f2 f2 00 00 00 00 f2 f2 f2 f2
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
==14469==ABORTING

Additional Context (optional)

• This was reported by @syalon who was trying to help on the memory corruption issue (Witness node crash, block id has trailing zeros #1256).

• The log was produced with boost 1.58. It's known that ASAN is not supported well in boost versions earlier than 1.65, in addition, Boost.Coroutine is deprecated so it won't work with ASAN, need to use Boost.Coroutine2 instead in order to get ASAN working (re-build with context-impl=ucontext + BOOST_USE_ASAN) (see AddressSanitizer support boostorg/coroutine#30).

• A similar issue when calling system_clock::now() is discussed here: AddressSanitizer stack-buffer-overflow STEllAR-GROUP/hpx#3824, it seems it's related to Boost.Coroutine as well.

Steps To Reproduce

1. build witness_node with AddressSanitizer

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_FLAGS=-fsanitize=address -fno-omit-frame-pointer -O1 -g ..

2. Run witness_node

Expected Behavior
No crash.

Host Environment
Please provide details about the host environment. Much of this information can be found running: witness_node --version.

• Host OS: Ubuntu 16.04 LTS
• Host Physical RAM 32GB
• BitShares Version: 3.3.1
• OpenSSL Version: 1.0.2g-ubuntu4.15
• Boost Version: 1.58
• Compiler: GCC 5.4

Impacts
Describe which portion(s) of BitShares Core may be impacted by this bug. Please tick at least one box.

• API (the application programming interface)
• Build (the build process or something prior to compiled code)
• CLI (the command line wallet)
• Deployment (the deployment process after building such as Docker, Travis, etc.)
• DEX (the Decentralized EXchange, market engine, etc.)
• P2P (the peer-to-peer network for transaction/block propagation)
• Performance (system or user efficiency, etc.)
• Protocol (the blockchain logic, consensus, validation, etc.)
• Security (the security of system or user data, etc.)
• UX (the User Experience)
• Other (please add below)

CORE TEAM TASK LIST

• Evaluate / Prioritize Bug Report
• Refine User Stories / Requirements
• Define Test Cases
• Design / Develop Solution
• Perform QA/Testing
• Update Documentation

[jmjatlanta]

Interesting side note: While attempting to test just the node layer (very light application layer), I also had trouble with the statistics_gathering_node_delegate_wrapper. It may be caused by tight coupling of the database and node layer. I haven't researched it yet. I was able to get around it by bypassing the delegate. My problem is probably my fault, but wanted to mention it here in case it is related.