function scoped static variable destruct early and cause a crash when used with ASAN

[zergvszerg]

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

Expected behavior

The application linked with rocksdb 8.1.1 is also compiled with -fsanitize=address. During exit() it should stop rocksdb background threads normally.

Actual behavior

In BlockBasedTable::PrefetchIndexAndFilterBlocks() the function scoped static variable kBuiltinNameAndAliases is reportedly already deconstructed by asan while being accessed.

Steps to reproduce the behavior

Main thread return from main function and asan caught above behaviour.

(gdb) bt
#0 0x00007fb1e9318017 in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fb1e85db0f7 in std::thread::join() () from /lib64/libstdc++.so.6
#2 0x0000000001536510 in rocksdb::ThreadPoolImpl::Impl::JoinThreads (this=0x611000021740, wait_for_jobs_to_complete=) at /kylin/rocksdb/util/threadpool_imp.cc:197
#3 0x000000000146f88d in rocksdb::(anonymous namespace)::PosixEnv::~PosixEnv (this=0x2025660 rocksdb::Env::Default()::default_env, __in_chrg=) at /kylin/rocksdb/env/env_posix.cc:222
#4 0x00007fb1e7c79ce9 in __run_exit_handlers () from /lib64/libc.so.6
#5 0x00007fb1e7c79d37 in exit () from /lib64/libc.so.6
#6 0x00007fb1e7c6255c in __libc_start_main () from /lib64/libc.so.6
#7 0x0000000000458667 in _start ()

=================================================================
==798==ERROR: AddressSanitizer: heap-use-after-free on address 0x60d000020110 at pc 0x000000ac3aa9 bp 0x7fb1e28983e0 sp 0x7fb1e28983d0
READ of size 8 at 0x60d000020110 thread T2
#0 0xac3aa8 in std::_Hashtable<std::string, std::string, std::allocatorstd::string, std::__detail::_Identity, std::equal_tostd::string, std::hashstd::string, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, tru
e, true> >::_M_find_before_node(unsigned long, std::string const&, unsigned long) const /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/hashtable.h:1541
#1 0x1379130 in std::_Hashtable<std::string, std::string, std::allocatorstd::string, std::__detail::_Identity, std::equal_tostd::string, std::hashstd::string, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, tr
ue, true> >::_M_find_node(unsigned long, std::string const&, unsigned long) const /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/hashtable.h:642
#2 0x1379130 in std::_Hashtable<std::string, std::string, std::allocatorstd::string, std::__detail::_Identity, std::equal_tostd::string, std::hashstd::string, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, tr
ue, true> >::find(std::string const&) const /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/hashtable.h:1438
#3 0x1379130 in std::unordered_set<std::string, std::hashstd::string, std::equal_tostd::string, std::allocatorstd::string >::find(std::string const&) const /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/unordered_set.h:653
#4 0x1379130 in rocksdb::BlockBasedTable::PrefetchIndexAndFilterBlocks(rocksdb::ReadOptions const&, rocksdb::FilePrefetchBuffer*, rocksdb::InternalIteratorBaserocksdb::Slice, rocksdb::BlockBasedTable, bool, rocksdb::BlockBasedTableOptions const&, int, unsigned long, unsigned long, rocksdb::BlockCacheLookupCont
ext*) /kylin/rocksdb/table/block_based/block_based_table_reader.cc:1016
#5 0x137a6f4 in rocksdb::BlockBasedTable::Open(rocksdb::ReadOptions const&, rocksdb::ImmutableOptions const&, rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&, rocksdb::InternalKeyComparator const&, std::unique_ptr<rocksdb::RandomAccessFileReader, std::default_delete<rocksdb::RandomAccessFileRead
er> >&&, unsigned long, std::unique_ptr<rocksdb::TableReader, std::default_deleterocksdb::TableReader >, std::shared_ptrrocksdb::CacheReservationManager, std::shared_ptr<rocksdb::SliceTransform const> const&, bool, bool, int, bool, unsigned long, bool, rocksdb::TailPrefetchStats, rocksdb::BlockCacheTracer*, unsi
gned long, std::string const&, unsigned long, std::array<unsigned long, 2ul>) /kylin/rocksdb/table/block_based/block_based_table_reader.cc:766
#6 0x1369f03 in rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::ReadOptions const&, rocksdb::TableReaderOptions const&, std::unique_ptr<rocksdb::RandomAccessFileReader, std::default_deleterocksdb::RandomAccessFileReader >&&, unsigned long, std::unique_ptr<rocksdb::TableReader, std::default_delete >, bool) const /kylin/rocksdb/table/block_based/block_based_table_factory.cc:581
#7 0x124fc5b in rocksdb::TableCache::GetTableReader(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, bool, bool, rocksdb::HistogramImpl, std::unique_ptr<rocksdb::TableReader, std::default_deleterocksdb::TableReader >, std::shared_ptr
<rocksdb::SliceTransform const> const&, bool, int, bool, unsigned long, rocksdb::Temperature) /kylin/rocksdb/db/table_cache.cc:140
#8 0x125146d in rocksdb::TableCache::FindTable(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, rocksdb::BasicTypedCacheInterface<rocksdb::TableReader, (rocksdb::CacheEntryRole)13, rocksdb::Cache>::TypedHandle**, std::shared_ptr const&, bool, bool, rocksdb::HistogramImpl*, bool, int, bool, unsigned long, rocksdb::Temperature) /kylin/rocksdb/db/table_cache.cc:187
#9 0x1253eb8 in rocksdb::TableCache::NewIterator(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, rocksdb::RangeDelAggregator*, std::shared_ptr<rocksdb::SliceTransform const> const&, rocksdb::TableReader**, rocksdb::HistogramImpl*, rocks
db::TableReaderCaller, rocksdb::Arena*, bool, int, unsigned long, rocksdb::InternalKey const*, rocksdb::InternalKey const*, bool, rocksdb::TruncatedRangeDelIterator**) /kylin/rocksdb/db/table_cache.cc:228
#10 0x14bf668 in operator() /kylin/rocksdb/db/compaction/compaction_job.cc:739
#11 0x14cb1aa in rocksdb::CompactionJob::Run() /kylin/rocksdb/db/compaction/compaction_job.cc:773
#12 0x11a6cc1 in rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority) /kylin/rocksdb/db/db_impl/db_impl_compaction_flush.cc:3539
#13 0x11a854b in rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority) /kylin/rocksdb/db/db_impl/db_impl_compaction_flush.cc:3054
#14 0x11a8bb5 in rocksdb::DBImpl::BGWorkCompaction(void*) /kylin/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2793
#15 0x1536855 in std::function<void ()>::operator()() const /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/std_function.h:706
#16 0x1536855 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) /kylin/rocksdb/util/threadpool_imp.cc:312
#17 0x1536a29 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) /kylin/rocksdb/util/threadpool_imp.cc:353
#18 0x184d3be in execute_native_thread_routine (/data/vol1/kts/tabletserver4/bin/tabletserver_9040+0x184d3be)
#19 0x7fb1e9316ea4 in start_thread (/lib64/libpthread.so.0+0x7ea4)
#20 0x7fb1e7d3eb0c in clone (/lib64/libc.so.6+0xfeb0c)

0x60d000020110 is located 32 bytes inside of 136-byte region [0x60d0000200f0,0x60d000020178)
freed by thread T0 here:
#0 0x7fb1e960c020 in operator delete(void*) (/lib64/libasan.so.4+0xe1020)
#1 0x7fb1e7c79ce8 in __run_exit_handlers (/lib64/libc.so.6+0x39ce8)

[ywave620]

Could you post your code?

[ywave620]

My guess is that somehow failed to call std::thread::join() for Thread T2

[zergvszerg]

My guess is that main thread only try to join other threads in ~PosixEnv() from __run_exit_handlers, at which point the culprit bg compaction thread while still running a task may bump into the destructing unordered_set since during __run_exit_handlers there is absolutedly ZERO guarantee on the destruction order of global variables?

[zergvszerg]

@ywave620
Sorry for the late reply,
actually our inhouse rocksdb fork is basically rocksdb 8.1.1 with some minor logging changes.

[ywave620]

I guess you forget to call Close() for every DB you open before the program exit(either by calling exit() or reaching the end of the main function). If this is the case, just bring Close() back.

Here is one possible explanation of your backtrace:
As the result of that the program exit, see https://en.cppreference.com/w/cpp/utility/program/exit

Objects with static storage duration are destroyed and functions registered by calling std::atexit are called:
a) Non-local objects with static storage duration are destroyed in the reverse order of the completion of their constructor.

The static variable kBuiltinNameAndAliases destructs before the PosixEnv, which joins all threads. Since you miss a DB::Close() (my assumption), compaction, a background task is running and access the destructed variable.

Note that calling DB::Close() will wait for all background task to finish.

[zergvszerg]

@ywave620
Thanks for the help, we do forget to Close one db.