Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed test StoreIngestTest.ConcurrentIngestAndWrite under tsan #9257

Closed
JaySon-Huang opened this issue Jul 26, 2024 · 3 comments · Fixed by #9259
Closed

Failed test StoreIngestTest.ConcurrentIngestAndWrite under tsan #9257

JaySon-Huang opened this issue Jul 26, 2024 · 3 comments · Fixed by #9259
Assignees
@JaySon-Huang
Labels
component/storage severity/moderate type/bug The issue is confirmed as a bug.

Comments

@JaySon-Huang
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. Build gtests_dbms under tsan build type
  2. Run gtests_dbms --gtest_filter=*StoreIngestTest.ConcurrentIngestAndWrite*

2. What did you expect to see? (Required)

3. What did you see instead (Required)

test failed or get blocked

4. What is your TiFlash version? (Required)

master
@JaySon-Huang JaySon-Huang added the type/bug The issue is confirmed as a bug. label Jul 26, 2024
@JaySon-Huang JaySon-Huang self-assigned this Jul 26, 2024
@JaySon-Huang
Copy link
Contributor Author 

ThreadSanitizer: CHECK failed: sanitizer_deadlock_detector.h:67 "((n_all_locks_)) < (((sizeof(all_locks_with_contexts_)/sizeof((all_locks_with_contexts_)[0]))))" (0x40, 0x40) (tid=1691207)

The main thread get blocked in, we need to figure out the other thread cause potential "dead lock"

image

@JaySon-Huang
Copy link
Contributor Author

The TSAN will failed when one thread hold locks larger than 64. This tests will try to do "segmentMerge" on many segments (more than 500) so causing this problem.

The code in question is in lib/sanitizer_common/sanitizer_deadlock_detector.h
It limits the number of simultaneously held locks in a given thread to an arbitrary large number 64.
If you hold 65 locks in one thread at once, this will fail.

google/sanitizers#950

@JaySon-Huang
Copy link
Contributor Author

another data race reported on xxh3, I'll suppress it.

==================
WARNING: ThreadSanitizer: data race (pid=2253513)
  Read of size 8 at 0x000014d2ce48 by thread T154:
    #0 XXH3_hashLong_64b_withSeed_selection /data/nvme0n1/jaysonhuang/tiflash/contrib/xxHash/xxh_x86dispatch.c:665:20 (gtests_dbms+0x103e7d99)
    #1 XXH3_64bits_internal /data/nvme0n1/jaysonhuang/tiflash/contrib/xxHash/cmake_unofficial/../xxhash.h:3860:12 (gtests_dbms+0x103e7d99)
    #2 XXH3_64bits_withSeed_dispatch /data/nvme0n1/jaysonhuang/tiflash/contrib/xxHash/xxh_x86dispatch.c:671:12 (gtests_dbms+0x103e7d99)
    #3 DB::Digest::XXH3::update(void const*, unsigned long) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/Checksum.h:126:17 (gtests_dbms+0xc0b4848)
    #4 DB::UnifiedDigest<DB::Digest::XXH3>::update(void const*, unsigned long) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/Checksum.h:195:70 (gtests_dbms+0xc0b4848)
    #5 DB::DM::DMFileMetaV2::finalize(DB::WriteBuffer&, std::__1::shared_ptr<DB::FileProvider> const&, std::__1::shared_ptr<DB::WriteLimiter> const&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileMetaV2.cpp:202:17 (gtests_dbms+0xc2acadf)
    #6 DB::DM::DMFileWriter::finalizeMeta() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileWriter.cpp:178:19 (gtests_dbms+0xc2e790d)
    #7 DB::DM::DMFileWriter::finalize() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileWriter.cpp:173:5 (gtests_dbms+0xc2e790d)
    #8 DB::DM::DMFileBlockOutputStream::writeSuffix() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileBlockOutputStream.h:44:33 (gtests_dbms+0xc181c24)
    #9 DB::DM::writeIntoNewDMFile(DB::DM::DMContext&, std::__1::shared_ptr<std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > > const&, std::__1::shared_ptr<DB::IBlockInputStream> const&, unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/Segment.cpp:205:20 (gtests_dbms+0xc181c24)
    #10 DB::DM::tests::genDMFile(std::__1::shared_ptr<DB::DM::DeltaMergeStore>, DB::DM::DMContext&, DB::Block const&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_simple_pk_test_basic.cpp:290:19 (gtests_dbms+0x58f8dc7)
    #11 DB::DM::tests::SimplePKTestBasic::ingestFiles(DB::DM::tests::SimplePKTestBasic::IngestFilesOptions const&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_simple_pk_test_basic.cpp:323:18 (gtests_dbms+0x58f95c3)
    #12 DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0::operator()() const /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_ingest.cpp:229:17 (gtests_dbms+0x57fbe1b)
    #13 decltype(static_cast<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&>(fp)()) std::__1::__invoke<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&>(DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0x57fbe1b)
    #14 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&>(DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/invoke.h:61:9 (gtests_dbms+0x57fbe1b)
    #15 std::__1::__function::__alloc_func<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0, std::__1::allocator<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0>, void ()>::operator()() /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:171:16 (gtests_dbms+0x57fbe1b)
    #16 std::__1::__function::__func<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0, std::__1::allocator<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0>, void ()>::operator()() /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:345:12 (gtests_dbms+0x57fbe1b)
    #17 std::__1::__function::__value_func<void ()>::operator()() const /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:498:16 (gtests_dbms+0xfd81811)
    #18 std::__1::function<void ()>::operator()() const /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:1175:12 (gtests_dbms+0xfd81811)
    #19 DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::worker(std::__1::__list_iterator<DB::ThreadFromGlobalPoolImpl<false>, void*>) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:306:17 (gtests_dbms+0xfd81811)
    #20 void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:160:73 (gtests_dbms+0xfd849fc)
    #21 decltype(static_cast<void>(fp)()) std::__1::__invoke_constexpr<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&>(void&&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/type_traits:3924:1 (gtests_dbms+0xfd849fc)
    #22 decltype(auto) std::__1::__apply_tuple_impl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(void&&, std::__1::tuple<>&, std::__1::__tuple_indices<>) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/tuple:1536:1 (gtests_dbms+0xfd849fc)
    #23 decltype(auto) std::__1::apply<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(void&&, std::__1::tuple<>&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/tuple:1545:1 (gtests_dbms+0xfd849fc)
    #24 DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.h:236:17 (gtests_dbms+0xfd849fc)
    #25 decltype(static_cast<void>(fp)()) std::__1::__invoke<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xfd849fc)
    #26 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&>(DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/invoke.h:61:9 (gtests_dbms+0xfd849fc)
    #27 std::__1::__function::__alloc_func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()>, void ()>::operator()() /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:171:16 (gtests_dbms+0xfd849fc)
    #28 std::__1::__function::__func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()>, void ()>::operator()() /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:345:12 (gtests_dbms+0xfd849fc)
    #29 std::__1::__function::__value_func<void ()>::operator()() const /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:498:16 (gtests_dbms+0xfd7f05a)
    #30 std::__1::function<void ()>::operator()() const /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:1175:12 (gtests_dbms+0xfd7f05a)
    #31 DB::ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:306:17 (gtests_dbms+0xfd7f05a)
    #32 void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:160:73 (gtests_dbms+0xfd82ab0)
    #33 decltype(static_cast<void>(fp)()) std::__1::__invoke<void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xfd82ab0)
    #34 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/thread:280:5 (gtests_dbms+0xfd82ab0)
    #35 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()> >(void*) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/thread:291:5 (gtests_dbms+0xfd82ab0)

  Previous write of size 8 at 0x000014d2ce48 by thread T157:
    #0 memcpy /root/llvm-project/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:827:5 (gtests_dbms+0x2c0a7f0)
    #1 setDispatch /data/nvme0n1/jaysonhuang/tiflash/contrib/xxHash/xxh_x86dispatch.c:639:18 (gtests_dbms+0x103e7e38)
    #2 XXH3_hashLong_64b_withSeed_selection /data/nvme0n1/jaysonhuang/tiflash/contrib/xxHash/xxh_x86dispatch.c:665:45 (gtests_dbms+0x103e7e38)
    #3 XXH3_64bits_internal /data/nvme0n1/jaysonhuang/tiflash/contrib/xxHash/cmake_unofficial/../xxhash.h:3860:12 (gtests_dbms+0x103e7e38)
    #4 XXH3_64bits_withSeed_dispatch /data/nvme0n1/jaysonhuang/tiflash/contrib/xxHash/xxh_x86dispatch.c:671:12 (gtests_dbms+0x103e7e38)
    #5 DB::Digest::XXH3::update(void const*, unsigned long) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/Checksum.h:126:17 (gtests_dbms+0xc0b4848)
    #6 DB::UnifiedDigest<DB::Digest::XXH3>::update(void const*, unsigned long) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/Checksum.h:195:70 (gtests_dbms+0xc0b4848)
    #7 DB::DM::DMFileMetaV2::finalize(DB::WriteBuffer&, std::__1::shared_ptr<DB::FileProvider> const&, std::__1::shared_ptr<DB::WriteLimiter> const&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileMetaV2.cpp:202:17 (gtests_dbms+0xc2acadf)
    #8 DB::DM::DMFileWriter::finalizeMeta() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileWriter.cpp:178:19 (gtests_dbms+0xc2e790d)
    #9 DB::DM::DMFileWriter::finalize() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileWriter.cpp:173:5 (gtests_dbms+0xc2e790d)
    #10 DB::DM::DMFileBlockOutputStream::writeSuffix() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileBlockOutputStream.h:44:33 (gtests_dbms+0xc181c24)
    #11 DB::DM::writeIntoNewDMFile(DB::DM::DMContext&, std::__1::shared_ptr<std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > > const&, std::__1::shared_ptr<DB::IBlockInputStream> const&, unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/Segment.cpp:205:20 (gtests_dbms+0xc181c24)
    #12 DB::DM::tests::genDMFile(std::__1::shared_ptr<DB::DM::DeltaMergeStore>, DB::DM::DMContext&, DB::Block const&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_simple_pk_test_basic.cpp:290:19 (gtests_dbms+0x58f8dc7)
    #13 DB::DM::tests::SimplePKTestBasic::ingestFiles(DB::DM::tests::SimplePKTestBasic::IngestFilesOptions const&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_simple_pk_test_basic.cpp:323:18 (gtests_dbms+0x58f95c3)
    #14 DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0::operator()() const /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_ingest.cpp:229:17 (gtests_dbms+0x57fbe1b)
    #15 decltype(static_cast<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&>(fp)()) std::__1::__invoke<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&>(DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0x57fbe1b)
    #16 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&>(DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/invoke.h:61:9 (gtests_dbms+0x57fbe1b)
    #17 std::__1::__function::__alloc_func<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0, std::__1::allocator<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0>, void ()>::operator()() /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:171:16 (gtests_dbms+0x57fbe1b)
    #18 std::__1::__function::__func<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0, std::__1::allocator<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0>, void ()>::operator()() /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:345:12 (gtests_dbms+0x57fbe1b)
    #19 std::__1::__function::__value_func<void ()>::operator()() const /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:498:16 (gtests_dbms+0xfd81811)
    #20 std::__1::function<void ()>::operator()() const /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:1175:12 (gtests_dbms+0xfd81811)
    #21 DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::worker(std::__1::__list_iterator<DB::ThreadFromGlobalPoolImpl<false>, void*>) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:306:17 (gtests_dbms+0xfd81811)
    #22 void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:160:73 (gtests_dbms+0xfd849fc)
    #23 decltype(static_cast<void>(fp)()) std::__1::__invoke_constexpr<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&>(void&&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/type_traits:3924:1 (gtests_dbms+0xfd849fc)
    #24 decltype(auto) std::__1::__apply_tuple_impl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(void&&, std::__1::tuple<>&, std::__1::__tuple_indices<>) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/tuple:1536:1 (gtests_dbms+0xfd849fc)
    #25 decltype(auto) std::__1::apply<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(void&&, std::__1::tuple<>&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/tuple:1545:1 (gtests_dbms+0xfd849fc)
    #26 DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.h:236:17 (gtests_dbms+0xfd849fc)
    #27 decltype(static_cast<void>(fp)()) std::__1::__invoke<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xfd849fc)
    #28 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&>(DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/invoke.h:61:9 (gtests_dbms+0xfd849fc)
    #29 std::__1::__function::__alloc_func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()>, void ()>::operator()() /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:171:16 (gtests_dbms+0xfd849fc)
    #30 std::__1::__function::__func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()>, void ()>::operator()() /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:345:12 (gtests_dbms+0xfd849fc)
    #31 std::__1::__function::__value_func<void ()>::operator()() const /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:498:16 (gtests_dbms+0xfd7f05a)
    #32 std::__1::function<void ()>::operator()() const /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:1175:12 (gtests_dbms+0xfd7f05a)
    #33 DB::ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:306:17 (gtests_dbms+0xfd7f05a)
    #34 void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:160:73 (gtests_dbms+0xfd82ab0)
    #35 decltype(static_cast<void>(fp)()) std::__1::__invoke<void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xfd82ab0)
    #36 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/thread:280:5 (gtests_dbms+0xfd82ab0)
    #37 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()> >(void*) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/thread:291:5 (gtests_dbms+0xfd82ab0)

  Location is global 'g_dispatch' of size 32 at 0x000014d2ce40 (gtests_dbms+0x000014d2ce48)

  Thread T154 'ThreadPool' (tid=2253846, running) created by main thread at:
    #0 pthread_create /root/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:977:3 (gtests_dbms+0x2c00bcd)
    #1 std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__threading_support:514:10 (gtests_dbms+0xfd824e8)
    #2 std::__1::thread::thread<void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'(), void>(void&&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/thread:307:16 (gtests_dbms+0xfd824e8)
    #3 void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:160:35 (gtests_dbms+0xfd7d67c)
    #4 DB::ThreadPoolImpl<std::__1::thread>::scheduleOrThrow(std::__1::function<void ()>, long, unsigned long, bool) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:198:5 (gtests_dbms+0xfd841c4)
    #5 DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.h:219:38 (gtests_dbms+0xfd841c4)
    #6 void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:160:35 (gtests_dbms+0xfd7fb1b)
    #7 DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleOrThrowOnError(std::__1::function<void ()>, long) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:182:5 (gtests_dbms+0xfd7f712)
    #8 DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_ingest.cpp:224:15 (gtests_dbms+0x57f8dc3)
    #9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc603115)
    #10 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc603115)
    #11 testing::Test::Run() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0xc5dccd5)
    #12 testing::TestInfo::Run() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xc5de504)
    #13 testing::TestCase::Run() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xc5df1cd)
    #14 testing::internal::UnitTestImpl::RunAllTests() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xc5ef380)
    #15 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc6045b5)
    #16 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc6045b5)
    #17 testing::UnitTest::Run() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xc5ee668)
    #18 RUN_ALL_TESTS() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x613bd2e)
    #19 main /data/nvme0n1/jaysonhuang/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:116:16 (gtests_dbms+0x613bd2e)

  Thread T157 'ThreadPool' (tid=2253849, running) created by main thread at:
    #0 pthread_create /root/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:977:3 (gtests_dbms+0x2c00bcd)
    #1 std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__threading_support:514:10 (gtests_dbms+0xfd824e8)
    #2 std::__1::thread::thread<void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'(), void>(void&&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/thread:307:16 (gtests_dbms+0xfd824e8)
    #3 void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:160:35 (gtests_dbms+0xfd7d67c)
    #4 DB::ThreadPoolImpl<std::__1::thread>::scheduleOrThrow(std::__1::function<void ()>, long, unsigned long, bool) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:198:5 (gtests_dbms+0xfd841c4)
    #5 DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.h:219:38 (gtests_dbms+0xfd841c4)
    #6 void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:160:35 (gtests_dbms+0xfd7fb1b)
    #7 DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleOrThrowOnError(std::__1::function<void ()>, long) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:182:5 (gtests_dbms+0xfd7f712)
    #8 DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_ingest.cpp:224:15 (gtests_dbms+0x57f8dc3)
    #9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc603115)
    #10 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc603115)
    #11 testing::Test::Run() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0xc5dccd5)
    #12 testing::TestInfo::Run() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xc5de504)
    #13 testing::TestCase::Run() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xc5df1cd)
    #14 testing::internal::UnitTestImpl::RunAllTests() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xc5ef380)
    #15 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc6045b5)
    #16 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc6045b5)
    #17 testing::UnitTest::Run() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xc5ee668)
    #18 RUN_ALL_TESTS() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x613bd2e)
    #19 main /data/nvme0n1/jaysonhuang/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:116:16 (gtests_dbms+0x613bd2e)

SUMMARY: ThreadSanitizer: data race /data/nvme0n1/jaysonhuang/tiflash/contrib/xxHash/xxh_x86dispatch.c:665:20 in XXH3_hashLong_64b_withSeed_selection
==================

@JaySon-Huang JaySon-Huang mentioned this issue Jul 26, 2024
Merged
12 tasks
@ti-chi-bot ti-chi-bot bot closed this as completed in 334d9b9 Jul 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JaySon-Huang JaySon-Huang

Labels
component/storage severity/moderate type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

tests: Limit the segments to be merged under tsan to suppress deadlock JaySon-Huang/tiflash
1 participant
@JaySon-Huang