Improve configurations for sanitized builds #6911
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kiburtse
force-pushed
the
kb/sanitizer_build_configs_improve
branch
3 times, most recently
from
9486893
to
5707838
Compare
|
So all builders with asan do catch an error like this as expected with proper output even on Windows with MSVC 2019: Was there any particular reason we only checked with a/tsan under Ubuntu with clang and omitted all other platforms? Seems to me that at least MacOS with arm64 should be covered. |
kiburtse
force-pushed
the
kb/sanitizer_build_configs_improve
branch
from
5707838
to
30e8e14
Compare
ironage
reviewed
Aug 23, 2023
Yeah, sure. These failures now very consistent from what i can tell. Essentially, now sanitizers apply to real production types builds (Release or RelWithDebInfo) as opposed to before how our sanitizer builds were just Debug builds with -O2 and redundant (-g) flags. So it should be close to real production builds type. |
fealebenpae
approved these changes
Aug 24, 2023
kiburtse
force-pushed
the
kb/sanitizer_build_configs_improve
branch
from
30e8e14
to
7a6ba86
Compare
* Enable address sanitizer for msvc * Allow to build with sanitizer for diffent optimized build (also Debug) * Make RelASAN, RelTSAN, RelUSAN, RelUSAN just shortcuts for half-optimized builds
|
Interesting asan failure under new windows builder with latest master: asan stack-use-after-scope report details[2023/08/24 22:29:39.978] 2: ==1700==ERROR: AddressSanitizer: stack-use-after-scope on address 0x00c3052faaa0 at pc 0x7ff786c216a0 bp 0x00c3052fa3a0 sp 0x00c3052fa3a8 [2023/08/24 22:29:39.980] 2: READ of size 8 at 0x00c3052faaa0 thread T8 [2023/08/24 22:29:40.229] 2: #0 0x7ff786c2169f in mpark::detail::visitation::variant::,, > const &,realm::sync::InternString const &,realm::sync::InternString const &>::invoke C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\external\mpark\variant.hpp:1575 [2023/08/24 22:29:40.229] 2: #1 0x7ff786c50bef in mpark::lib::cpp17::detail::invoke,, > const &>,mpark::detail::alt<0,realm::sync::InternString> const &,mpark::detail::alt<0,realm::sync::InternString> const &> C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\external\mpark\variant.hpp:635 [2023/08/24 22:29:40.230] 2: #2 0x7ff786c428e5 in mpark::detail::visitation::base::make_fmatrix_impl,, > const &> &&,mpark::detail::base<0,realm::sync::InternString,unsigned int> const &,mpark::detail::base<0,realm::sync::InternString,unsigned int> const &>::dispatch<0,0> C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\external\mpark\variant.hpp:1391 [2023/08/24 22:29:40.230] 2: #3 0x7ff786cbd13a in mpark::detail::visitation::alt::visit_alt,, > const &>,mpark::detail::impl const &,mpark::detail::impl const &> C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\external\mpark\variant.hpp:1516 [2023/08/24 22:29:40.230] 2: #4 0x7ff786cbd4b5 in mpark::detail::visitation::variant::visit_value,, > const &,mpark::variant const &,mpark::variant const &> C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\external\mpark\variant.hpp:1616 [2023/08/24 22:29:40.231] 2: #5 0x7ff786c95acf in mpark::visit,, > const &,mpark::variant const &,mpark::variant const &> C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\external\mpark\variant.hpp:2680 [2023/08/24 22:29:40.231] 2: #6 0x7ff786c1f8a8 in realm::`anonymous namespace'::MergeUtils::same_path_element C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\realm\sync\transform.cpp:1013 [2023/08/24 22:29:40.232] 2: #7 0x7ff786c1fc6d in realm::`anonymous namespace'::MergeUtils::same_path C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\realm\sync\transform.cpp:1020 [2023/08/24 22:29:40.232] 2: #8 0x7ff786c1fdc5 in realm::`anonymous namespace'::MergeUtils::same_path C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\realm\sync\transform.cpp:1065 [2023/08/24 22:29:40.232] 2: #9 0x7ff786d121ad in realm::`anonymous namespace'::Merge::DoMerge::do_merge C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\realm\sync\transform.cpp:2250 [2023/08/24 22:29:40.233] 2: #10 0x7ff786c590eb in realm::_impl::merge_instructions_2 C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\realm\sync\transform.cpp:2338 [2023/08/24 22:29:40.233] 2: #11 0x7ff786caed0d in realm::sync::Instruction::visit< > C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\realm\sync\instructions.hpp:979 [2023/08/24 22:29:40.233] 2: #12 0x7ff786c29ac0 in ::operator() C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\realm\sync\transform.cpp:2387 [2023/08/24 22:29:40.234] 2: #13 0x7ff786c9dc3b in realm::sync::Instruction::visit< > C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\realm\sync\instructions.hpp:979 [2023/08/24 22:29:40.234] 2: #14 0x7ff786c1d47f in realm::_impl::TransformerImpl::Transformer::merge_instructions C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\realm\sync\transform.cpp:2386 [2023/08/24 22:29:40.234] 2: #15 0x7ff786d2b08c in realm::_impl::TransformerImpl::Transformer::transform_major C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\realm\sync\transform.cpp:842 [2023/08/24 22:29:40.235] 2: #16 0x7ff786d2ab13 in realm::_impl::TransformerImpl::Transformer::transform C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\realm\sync\transform.cpp:630 [2023/08/24 22:29:40.235] 2: #17 0x7ff786c1a194 in realm::_impl::TransformerImpl::merge_changesets C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\realm\sync\transform.cpp:2528 [2023/08/24 22:29:40.244] 2: #18 0x7ff7861f7a4c in realm::test_util::ShortCircuitHistory::TransformerImpl::merge_changesets C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\test\peer.hpp:384 [2023/08/24 22:29:40.245] 2: #19 0x7ff786c185b1 in realm::_impl::TransformerImpl::transform_remote_changesets C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\realm\sync\transform.cpp:2598 [2023/08/24 22:29:40.245] 2: #20 0x7ff7861f5fca in realm::test_util::ShortCircuitHistory::integrate_remote_changesets C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\test\peer.hpp:495 [2023/08/24 22:29:40.246] 2: #21 0x7ff7861f50bb in realm::test_util::Peer::integrate_next_changesets_from C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\test\peer.hpp:639 [2023/08/24 22:29:40.247] 2: #22 0x7ff78686e742 in `anonymous namespace'::Realm_UnitTest__Transform_SetErase_Clear_same_path::test_run C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\test\test_transform.cpp:2329 [2023/08/24 22:29:40.254] 2: #23 0x7ff78686fb6b in realm::test_util::unit_test::RegisterTest<`anonymous namespace'::Realm_UnitTest__Transform_SetErase_Clear_same_path>::run_test C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\test\util\unit_test.hpp:640 [2023/08/24 22:29:40.255] 2: #24 0x7ff78741d411 in realm::test_util::unit_test::TestList::ThreadContextImpl::run C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\test\util\unit_test.cpp:728 [2023/08/24 22:29:40.262] 2: #25 0x7ff78741c360 in realm::test_util::unit_test::TestList::ThreadContextImpl::run C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\test\util\unit_test.cpp:682 [2023/08/24 22:29:40.262] 2: #26 0x7ff78741e14d in ::operator() C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\test\util\unit_test.cpp:641 [2023/08/24 22:29:40.262] 2: #27 0x7ff78743c043 in realm::util::Thread::entry_point< > C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\realm\util\thread.hpp:483 [2023/08/24 22:29:40.264] 2: #28 0x7ff786292f88 in std::thread::_Invoke,0,1> C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\include\thread:55 [2023/08/24 22:29:40.275] 2: #29 0x7ff78898a057 in invoke_thread_procedure minkernel\crts\ucrt\src\appcrt\startup\thread.cpp:91 [2023/08/24 22:29:40.275] 2: #30 0x7ff788989c00 in thread_start minkernel\crts\ucrt\src\appcrt\startup\thread.cpp:115 [2023/08/24 22:29:40.276] 2: #31 0x7ff786bee339 in __asan::AsanThread::ThreadStart D:\a\_work\1\s\src\vctools\crt\asan\llvm\compiler-rt\lib\asan\asan_thread.cpp:262 [2023/08/24 22:29:40.277] 2: #32 0x7ff802927ac3 in BaseThreadInitThunk+0x13 (C:\Windows\System32\KERNEL32.DLL+0x180017ac3) [2023/08/24 22:29:40.279] 2: #33 0x7ff802f3a350 in RtlUserThreadStart+0x20 (C:\Windows\SYSTEM32\ntdll.dll+0x18005a350) [2023/08/24 22:29:40.281] 2: [2023/08/24 22:29:40.281] 2: Address 0x00c3052faaa0 is located in stack of thread T8 at offset 32 in frame [2023/08/24 22:29:40.281] 2: #0 0x7ff786c1f6df in realm::`anonymous namespace'::MergeUtils::same_path_element C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\realm\sync\transform.cpp:999 [2023/08/24 22:29:40.281] 2: [2023/08/24 22:29:40.281] 2: This frame has 1 object(s): [2023/08/24 22:29:40.281] 2: [32, 41) 'compiler temporary' <== Memory access at offset 32 is inside this variable [2023/08/24 22:29:40.281] 2: HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork [2023/08/24 22:29:40.282] 2: (longjmp, SEH and C++ exceptions *are* supported) [2023/08/24 22:29:40.282] 2: Thread T8 created by T0 here: [2023/08/24 22:29:40.283] 2: #0 0x7ff786becd28 in __asan_wrap_CreateThread D:\a\_work\1\s\src\vctools\crt\asan\llvm\compiler-rt\lib\asan\asan_win.cpp:149 [2023/08/24 22:29:40.283] 2: #1 0x7ff78898a46e in _beginthreadex minkernel\crts\ucrt\src\appcrt\startup\thread.cpp:221 [2023/08/24 22:29:40.284] 2: #2 0x7ff786299e45 in std::thread::_Start C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\include\thread:75 [2023/08/24 22:29:40.284] 2: #3 0x7ff786304bc4 in realm::util::Thread::start C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\realm\util\thread.hpp:469 [2023/08/24 22:29:40.284] 2: #4 0x7ff7874441a9 in realm::util::Thread::start< > C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\realm\util\thread.hpp:450 [2023/08/24 22:29:40.285] 2: #5 0x7ff7873fdafc in realm::test_util::unit_test::TestList::run C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\test\util\unit_test.cpp:645 [2023/08/24 22:29:40.286] 2: #6 0x7ff78612f350 in `anonymous namespace'::run_tests C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\test\test_all.cpp:511 [2023/08/24 22:29:40.292] 2: #7 0x7ff786125191 in test_all C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\test\test_all.cpp:579 [2023/08/24 22:29:40.293] 2: #8 0x7ff786124aa8 in main C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\test\main.cpp:26 [2023/08/24 22:29:40.294] 2: #9 0x7ff788921f88 in invoke_main D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78 [2023/08/24 22:29:40.294] 2: #10 0x7ff788921edd in __scrt_common_main_seh D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288 [2023/08/24 22:29:40.294] 2: #11 0x7ff788921d9d in __scrt_common_main D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:330 [2023/08/24 22:29:40.294] 2: #12 0x7ff788921ffd in mainCRTStartup D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_main.cpp:16 [2023/08/24 22:29:40.294] 2: #13 0x7ff802927ac3 in BaseThreadInitThunk+0x13 (C:\Windows\System32\KERNEL32.DLL+0x180017ac3) [2023/08/24 22:29:40.294] 2: #14 0x7ff802f3a350 in RtlUserThreadStart+0x20 (C:\Windows\SYSTEM32\ntdll.dll+0x18005a350) [2023/08/24 22:29:40.295] 2: [2023/08/24 22:29:40.295] 2: SUMMARY: AddressSanitizer: stack-use-after-scope C:\data\mci\2407e4a8f6458a061c70eee00fc68c36\realm-core\src\external\mpark\variant.hpp:1575 in mpark::detail::visitation::variant::visit_exhaustiveness_check,, > const &,realm::sync::InternString const &,realm::sync::InternString const &>::invoke [2023/08/24 22:29:40.295] 2: Shadow bytes around the buggy address: [2023/08/24 22:29:40.295] 2: 0x02db519df500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [2023/08/24 22:29:40.295] 2: 0x02db519df510: 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1 02 f3 [2023/08/24 22:29:40.295] 2: 0x02db519df520: f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00 00 [2023/08/24 22:29:40.295] 2: 0x02db519df530: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [2023/08/24 22:29:40.295] 2: 0x02db519df540: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [2023/08/24 22:29:40.295] 2: =>0x02db519df550: f1 f1 f1 f1[f8]f8 f3 f3 f3 f3 00 00 00 00 00 00 [2023/08/24 22:29:40.295] 2: 0x02db519df560: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [2023/08/24 22:29:40.295] 2: 0x02db519df570: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [2023/08/24 22:29:40.295] 2: 0x02db519df580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [2023/08/24 22:29:40.295] 2: 0x02db519df590: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 [2023/08/24 22:29:40.295] 2: 0x02db519df5a0: f1 f1 00 00 00 00 00 00 f3 f3 f3 f3 00 00 00 00 [2023/08/24 22:29:40.295] 2: Shadow byte legend (one shadow byte represents 8 application bytes): [2023/08/24 22:29:40.295] 2: Addressable: 00 [2023/08/24 22:29:40.295] 2: Partially addressable: 01 02 03 04 05 06 07 [2023/08/24 22:29:40.295] 2: Heap left redzone: fa [2023/08/24 22:29:40.295] 2: Freed heap region: fd [2023/08/24 22:29:40.295] 2: Stack left redzone: f1 [2023/08/24 22:29:40.295] 2: Stack mid redzone: f2 [2023/08/24 22:29:40.295] 2: Stack right redzone: f3 [2023/08/24 22:29:40.295] 2: Stack after return: f5 [2023/08/24 22:29:40.295] 2: Stack use after scope: f8 [2023/08/24 22:29:40.295] 2: Global redzone: f9 [2023/08/24 22:29:40.295] 2: Global init order: f6 [2023/08/24 22:29:40.295] 2: Poisoned by user: f7 [2023/08/24 22:29:40.295] 2: Container overflow: fc [2023/08/24 22:29:40.295] 2: Array cookie: ac [2023/08/24 22:29:40.295] 2: Intra object redzone: bb [2023/08/24 22:29:40.295] 2: ASan internal: fe [2023/08/24 22:29:40.295] 2: Left alloca redzone: ca [2023/08/24 22:29:40.295] 2: Right alloca redzone: cb [2023/08/24 22:29:40.295] 2: Shadow gap: cc [2023/08/24 22:29:40.322] 2: ==1700==ABORTING [2023/08/24 22:29:40.331] 1/1 Test #2: SyncTests ........................***Failed 0.90 sec it fails consistently now (also on my local setup). I don't think i saw it a few weeks ago when i was testing this first time. Weird. |
kiburtse
force-pushed
the
kb/sanitizer_build_configs_improve
branch
from
7a6ba86
to
a03a96e
Compare
yeah, that i figured out that there is something weird with mpark and msvc 2019. Thanks for the context! At least it's not something new... Btw, what is interesting that msvc 2022 works fine, although there different issues with it :D |
|
TSAN issues on macos seems to be logging related, so we need to make Logger::do_log thread safe and some more. The issue on Ubuntu seems to some serious thing, although i'm not quite sure yet if it's not the same problem as with #6844 Extract of TSAN stacktraces from previous runs |
This was referenced Aug 29, 2023
* maybe fix a buffer overrun in a test * maybe fix a race in async_open_realm test util
kiburtse
force-pushed
the
kb/sanitizer_build_configs_improve
branch
from
f82f9ec
to
c2fd050
Compare
kiburtse
force-pushed
the
kb/sanitizer_build_configs_improve
branch
from
1478165
to
e4da3fd
Compare
Pull Request Test Coverage Report for Build kirill.burtsev_99
💛 - Coveralls |
…led from non main thread and requesting a realm
kiburtse
force-pushed
the
kb/sanitizer_build_configs_improve
branch
from
b09fd73
to
e6c54bc
Compare
3 tasks
|
So, the CI is green with all new configurations. Is it good to go? Workaround for tsan issues with libuv on ubuntu was enough to make the CI happy, but i've filled #7083. I think it would need to solved somehow. Our api currently allows for this invalid usecase. |
ironage
approved these changes
Oct 26, 2023
tgoyne
reviewed
Oct 27, 2023
| @@ -112,6 +112,8 @@ const std::string_view Logger::level_to_string(Level level) noexcept | |||
|
|
|||
| void StderrLogger::do_log(Level level, const std::string& message) | |||
| { | |||
| static Mutex mutex; | |||
| LockGuard l(mutex); | |||
There was a problem hiding this comment.
StderrLogger should be wrapped with ThreadSafeLogger when this behavior is desired rather than adding it to each logger.
There was a problem hiding this comment.
Sure, i'll try to revert this back on next pr in order to not let this one to stale for longer. It's just that wrapping StderrLogger is not the only thing. The very same instance should be then shared everywhere on every config. It's not the case now. Also, the problem is with a few places where logger is initialized from realm default, which doesn't know anything about other parts. So, it's easy to misuse as it turned out during these fixes here.
| @@ -902,10 +902,10 @@ class ColKeyIterator { | |||
|
|
|||
| private: | |||
| friend class ColKeys; | |||
| const Table* m_table; | |||
| ConstTableRef m_table; | |||
There was a problem hiding this comment.
I guess this is probably harmless, but using ColKeys as anything other than a temporary that doesn't need a strong reference to the table seems very suspicious and is probably broken even if it doesn't hit UB.
There was a problem hiding this comment.
it was caught by a few new asan configuraions. How it wasn't fixed in the test in 5 years i don't understand really. Now just it will at least check this situation with the ref.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What, How & Why?
☑️ ToDos
[ ] 🚦 Tests (or not relevant)[ ] C-API, if public C++ API changed.