tests: Fix MPPTask-Moniter may live longer than TiFlashMetrics #9096
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #9092, close #9097
Problem Summary:
For #9092
TMTContext will start a thread "MPPTask-Moniter" for running
checkLongLiveMPPTasks
.tiflash/dbms/src/Storages/KVStore/TMTContext.cpp
Lines 121 to 126 in 38ab3f9
When the TiFlash is shutting down, the thread is not explicitly stopped. And the thread may live longer than the TiFlashMetrics instance. If the TiFlashMetrics instance is released before
checkLongLiveMPPTasks
run, thencheckLongLiveMPPTasks
will access to a random address and cause use-after-free data race when shutting down.For #9097
Seems the race is reported in
backtrace-rs
, there is nothing we can do in tiflash code, just ignoreWhat is changed and how it works?
For #9092
In
TMTContext::shutdown
, set theMPPTaskMonitor->is_shutdown = true
. So the thread is expected to be stopped after TMTContext::shutdown is called and beforeTiFlashMetrics
is release. And whenmonitor->is_shutdown == true
, the thread don't report the metircs toTiFlashMetrics
For #9097
Add
race:StackTrace::toString, race:DB::SyncPointCtl::sync
to tsan.suppression. And they will be ignored when running withTSAN_OPTIONS="suppressions=/tests/sanitize/tsan.suppression" ./dbms/gtests_dbms --gtest_filter=...
Check List
Tests
Side effects
Documentation
Release note