avoid tiflash crash when query is killed #3434

windtalker · 2021-11-14T09:26:23Z

What problem does this PR solve?

Issue Number: close #3401

Problem Summary:

What is changed and how it works?

Proposal: xxx

What's Changed:

Root cause:

NPE by memory tracker

2021.11.12 12:36:14.389127 [ 112803 ] <Error> BaseDaemon: ########################################                                                                                                 
2021.11.12 12:36:14.393874 [ 112803 ] <Error> BaseDaemon: (from thread 112614) Received signal Segmentation fault (11).
2021.11.12 12:36:14.393924 [ 112803 ] <Error> BaseDaemon: Address: NULL pointer.
2021.11.12 12:36:14.393959 [ 112803 ] <Error> BaseDaemon: Access: read.
2021.11.12 12:36:14.393980 [ 112803 ] <Error> BaseDaemon: Unknown si_code.
2021.11.12 12:36:14.444986 [ 112803 ] <Error> BaseDaemon: 0. /data1/xufei/grafana_test_51x/tiflash1/tiflash(MemoryTracker::free(long)+0x22) [0x3701782]
2021.11.12 12:36:14.445027 [ 112803 ] <Error> BaseDaemon: 1. /data1/xufei/grafana_test_51x/tiflash1/tiflash(MemoryTracker::alloc(long)+0x776) [0x3701fe6]
2021.11.12 12:36:14.445059 [ 112803 ] <Error> BaseDaemon: 2. /data1/xufei/grafana_test_51x/tiflash1/tiflash(Allocator<true>::realloc(void*, unsigned long, unsigned long, unsigned long)+0x100) [0x
2021.11.12 12:36:14.445085 [ 112803 ] <Error> BaseDaemon: 3. /data1/xufei/grafana_test_51x/tiflash1/tiflash() [0x7659dd2]
2021.11.12 12:36:14.445105 [ 112803 ] <Error> BaseDaemon: 4. /data1/xufei/grafana_test_51x/tiflash1/tiflash() [0x767a81a]
2021.11.12 12:36:14.445142 [ 112803 ] <Error> BaseDaemon: 5. /data1/xufei/grafana_test_51x/tiflash1/tiflash(DB::Join::insertFromBlockInternal(DB::Block*, unsigned long)+0x14ad) [0x7686d4d]
2021.11.12 12:36:14.445165 [ 112803 ] <Error> BaseDaemon: 6. /data1/xufei/grafana_test_51x/tiflash1/tiflash(DB::Join::insertFromBlock(DB::Block const&, unsigned long)+0x4a1) [0x76885f1]
2021.11.12 12:36:14.445185 [ 112803 ] <Error> BaseDaemon: 7. /data1/xufei/grafana_test_51x/tiflash1/tiflash(DB::HashJoinBuildBlockInputStream::readImpl()+0x3c) [0x798be8c]
2021.11.12 12:36:14.445227 [ 112803 ] <Error> BaseDaemon: 8. /data1/xufei/grafana_test_51x/tiflash1/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator<fals
2021.11.12 12:36:14.445248 [ 112803 ] <Error> BaseDaemon: 9. /data1/xufei/grafana_test_51x/tiflash1/tiflash(DB::IProfilingBlockInputStream::read()+0x17) [0x6902ff7]
2021.11.12 12:36:14.445274 [ 112803 ] <Error> BaseDaemon: 10. /data1/xufei/grafana_test_51x/tiflash1/tiflash(DB::ParallelInputsProcessor<DB::UnionBlockInputStream<(DB::StreamUnionMode)0>::Handler
2021.11.12 12:36:14.445300 [ 112803 ] <Error> BaseDaemon: 11. /data1/xufei/grafana_test_51x/tiflash1/tiflash(DB::ParallelInputsProcessor<DB::UnionBlockInputStream<(DB::StreamUnionMode)0>::Handler
2021.11.12 12:36:14.445321 [ 112803 ] <Error> BaseDaemon: 12. /data1/xufei/grafana_test_51x/tiflash1/tiflash() [0x873e10f]
2021.11.12 12:36:14.445342 [ 112803 ] <Error> BaseDaemon: 13. /usr/lib64/libpthread.so.0(+0x6e63) [0x7f6344890e63]

The memory tracker is destructed before it is used
2. NPE by join

2021.11.12 23:06:59.506368 [ 843819 ] <Error> BaseDaemon: ########################################
2021.11.12 23:06:59.516163 [ 843819 ] <Error> BaseDaemon: (from thread 843816) Received signal Segmentation fault (11).
2021.11.12 23:06:59.516198 [ 843819 ] <Error> BaseDaemon: Address: NULL pointer.
2021.11.12 23:06:59.516217 [ 843819 ] <Error> BaseDaemon: Access: read.
2021.11.12 23:06:59.516234 [ 843819 ] <Error> BaseDaemon: Address not mapped to object. 
2021.11.12 23:06:59.623789 [ 843819 ] <Error> BaseDaemon: 0. /data1/xufei/grafana_test_51x/tiflash0/tiflash() [0x76cdb60]
2021.11.12 23:06:59.623876 [ 843819 ] <Error> BaseDaemon: 1. /data1/xufei/grafana_test_51x/tiflash0/tiflash(void DB::Join::joinBlockImpl<(DB::ASTTableJoin::Kind)0, (DB::ASTTableJoin::Strictness)2, DB::Join::MapsTemplate<DB::Join::WithUsedFlag<false, DB::Join::RowRefList> > >(DB::Block&, DB::Join::MapsTemplate<DB::Join::WithUsedFlag<false, DB::Join::RowRefList> > const&) const+0xed4) [0x76d0ae4]
2021.11.12 23:06:59.623908 [ 843819 ] <Error> BaseDaemon: 2. /data1/xufei/grafana_test_51x/tiflash0/tiflash(DB::Join::joinBlock(DB::Block&) const+0x252) [0x766b2a2]
2021.11.12 23:06:59.623939 [ 843819 ] <Error> BaseDaemon: 3. /data1/xufei/grafana_test_51x/tiflash0/tiflash(DB::ExpressionAction::execute(DB::Block&) const+0x9f) [0x761357f]
2021.11.12 23:06:59.623964 [ 843819 ] <Error> BaseDaemon: 4. /data1/xufei/grafana_test_51x/tiflash0/tiflash(DB::ExpressionActions::execute(DB::Block&) const+0x6a) [0x7619aba]
2021.11.12 23:06:59.623988 [ 843819 ] <Error> BaseDaemon: 5. /data1/xufei/grafana_test_51x/tiflash0/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x34) [0x74da8b4]
2021.11.12 23:06:59.624018 [ 843819 ] <Error> BaseDaemon: 6. /data1/xufei/grafana_test_51x/tiflash0/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool)+0x3bd) [0x6902dfd]
2021.11.12 23:06:59.624042 [ 843819 ] <Error> BaseDaemon: 7. /data1/xufei/grafana_test_51x/tiflash0/tiflash(DB::IProfilingBlockInputStream::read()+0x17) [0x6902fe7]                               
2021.11.12 23:06:59.624067 [ 843819 ] <Error> BaseDaemon: 8. /data1/xufei/grafana_test_51x/tiflash0/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1b) [0x74da89b]
2021.11.12 23:06:59.624101 [ 843819 ] <Error> BaseDaemon: 9. /data1/xufei/grafana_test_51x/tiflash0/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool)+0x3bd) [0x6902dfd]
2021.11.12 23:06:59.624126 [ 843819 ] <Error> BaseDaemon: 10. /data1/xufei/grafana_test_51x/tiflash0/tiflash(DB::IProfilingBlockInputStream::read()+0x17) [0x6902fe7]
2021.11.12 23:06:59.624150 [ 843819 ] <Error> BaseDaemon: 11. /data1/xufei/grafana_test_51x/tiflash0/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1b) [0x74da89b]
2021.11.12 23:06:59.624178 [ 843819 ] <Error> BaseDaemon: 12. /data1/xufei/grafana_test_51x/tiflash0/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool)+0x3bd) [0x6902dfd]
2021.11.12 23:06:59.624217 [ 843819 ] <Error> BaseDaemon: 13. /data1/xufei/grafana_test_51x/tiflash0/tiflash(DB::IProfilingBlockInputStream::read()+0x17) [0x6902fe7]
2021.11.12 23:06:59.624242 [ 843819 ] <Error> BaseDaemon: 14. /data1/xufei/grafana_test_51x/tiflash0/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1b) [0x74da89b]
2021.11.12 23:06:59.624270 [ 843819 ] <Error> BaseDaemon: 15. /data1/xufei/grafana_test_51x/tiflash0/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool)+0x3bd) [0x6902dfd]
2021.11.12 23:06:59.624294 [ 843819 ] <Error> BaseDaemon: 16. /data1/xufei/grafana_test_51x/tiflash0/tiflash(DB::IProfilingBlockInputStream::read()+0x17) [0x6902fe7]
2021.11.12 23:06:59.624317 [ 843819 ] <Error> BaseDaemon: 17. /data1/xufei/grafana_test_51x/tiflash0/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1b) [0x74da89b]
2021.11.12 23:06:59.624344 [ 843819 ] <Error> BaseDaemon: 18. /data1/xufei/grafana_test_51x/tiflash0/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool)+0x3bd) [0x6902dfd]

The exception happens during hash table build is lost, so hash table probe begins when the hash table is still in build stage.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch:

Check List

Tests

Manual test (add detailed scripts or steps below)

Side effects

Release note

Fix tiflash randomly crash when a mpp query is killed.

ti-chi-bot · 2021-11-14T09:26:25Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

fzhedu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

windtalker · 2021-11-14T10:13:11Z

/run-all-tests

fzhedu · 2021-11-15T02:29:11Z

dbms/src/Flash/Mpp/MPPTask.cpp

+    if (current_memory_tracker != memory_tracker)
+        current_memory_tracker = memory_tracker;


seems be unnecessary?

It's just a double check, if the current_memory_tracker is already memory_tracker, do not need to overwrite current_memory_tracker

fzhedu · 2021-11-16T07:06:35Z

dbms/src/Flash/Mpp/MPPTask.h

    /// store io in MPPTask to keep the life cycle of memory_tracker for the current query
    /// BlockIO contains some information stored in Context and DAGContext, so need deconstruct it before Context and DAGContext
    BlockIO io;
+    /// The inputStreams should be released in the destructor of BlockIO, since DAGContext contains some reference to inputStreams, so it need
+    /// to be destructed before BlockIO
+    std::unique_ptr<DAGContext> dag_context;


update the dependency

fzhedu

LGTM

windtalker · 2021-11-16T07:15:23Z

/merge

ti-chi-bot · 2021-11-16T07:15:24Z

@windtalker: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-11-16T07:15:30Z

This pull request has been accepted and is ready to merge.

Commit hash: cdd99c0

sre-bot · 2021-11-16T07:47:22Z

Coverage detail: https://ci-internal.pingcap.net/job/tics_ghpr_unit_test/75/cobertura/

sre-bot · 2021-11-16T08:17:54Z

Coverage detail: https://ci-internal.pingcap.net/job/tics_ghpr_unit_test/77/cobertura/

ti-chi-bot · 2021-11-16T08:21:46Z

In response to a cherrypick label: new pull request created: #3448.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2021-11-16T08:22:23Z

In response to a cherrypick label: new pull request created: #3449.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2021-11-16T08:23:01Z

In response to a cherrypick label: new pull request created: #3450.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2021-11-16T08:23:39Z

In response to a cherrypick label: new pull request created: #3451.

close #3401

* resovle conflict * format * format code Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn> Co-authored-by: xufei <xufeixw@mail.ustc.edu.cn>

ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 14, 2021

fzhedu reviewed Nov 15, 2021

View reviewed changes

fzhedu reviewed Nov 16, 2021

View reviewed changes

fzhedu approved these changes Nov 16, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 16, 2021

windtalker force-pushed the avoid_tiflash_crash branch from 6f5972e to c318320 Compare November 16, 2021 07:12

ti-chi-bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 16, 2021

avoid tiflash crash when mpp query is killed

cdd99c0

windtalker force-pushed the avoid_tiflash_crash branch from c318320 to cdd99c0 Compare November 16, 2021 07:14

ti-chi-bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 16, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 16, 2021

Merge branch 'master' into avoid_tiflash_crash

c6ac137

ti-chi-bot merged commit d036d55 into pingcap:master Nov 16, 2021

ti-chi-bot mentioned this pull request Nov 16, 2021

avoid tiflash crash when query is killed (#3434) #3448

Merged

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Nov 16, 2021

This is an automated cherry-pick of pingcap#3434

5d90f1c

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Nov 16, 2021

avoid tiflash crash when query is killed (#3434) #3449

Merged

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Nov 16, 2021

This is an automated cherry-pick of pingcap#3434

b845848

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Nov 16, 2021

avoid tiflash crash when query is killed (#3434) #3450

Merged

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Nov 16, 2021

This is an automated cherry-pick of pingcap#3434

56dff20

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Nov 16, 2021

avoid tiflash crash when query is killed (#3434) #3451

Merged

windtalker deleted the avoid_tiflash_crash branch November 16, 2021 08:26

ti-chi-bot added a commit that referenced this pull request Dec 18, 2021

avoid tiflash crash when query is killed (#3434) (#3448)

21b08f0

ti-chi-bot added a commit that referenced this pull request Jan 26, 2022

avoid tiflash crash when query is killed (#3434) (#3449)

e182581

ti-chi-bot added a commit that referenced this pull request Feb 23, 2022

avoid tiflash crash when query is killed (#3434) (#3451)

9b47191

close #3401

windtalker added a commit that referenced this pull request Apr 12, 2022

avoid tiflash crash when query is killed (#3434) (#3450)

4067ebc

* resovle conflict * format * format code Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn> Co-authored-by: xufei <xufeixw@mail.ustc.edu.cn>

gengliqi mentioned this pull request Jul 21, 2022

TiFlash crash when Aggregation FailPoints in createAggregateStates triggers #5356

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

avoid tiflash crash when query is killed #3434

avoid tiflash crash when query is killed #3434

windtalker commented Nov 14, 2021

ti-chi-bot commented Nov 14, 2021 •

edited

Loading

windtalker commented Nov 14, 2021

fzhedu Nov 15, 2021

windtalker Nov 15, 2021

fzhedu Nov 16, 2021

fzhedu left a comment

windtalker commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

sre-bot commented Nov 16, 2021

sre-bot commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

		if (current_memory_tracker != memory_tracker)
		current_memory_tracker = memory_tracker;

avoid tiflash crash when query is killed #3434

avoid tiflash crash when query is killed #3434

Conversation

windtalker commented Nov 14, 2021

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

ti-chi-bot commented Nov 14, 2021 • edited Loading

windtalker commented Nov 14, 2021

fzhedu Nov 15, 2021

Choose a reason for hiding this comment

windtalker Nov 15, 2021

Choose a reason for hiding this comment

fzhedu Nov 16, 2021

Choose a reason for hiding this comment

fzhedu left a comment

Choose a reason for hiding this comment

windtalker commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

sre-bot commented Nov 16, 2021

sre-bot commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

ti-chi-bot commented Nov 14, 2021 •

edited

Loading