PageStorage control add more args to debug. #4994

jiaqizho · 2022-05-25T06:47:21Z

What problem does this PR solve?

Issue Number: ref #3594

Problem Summary:

What is changed and how it works?

Add only_restore_snapshot_log/not_restore_blob/skip_mvcc_gc
It can help dump wal/snapshot log without BlobStore init.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

close #4265

close #4223

close #4262

close #4215

* Remove previous pingcap license header Signed-off-by: yibin <huyibin@pingcap.com> * Process cluster_manage folder to add license folder Signed-off-by: yibin <huyibin@pingcap.com> * Remove empty lines Signed-off-by: yibin <huyibin@pingcap.com> * Process release-centos7-llvm for adding license header Signed-off-by: yibin <huyibin@pingcap.com> * Process release-darwin for adding license header Signed-off-by: yibin <huyibin@pingcap.com> * Process metrics for adding lh Signed-off-by: yibin <huyibin@pingcap.com> * Process tests for adding lh Signed-off-by: yibin <huyibin@pingcap.com> * Process deprected for adding lh Signed-off-by: yibin <huyibin@pingcap.com> * Process root CMakeLists.txt for adding lh Signed-off-by: yibin <huyibin@pingcap.com> * Process cmake for adding lh Signed-off-by: yibin <huyibin@pingcap.com> * Process root files for adding lh Signed-off-by: yibin <huyibin@pingcap.com> * Process dbms for adding lh Signed-off-by: yibin <huyibin@pingcap.com> * Process libs for adding lh Signed-off-by: yibin <huyibin@pingcap.com> * Makeup for several files added very recently that missing license head Signed-off-by: yibin <huyibin@pingcap.com> * Process one more recently commit file Signed-off-by: yibin <huyibin@pingcap.com> Co-authored-by: ruoxi <zanmato1984@gmail.com>

close #4284

ref #3594

ref #4141

…n union/agg block input stream (#4285) ref #4229

close #4150

* 1.add metrics of calldata&mpptunnel 2.refine shutdown logic Signed-off-by: bestwoody <bestwoody@163.com> * update * Apply suggestions from code review Co-authored-by: Fu Zhe <fuzhe1989@gmail.com> * Update dbms/src/Flash/EstablishCall.cpp Co-authored-by: Fu Zhe <fuzhe1989@gmail.com> * add harm limit to wait Signed-off-by: bestwoody <bestwoody@163.com> * fix Signed-off-by: bestwoody <bestwoody@163.com> Co-authored-by: Fu Zhe <fuzhe1989@gmail.com>

ref #3594

ref #4141

close #4248

close #4292

close #4299

ref #4019

close #4280

close #4298

close #4305

close #4204

close #4232

close #4256

close #4313

#4317) close #4316

close #4311

close #4308

…d local_tunnel (#4322) close #4321

ref #4298

ref #4316

…4950) close #4949

close #4953

* flush cache before segment merge * keep flush until success * check whether segment is valid if flush failed * Update dbms/src/Storages/DeltaMerge/tests/gtest_dm_segment.cpp Co-authored-by: JaySon <tshent@qq.com> * add more fix * check flush result in segment::write Co-authored-by: JaySon <tshent@qq.com>

close #4939

…uled task with exceeded state from the waiting tasks queue (#4958) close #4954

ref #4909

close #4937

ref #4909

…DAGQueryBlockInterpreter` (#4961) ref #4605

ref #4118

ref #3594

close #4985

…id "being_ref_count" (#4987) close #4957, close #4986

ti-chi-bot · 2022-05-25T06:47:22Z

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

JaySon-Huang · 2022-05-25T08:43:54Z

dbms/src/Storages/Page/V3/WAL/WALReader.cpp

+#ifndef NDEBUG
+                if (only_snapshot_log)
+                {
+                    if (name.level_num >= 1)
+                    {
+                        log_files.insert(name);
+                    }
+                    break;
+                }
+#endif


I think we don't need this. We can tell where the entry from by checking this log

tiflash/dbms/src/Storages/Page/V3/WAL/WALReader.cpp

Line 217 in cfcaaec

LOG_FMT_DEBUG(logger, "Open log file for reading [file={}]", fullname);

JaySon-Huang · 2022-05-25T08:46:27Z

dbms/src/Storages/Page/V3/tests/page_storage_ctl.cpp

@@ -470,8 +526,14 @@ class PageStorageControl

 using namespace DB::PS::V3;
 int main(int argc, char ** argv)
+try
 {
    const auto & options = ControlOptions::parse(argc, argv);


Suggest use TiFlashTestEnv::setupLogger to setup logger and replace all cerr/cout by LOG_FMT_XXX

tiflash/dbms/src/TestUtils/TiFlashTestEnv.cpp

Lines 108 to 116 in cfcaaec

void TiFlashTestEnv::setupLogger(const String & level, std::ostream & os)

{

Poco::AutoPtr<Poco::ConsoleChannel> channel = new Poco::ConsoleChannel(os);

Poco::AutoPtr<UnifiedLogPatternFormatter> formatter(new UnifiedLogPatternFormatter());

formatter->setProperty("pattern", "%L%Y-%m-%d %H:%M:%S.%i [%I] <%p> %s: %t");

Poco::AutoPtr<Poco::FormattingChannel> formatting_channel(new Poco::FormattingChannel(formatter, channel));

Poco::Logger::root().setChannel(formatting_channel);

Poco::Logger::root().setLevel(level);

}

ti-chi-bot · 2022-05-26T05:43:49Z

@jiaqizho: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

lidezhu and others added 30 commits March 15, 2022 12:49

Fix unstable test (#4264)

fecc33b

close #4265

disable cache small column by default (#4225)

d90530a

close #4223

fix coredump when TiFlash shutdown (#4272)

060c474

close #4262

Support ddl job type of ActionRenameTables in TiFlash (#4234)

afe8091

close #4215

Fix two issues introduced by adding license header (#4286)

da252dc

close #4284

Use CRC64 as checksum algo for LogFile (#4261)

0ec758c

ref #3594

Clean deprecated directories and internal links (#4278)

14185f7

ref #4141

fix bug that ExchangeReceiver is not cancelled if exception happens i…

0e40b9e

…n union/agg block input stream (#4285) ref #4229

Support dayofweek and dayofyear pushdown to tiflash (#4222)

0daf533

close #4150

WALStore support multi path (#4085)

6141520

ref #3594

Limit build release parallelism (#4301)

b9379c4

ref #4141

add HashJoinProbeBlockInputStream (#4246)

502dd4a

close #4248

Add CI license check flow config file (#4294)

6ffc427

close #4292

make mpp_fail test stable (#4300)

f030da8

close #4299

README.md: Rename tics to tiflash. (#4296)

d385f0a

ref #4019

fix potential asan error for string tests (#4279)

946b72e

close #4280

MPP: ExchangeReceiver supports async gRPC (#3715)

de7bc99

close #4298

Replace old CK license with standard Apache 2.0 License (#4306)

0948c8a

close #4305

Fix crash when read ColumnFileTiny in MemTableSet (#4297)

59d9ed0

close #4204

cleanup benchmark (#4233)

8723b25

close #4232

Min tso metrics (#4260)

c0c63f3

close #4256

fix bug of handling zero date in dayofweek and dayofyear (#4314)

a53783d

close #4313

use the right way to wait tiflash replica ready in least_greatest.test (

19c3e48

#4317) close #4316

print table_id for table which is applying snapshot (#4310)

4e52ac8

close #4311

Scheduler: release threads by tasks (#4307)

aacd920

close #4308

Fix bug that TiFlash will crash when enable both async_grpc_client an…

fe5a3d7

…d local_tunnel (#4322) close #4321

MPP: add metrics for ExchangeReceiver (#4327)

dfc6dcd

ref #4298

test: check "," when execute wait-table (#4319)

75b2bf2

ref #4316

jiaqizho and others added 14 commits May 20, 2022 16:42

Fix StorageDeltaMerge constructor not match in gtest_manual_compact (#…

120d0a8

…4950) close #4949

Add note about how to handle build oom or hang (#4951)

ebf12fa

close #4953

Add a global max_id to fix reuse page_id problem (#4948)

4044129

close #4939

Schedule: track the waiting tasks with task ID, and deleted the sched…

26b11f9

…uled task with exceeded state from the waiting tasks queue (#4958) close #4954

Enable LTO in release build process (#4924)

f9eb43b

ref #4909

Revert the retry applyRecord when loadEdit (#4938)

2948e03

close #4937

Remove useless template instantiations (#4978)

caaff2a

ref #4909

fix clang tidy err for ExpressionAnalyzer, Join, StorageJoin, `…

1a79d29

…DAGQueryBlockInterpreter` (#4961) ref #4605

refine handleJoin (#4722)

5e55fa8

ref #4118

Change BlobFileId to uint64. (#4984)

2f845ef

ref #3594

*: Rename the function with “withCollation” postfix (#3790)

11224ce

close #4985

PageStorage: Fix the WAL snapshot dumped into file may contains inval…

5e781b2

…id "being_ref_count" (#4987) close #4957, close #4986

add more args

dc00b91

jiaqizho requested a review from JaySon-Huang May 25, 2022 06:47

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 25, 2022

jiaqizho requested a review from hehechen May 25, 2022 06:47

jiaqizho added 2 commits May 25, 2022 15:09

merge to master

a63c78e

fix 32

b80c1e2

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 25, 2022

change type

cee698a

JaySon-Huang reviewed May 25, 2022

View reviewed changes

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 26, 2022

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 26, 2022

jiaqizho closed this May 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PageStorage control add more args to debug. #4994

PageStorage control add more args to debug. #4994

jiaqizho commented May 25, 2022

ti-chi-bot commented May 25, 2022

JaySon-Huang May 25, 2022

JaySon-Huang May 25, 2022

ti-chi-bot commented May 26, 2022

	void TiFlashTestEnv::setupLogger(const String & level, std::ostream & os)
	{
	Poco::AutoPtr<Poco::ConsoleChannel> channel = new Poco::ConsoleChannel(os);
	Poco::AutoPtr<UnifiedLogPatternFormatter> formatter(new UnifiedLogPatternFormatter());
	formatter->setProperty("pattern", "%L%Y-%m-%d %H:%M:%S.%i [%I] <%p> %s: %t");
	Poco::AutoPtr<Poco::FormattingChannel> formatting_channel(new Poco::FormattingChannel(formatter, channel));
	Poco::Logger::root().setChannel(formatting_channel);
	Poco::Logger::root().setLevel(level);
	}

PageStorage control add more args to debug. #4994

PageStorage control add more args to debug. #4994

Conversation

jiaqizho commented May 25, 2022

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented May 25, 2022

JaySon-Huang May 25, 2022

Choose a reason for hiding this comment

JaySon-Huang May 25, 2022

Choose a reason for hiding this comment

ti-chi-bot commented May 26, 2022