topsql: support multiple datasinks #30808

zhongzc · 2021-12-16T09:34:26Z

Signed-off-by: Zhenchi zhongzc_arch@outlook.com

What problem does this PR solve?

Part of #29020

Problem Summary:

Now the datasink is hardcoded.

What is changed and how it works?

Add datasink registration mechanism:

Add DataSinkRegHandle method to interface TopSQLReporter. Users can access the registration handle provided by the reporter, then register a datasink.
Add registration handle logic for RemoteTopSQLReporter.

Check List

Tests

Unit test

Release note

Support multiple datasinks for TopSQL

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

ti-chi-bot · 2021-12-16T09:34:28Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

breeswish
crazycs520

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

zhongzc · 2021-12-16T09:35:07Z

/cc @breeswish @crazycs520

zhongzc · 2021-12-16T09:53:35Z

/rebuild

breezewish · 2021-12-16T10:14:48Z

util/topsql/reporter/reporter.go

@@ -50,9 +50,15 @@ type TopSQLReporter interface {
 	tracecpu.Collector
 	RegisterSQL(sqlDigest []byte, normalizedSQL string, isInternal bool)
 	RegisterPlan(planDigest []byte, normalizedPlan string)
+	DataSinkRegHandle() DataSinkRegHandle


How about letting the TopSQLReporter having RegisterDataSink directly?

breezewish

the rest LGTM

breezewish · 2021-12-16T10:25:09Z

util/topsql/reporter/reporter.go

+	for _, dataSink := range tsr.dataSinks {
+		dataSink.Close()
+	}
+	tsr.dataSinks = nil


is it safe when Close is called multiple times?

breezewish · 2021-12-16T10:27:24Z

util/topsql/reporter/reporter.go

 		select {
+		case dataSink := <-tsr.dataSinkRegCh:
+			tsr.dataSinks = append(tsr.dataSinks, dataSink)


I guess it's fine to introduce a mutex for registering and reading the data sink, as it is a very low frequent operation (and reading / modifying the data sink list is also very quick). Then you will find the design much simplified. There is no need for everything to be channel based. This also makes it possible to unregister the data sink quite easily.

It feels more comfortable for me to think of these as actor models. I am not good at figuring out the control flow of mutex models.

BTW, it's my fault to cleaning up dataSinks at RemoteTopSQLReporter.Close...

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

breezewish

The rest LGTM

breezewish · 2021-12-16T12:06:57Z

util/topsql/reporter/reporter.go

+
+		// An unbuffered channel for registration. A call for Register will block
+		// until the registration is successful or the RemoteTopSQLReporter is closed.
+		dataSinkRegCh: make(chan DataSink),


It may be blocked for a long while since there is a 100ms in the loop and the reporter will not handle multiple register for once. Is this expected in Pub/Sub mode? (You will not face such long block when using mutex).

I am adding context as a parameter. It looks better.

It's by design. If the channel is buffered and the reporter is down, it provides the possibility that the datasink will not receive any feedback, and it will never know the reporter is down.

breezewish

Seems that IsPaused is unused.. It seems to be wired.

breezewish · 2021-12-16T12:17:50Z

util/topsql/reporter/reporter.go

@@ -585,7 +649,9 @@ func (tsr *RemoteTopSQLReporter) doReport(data *ReportData) {
 		}
 	})
 	deadline := time.Now().Add(timeout)
-	if err := tsr.dataSink.TrySend(data, deadline); err != nil {
-		logutil.BgLogger().Warn("[top-sql] failed to send data to datasink", zap.Error(err))
+	for _, ds := range tsr.dataSinks {


If the datasink is paused, maybe you should skip it.

breezewish · 2021-12-16T12:20:17Z

util/topsql/reporter/reporter.go

+	idx := 0
+	for _, dataSink := range tsr.dataSinks {
+		if dataSink.IsDown() {
+			dataSink.Close()


Does this mean that dataSink's should only mark itself as down (but not clean up resources) and only clean up resource when called by Close?

I rethought just before. Close is just a notification to DataSink that the Reporter is dropping it.

zhongzc · 2021-12-16T12:52:49Z

Seems that IsPaused is unused.. It seems to be wired.

Next PR will utilize it.

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

sre-bot · 2021-12-17T09:05:43Z

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/dba539671270127b268768c143bbc6ebf4dcac1f

breezewish

I have not carefully reviewed single_target.go since it is meant to be removed soon. The rest LGTM.

util/topsql/reporter/reporter.go

breezewish · 2021-12-17T08:58:54Z

util/topsql/reporter/reporter.go

+		tsr.dataSinks.m[dataSink] = struct{}{}
+
+		if len(tsr.dataSinks.m) > 0 {
+			variable.TopSQLVariable.Enable.Store(true)


Seems that now TopSQLVariable.Enable is actually a state of the internal, instead of a controllable variable, right?

Just a transition here. No more controllable variable after a few PRs.

The profiler is expected to run when len(tsr.dataSinks.m) > 0 and stop when len(tsr.dataSinks.m) == 0, and for now the only way to control it is modifying variable.TopSQLVariable.Enable.

util/topsql/reporter/datasink.go

breezewish · 2021-12-17T09:03:18Z

util/topsql/reporter/reporter.go

+		delete(tsr.dataSinks.m, dataSink)
+
+		if len(tsr.dataSinks.m) == 0 {
+			variable.TopSQLVariable.Enable.Store(false)
+		}


Why dataSink doesn't need to call CloseDataSink? In this case the reporter is also dropping it.

Have changed CloseDataSink to OnDeregisterFromReporter. Clearer name answers this question.

util/topsql/reporter/reporter.go

util/topsql/reporter/single_target.go

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

util/topsql/reporter/reporter.go

util/topsql/reporter/datasink.go

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

zhongzc · 2021-12-17T11:56:01Z

/merge

ti-chi-bot · 2021-12-17T11:56:04Z

@zhongzc: /merge is only allowed for the committers, you can assign this pull request to the committer in list by filling /assign @committer in the comment to help merge this pull request.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

tangenta · 2021-12-17T11:59:23Z

/merge

ti-chi-bot · 2021-12-17T11:59:26Z

This pull request has been accepted and is ready to merge.

Commit hash: 553a691

ti-chi-bot · 2021-12-17T11:59:38Z

@zhongzc: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

zhongzc · 2021-12-17T13:27:26Z

/run-all-tests

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

breezewish · 2021-12-17T16:19:50Z

/merge

ti-chi-bot · 2021-12-17T16:19:54Z

This pull request has been accepted and is ready to merge.

Commit hash: dba5396

* dumpling: fix default collation with upstream when dump database and table (#30292) * ddl: fix the enum default value by triming trailing space (#30356) * expression: migrate test-infra to testify for flag_simplify_test.go (#30407) * server: refine code logic in handleDownloadFile (#30422) * refine logic Signed-off-by: yisaer <disxiaofei@163.com> * fix Signed-off-by: yisaer <disxiaofei@163.com> * ddl: migrate test-infra to testify for ddl/table_test.go (#30267) * ddl: handle the error from `addBatchDDLJobs()` correctly (#30401) * br: fix the integration tests (#30423) * util, cmd: remove unused filesort (#30438) * *: update client-go for small backoff time (#30436) * server: Fix unstable tests with FakeAuthSwitch (#30287) * dumpling: fix dump failed when sequence exists (#30164) * *: replace compareDatum by compare (#30421) * lightning: fix gcs max key limit (#30393) * expression, parser: add built-in func is_uuid (#30318) * expression: migrate test-infra to testify for constant_fold_test.go (#30424) * executor: fix pipelined window invalid memory address (#30418) * makefile: add gotestsum for verify ci (#29848) * server: close sql rows to fix unstable test (#30306) * Makefile: add coverage record for BR and Dumpling (#30457) * executor: track the mem usage of IndexMergeReader (#30210) * infosync: close body when ReadAll encounters error (#30462) * planner: show accessed partition when explain mpp query over partition table (#30367) * *: Fix use of user identity in SHOW GRANTS + error messages (#30294) * ddl: add not null flag for auto_increment column (#30477) * expression: make some unstable test serial (#30323) * expression: migrate test-infra to testify for constant_propagation_test.go (#30430) * executor: stable test TestSetDDLReorgBatchSize and TestSetDDLReorgWorkerCnt (#30480) * statistics, util/ranger: add cardinality estimation trace for `GetRowCountBy...` (#30321) * *: skip mysql client goroutine leak detection in integration ddl (#30467) * executor,util: write slow query to slow log no matter what log level (#30461) * executor: enable index_merge used in transaction. (#29875) * logutil: add testcase for SlowQueryLogger.MaxDays/MaxSize/MaxBackups (#30316) * expression: fix data race in builtin_other_vec_generated_test.go (#30503) * expression: fix data race in the collationInfo (#30490) * planner/core, session: fix error message of wrong variable scope (#30510) * lightning: support Re/ReregisterMySQL by different tls name (#30463) * executor: TestBatchGetandPointGetwithHashPartition test typo (#29669) (#29671) * mockstore: improve log to avoid panic for nil pointer (#30513) * *: replace compareDatum by compare, PR 10 (#30456) * planner: Disable dynamic partition prune mode for all non-autocommit (#27532) (#30505) * expression: change the log level of an confusing log from warn to debug (#30484) * br: Check crypter.key valid before backup (#29991) * *: replace compareDatum by compare, PR 11 (#30465) * dumpling: fix default column collation with upstream when dump table (#30531) * server: fix prepared cursor select (#30285) * executor: HashJoinExec checks the buildError even if the probeSide is empty (#30471) * parser, expression: follow mysql, increase interval precedence (#30528) * makefile: set timeout 25m for make race (#30555) * planner: fix the unstable test TestAnalyzeGlobalStatsWithOpts/2 (#30576) * expression,types: Adjusts UNIX_TIMESTAMP() for non-existing DST values (#28739) (#30405) * br: add res.Body.close to avoid leak (#30545) * lightning: add back integration test lightning_error_summary (#30547) * sessionctx/variable: small refactor (split large file) (#30511) * ddl: let `admin cancel ddl jobs` run in a new transaction (#30549) * *: Retry when placement PutBundles failed (#30590) * dumpling: delete unit test in github actions (#30562) * *: support trace plan target='estimation' statement (#30491) * expression: migrate test-infra to testify for integration_test.go (#30548) * planner: support trace for min/max eliminate (#30441) * support min/max trace Signed-off-by: yisaer <disxiaofei@163.com> * address the comment Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * br: remove cdclog in br (#30573) * *: show cmd to check if all needed histograms are loaded (#29672) * expression: clone repertoire when clone the scalar function (#30602) * *: use the real StateRemote interface implementation for cached table (#30066) * *: query failed after add index / timestamp out-of-range (#28424) (#29323) * planner: implement collecting predicate columns from logical plan (#29878) * *: show PK name when decoding the clustered index row key (#30623) * ddl/callback_test.go: migrate test-infra to testify (#30317) * *: Rename some names of placement ddl operation (#30622) * executor: fix data race in the index_lookup_hash_join (#30619) * ddl: remove unnecessary locking when adding an index (#29772) * server: try to make `TidbTestSuite` more stable (#30643) * *: Add some PD tests for placement and fix some bug found (#30621) * *: migrate sync.WaitGroup to util.WaitGroupWrapper (#30644) * planner: add trace for join eliminate rule (#30343) * executor: migrate test-infra to testify for executor/shuffle_test.go (#30514) * planner: make (*AccessPath).OnlyPointRange more succinct (#30520) * planner: add trace for join reorder (#30394) * executor: migrate test-infra to testify for executor/union_scan_test.go (#30525) * expression: make cast return error if cast binary literal to another character set (#30537) * *: update tikv client (#30670) * *: update sysutil in go.mod to fix panic when search log (#30523) * topsql: shouldn't evict the SQL meta, since the evicted SQL can be appear on Other components (TiKV) TopN records (#27050) * testify: migrate test-infra to testify for analyze_test.go (#30640) * util: replace compareDatum by compare, point part (#30575) * test: make all the tests run in serial (#30692) * statistics: add mutex for Handle.globalMap and Handle.feedback (#30550) * executor: fix regular expression in json so that it could match identifer start with '$' (#29750) * util/testkit/testkit.go: fix typo (#30638) * planner: Introduce a new global variable to control the historical statistics feature (#30646) * topsql: introduce datasink interface (#30662) * planner: unify the argument of stats functions to use SessionCtx instead of StatementContext (#30668) * metrics: fix the Max SafeTS Gap metrics (#30689) * lightning: Add source dir existence check for s3 (#30674) * golangci-lint: support durationcheck (#30027) * executor: fix data race on IndexHashJoin.cancelFunc (#30701) * sessionctx/variable: change tidb_store_limit to global only (#30522) * statistics: remove reassignment of Handle.pool in NewHandle (#30675) * br: fix some unstable unit test cases. (#30716) * bindinfo: fix the comment typo (#30616) * server: support decoding prepared string args to character_set_client (#30723) * expression: fix enum type join binary get wrong result (#30445) * cmd/explaintest: fix wrong result comparison for explain test (#30717) * parallel create tables in br * metrics: fix copr-cache metrics (#30712) * test: merge executor's serial tests to other tests (#30711) * statistics: avoid deadlock when create/drop extended stats and analyze at the same time (#30566) * ddl: add batch create table api Signed-off-by: xhe <xw897002528@gmail.com> * ddl: add unit tests Signed-off-by: xhe <xw897002528@gmail.com> * ddl: fix fmt Signed-off-by: xhe <xw897002528@gmail.com> * ddl: typo Co-authored-by: Arenatlx <ailinsilence4@gmail.com> * ddl: fix tests Signed-off-by: xhe <xw897002528@gmail.com> * ddl: rename to BatchCreateTableWithInfo Signed-off-by: xhe <xw897002528@gmail.com> * ddl: trace the error Signed-off-by: xhe <xw897002528@gmail.com> * ddl: comments Signed-off-by: xhe <xw897002528@gmail.com> * ddl: cancle the job right Signed-off-by: xhe <xw897002528@gmail.com> * ddl: cancel the job right 2 Signed-off-by: xhe <xw897002528@gmail.com> * ddl: report error if entry too large Signed-off-by: xhe <xw897002528@gmail.com> * ddl: report error when table is duplicated Signed-off-by: xhe <xw897002528@gmail.com> * ddl: go fmt Signed-off-by: xhe <xw897002528@gmail.com> * infoschema: improve batch memory perf Signed-off-by: xhe <xw897002528@gmail.com> * ddl: retain ID Signed-off-by: xhe <xw897002528@gmail.com> * sessionctx: fix the value of analyze_version when upgrading 4.x to 5.… (#30743) * ddl: reduce log frequency Signed-off-by: xhe <xw897002528@gmail.com> * ddl: fix tests Signed-off-by: xhe <xw897002528@gmail.com> * server: disable socket listener for `basicHTTPHandlerTestSuite` (#30680) * planner: support the plan cache aware of bindings (#30169) * planner: fix early set of plan's statisticsTable (#30754) * *: implement renew write lock lease for cached table (#30206) * *: Modify placement rule index to reserve some indexes for future work (#30737) * executor: add an unit test case for unreasonable invoking Close (#30696) * planner: fix wrong subquery's coercibility (#30750) * executor: add more testcases for index merge (#30497) * server: add grpc server config for a suitable behavior (#30774) * config, charset: make charset config not affected by collation config (#30572) * lightning: emit tidb log by change FilterCore to only allow matched packages (#30700) * topsql: a centralized place to generate tipb report data (#30781) * planner: add trace for partition pruning (#30546) * planner: refine collation handling for between (#30793) * test: merge serial tests in bindinfo, expression, parser and statistics (#30749) * br: update log description for split check (#30763) * *: replace compareDatum by compare, range part (#30710) * *: placement policy ref will be converted to direct options when recover or flashback table (#30705) * ddl: handle the incorrect number of placement followers (#30715) * ddl: revert "ddl: remove unnecessary locking when adding an index" (#30667) * br/pkg/task: migrate test-infra to testify (#30605) * *: fix the flen type datetime for union/case-when/control-funcs (#30588) * types, util: clean up compareDatum (#30815) * ddl: add helper function to set and query TiFlash's sync status (#30473) * dumpling: fix more dumpling log level query template (#30176) * parser: support `admin flush plan_cache` (#30747) * topsql: support multiple datasinks (#30808) * br: update permission, so tikv can write to folder when backup to local (#30396) * session: fix bootstrap to only persist global variables (#30593) close #28667 * docs/design: update collation compatibility issues in charsets doc (#30806) * executor: improve SET sysvar=DEFAULT handling (#29680) close #29670 * br: add error handling for group context cancel when restore file is corrupted (#30190) close #30135 * executor: buildWindow cannot call typeInfer twice (#30773) close #30402 * *: refactor encoding and uniform usages (#30288) * lightning: optimize region split check logic (#30428) close #30018 * br: ignore mock directory when gcov in br (#30586) * *: forbid set tiflash replica count for a placement table (#30844) close #30741 * execute: don't transform charset in internal sql (#30843) close #30789 * planner: update PlanBuilder.windowSpecs when building subquery (#30878) close #30804 * br: fix S3 backup endpoint suffix (#30530) close #30104 * lightning: make pre-check output message clearer (#30439) close #30395 * expression: wrap to_binary and from_binary for cast function's argument (#30706) * executor: fix bug when using IndexMerge in transaction (#30719) close #30685 * ddl: migrate test-infra to testify for ddl/foreign_key_test.go (#30853) close #29103 * expression: fix wrong retType for reverse function (#30829) close #30809 * planner: support trace topn push down (#30800) ref #29661 * github: add issue requirement to pull request template (#30817) close #30814 * fix merge issue * topsql: introduce stmtstats and sql execution count (#30277) * topsql: add pubsub datasink (#30860) * executor: fix the incorrect untouch used in optimistic transactions (#30447) close #30410 * expression, cmd: let crc32() support gbk (#30900) close #30898 * server: Add uptime status var and statistics (#29790) close #8842 * br: error log optimization (#29640) close #27015 * planner: fix wrong collation when rewrite in condition (#30492) close #30486 * planner: add extractor for tikv_region_peers (#30656) * fix issue that loss table restore * lightning: add back table empty check and add a switch config (#30887) close #27919 * br: improve backoff unit test (#30892) * *: add TxnManager to manage txn in session (#30574) * *: add TxnManager to manage txn in session * modify * add tests * move failpoint content to a single file * Makefile: add `t.Parallel` check to ensure tests are run in serial (#30869) * refactoring code * refactoring code * placement: remove isolationlevel (#30859) close #30858 * planner: revise the optimize trace output (#30882) * table: set the datum collation correctly in CastValue() (#30931) close #30930 * *: Use TxnManager.GetTxnInfoSchema() to get the txn infoschema (#30934) close #30933 * parser: add IsValid() to Encoding to speed up string validation for UTF-8 (#30937) close #30936 * planner: rename pstmtPlanCacheXX to PlanCacheXX (#30909) * table/tables: make CI TestCacheTableBasicReadAndWrite more stable (#30924) close #30922 * restore: use new ScatterRegions API (#30899) close #30425 * *: when placement conflicts with tiflash, cancel the job (#30945) * Makefile,tools: make CI great again! (#30828) close #30822 * br/pkg/membuf: remove global buffer pool (#29934) * ddl: add format error for incorrect dict syntax in the placement rule (#30919) close #30454 * planner: fix index merge plan when expr cannot be pushed to tikv (#30341) close #30200 * executor: display 'show create table' and INFOSCHEMA for cached table correctly (#30951) close #30950 * br: extend the timeout for scan region since 3 seconds is not enough (#30889) close #30720 * planner: remove bindSQL from planCacheKey to planCacheValue (#30916) * execution: refine precision of cast as decimal in agg func (#30805) * *: fix data race in the tikv_client (#30964) close #30658 * ddl: migrate test-infra to testify for ddl/db_partition_test.go (#30952) close #28635 * planner: fix `AccessPath.TableFilters` got modified unexpectedly (#30966) close #30965 * test: merge serial tests in ddl, infoschema, session, store, table, telemetry and types (#30874) * executor: fix the returned field count of the prepare statement (#30981) close #30971 * binlog: allow multiple ddl targets (#30904) * planner: trace predicate push down (#30902) ref #29661 * placement: give default 2 followers for non-sugar syntax (#31000) * flatten the json output (#30905) Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * test: control log level with environment variables (#30871) * planner: add usage of paging copr in optimizer (#30536) close #30578 * test: merge serial tests in cmd, planner, server, util (#31003) * planner: change predicateColumnCollector to columnStatsUsageCollector and collect histogram-needed columns (#30671) * executor: migrate test-infra to testify for distsql_test.go (#31023) close #28574 * remote uncessary package errors * reused the retry code from lightning * refactoring retryable * ddl: add batch create table api Signed-off-by: xhe <xw897002528@gmail.com> * ddl: add unit tests Signed-off-by: xhe <xw897002528@gmail.com> * ddl: fix fmt Signed-off-by: xhe <xw897002528@gmail.com> * br ddl code * parallel create tables in br * ddl: add batch create table api Signed-off-by: xhe <xw897002528@gmail.com> * ddl: add unit tests Signed-off-by: xhe <xw897002528@gmail.com> * ddl: fix fmt Signed-off-by: xhe <xw897002528@gmail.com> * ddl: typo Co-authored-by: Arenatlx <ailinsilence4@gmail.com> * ddl: fix tests Signed-off-by: xhe <xw897002528@gmail.com> * ddl: rename to BatchCreateTableWithInfo Signed-off-by: xhe <xw897002528@gmail.com> * ddl: trace the error Signed-off-by: xhe <xw897002528@gmail.com> * ddl: comments Signed-off-by: xhe <xw897002528@gmail.com> * ddl: cancle the job right Signed-off-by: xhe <xw897002528@gmail.com> * ddl: cancel the job right 2 Signed-off-by: xhe <xw897002528@gmail.com> * ddl: report error if entry too large Signed-off-by: xhe <xw897002528@gmail.com> * ddl: report error when table is duplicated Signed-off-by: xhe <xw897002528@gmail.com> * ddl: go fmt Signed-off-by: xhe <xw897002528@gmail.com> * infoschema: improve batch memory perf Signed-off-by: xhe <xw897002528@gmail.com> * ddl: retain ID Signed-off-by: xhe <xw897002528@gmail.com> * ddl: reduce log frequency Signed-off-by: xhe <xw897002528@gmail.com> * ddl: fix tests Signed-off-by: xhe <xw897002528@gmail.com> * ddl: remove retainID from the interface Signed-off-by: xhe <xw897002528@gmail.com> * ddl: fix tests Signed-off-by: xhe <xw897002528@gmail.com> * executor: fix rebasing problem Signed-off-by: xhe <xw897002528@gmail.com> Co-authored-by: WizardXiao <89761062+WizardXiao@users.noreply.github.com> Co-authored-by: sylzd <liuzidong@xiaomi.com> Co-authored-by: tison <wander4096@gmail.com> Co-authored-by: Song Gao <disxiaofei@163.com> Co-authored-by: Weizhen Wang <wangweizhen@pingcap.com> Co-authored-by: tangenta <tangenta@126.com> Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: wjHuang <huangwenjun1997@gmail.com> Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> Co-authored-by: Daniël van Eeden <git@myname.nl> Co-authored-by: Jianjun Liao <36503113+Leavrth@users.noreply.github.com> Co-authored-by: unconsolable <chenzhipeng2012@gmail.com> Co-authored-by: Shenghui Wu <793703860@qq.com> Co-authored-by: guo-shaoge <shaoge1994@163.com> Co-authored-by: Ryan Leung <rleungx@gmail.com> Co-authored-by: xufei <xufei@pingcap.com> Co-authored-by: Morgan Tocker <tocker@gmail.com> Co-authored-by: Zhou Kunqin <25057648+time-and-fate@users.noreply.github.com> Co-authored-by: 王超 <cclcwangchao@hotmail.com> Co-authored-by: TonsnakeLin <87681388+TonsnakeLin@users.noreply.github.com> Co-authored-by: Ehco <zh19960202@gmail.com> Co-authored-by: Mattias Jonsson <mjonss@users.noreply.github.com> Co-authored-by: HuaiyuXu <391585975@qq.com> Co-authored-by: Zak Zhao <57036248+joccau@users.noreply.github.com> Co-authored-by: xhe <xw897002528@gmail.com> Co-authored-by: Hangjie Mo <mohangjie1995@gmail.com> Co-authored-by: Yuanjia Zhang <zhangyuanjia@pingcap.com> Co-authored-by: glorv <glorvs@163.com> Co-authored-by: djshow832 <zhangming@pingcap.com> Co-authored-by: Chunzhu Li <lichunzhu@stu.xjtu.edu.cn> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Co-authored-by: Xiaoju Wu <wuxiaoju@pingcap.com> Co-authored-by: xiongjiwei <xiongjiwei1996@outlook.com> Co-authored-by: tiancaiamao <tiancaiamao@gmail.com> Co-authored-by: Yifan Xu <30385241+xuyifangreeneyes@users.noreply.github.com> Co-authored-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Zach <51114270+zach030@users.noreply.github.com> Co-authored-by: bb7133 <bb7133@gmail.com> Co-authored-by: lvtu <37565148+tongtongyin@users.noreply.github.com> Co-authored-by: crazycs <crazycs520@gmail.com> Co-authored-by: znhh6018 <44599853+znhh6018@users.noreply.github.com> Co-authored-by: eddie lin <eddievim@foxmail.com> Co-authored-by: dongjunduo <andj4cn@gmail.com> Co-authored-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: wangggong <793160615@qq.com> Co-authored-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> Co-authored-by: Jack Yu <yusp@pingcap.com> Co-authored-by: Arenatlx <ailinsilence4@gmail.com> Co-authored-by: Yiding Cui <winoros@gmail.com> Co-authored-by: Chengpeng Yan <41809508+Reminiscent@users.noreply.github.com> Co-authored-by: bestwoody <89765764+bestwoody@users.noreply.github.com> Co-authored-by: Calvin Neo <CalvinNeo@users.noreply.github.com> Co-authored-by: Lynn <zimu_xia@126.com> Co-authored-by: Zhuhe Fang <fzhedu@gmail.com> Co-authored-by: Mini256 <minianter@foxmail.com> Co-authored-by: Xiang Zhang <angwerzx@126.com> Co-authored-by: Yexiang Zhang <mornyx.z@gmail.com> Co-authored-by: cfzjywxk <lsswxrxr@163.com> Co-authored-by: db <39407623+IcePigZDB@users.noreply.github.com> Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com> Co-authored-by: Yujie Xia <xiayjchn@gmail.com> Co-authored-by: Yilong Li <liyilongko@gmail.com> Co-authored-by: tuuuuuu <83738345+MiaoMiaoGarden@users.noreply.github.com> Co-authored-by: qupeng <qupeng@pingcap.com> Co-authored-by: you06 <you1474600@gmail.com>

* br: fix the integration tests (#30423) * util, cmd: remove unused filesort (#30438) * *: update client-go for small backoff time (#30436) * server: Fix unstable tests with FakeAuthSwitch (#30287) * dumpling: fix dump failed when sequence exists (#30164) * *: replace compareDatum by compare (#30421) * lightning: fix gcs max key limit (#30393) * expression, parser: add built-in func is_uuid (#30318) * expression: migrate test-infra to testify for constant_fold_test.go (#30424) * executor: fix pipelined window invalid memory address (#30418) * makefile: add gotestsum for verify ci (#29848) * server: close sql rows to fix unstable test (#30306) * Makefile: add coverage record for BR and Dumpling (#30457) * executor: track the mem usage of IndexMergeReader (#30210) * infosync: close body when ReadAll encounters error (#30462) * planner: show accessed partition when explain mpp query over partition table (#30367) * *: Fix use of user identity in SHOW GRANTS + error messages (#30294) * ddl: add not null flag for auto_increment column (#30477) * expression: make some unstable test serial (#30323) * expression: migrate test-infra to testify for constant_propagation_test.go (#30430) * executor: stable test TestSetDDLReorgBatchSize and TestSetDDLReorgWorkerCnt (#30480) * statistics, util/ranger: add cardinality estimation trace for `GetRowCountBy...` (#30321) * *: skip mysql client goroutine leak detection in integration ddl (#30467) * executor,util: write slow query to slow log no matter what log level (#30461) * executor: enable index_merge used in transaction. (#29875) * logutil: add testcase for SlowQueryLogger.MaxDays/MaxSize/MaxBackups (#30316) * expression: fix data race in builtin_other_vec_generated_test.go (#30503) * expression: fix data race in the collationInfo (#30490) * planner/core, session: fix error message of wrong variable scope (#30510) * lightning: support Re/ReregisterMySQL by different tls name (#30463) * executor: TestBatchGetandPointGetwithHashPartition test typo (#29669) (#29671) * mockstore: improve log to avoid panic for nil pointer (#30513) * *: replace compareDatum by compare, PR 10 (#30456) * planner: Disable dynamic partition prune mode for all non-autocommit (#27532) (#30505) * expression: change the log level of an confusing log from warn to debug (#30484) * br: Check crypter.key valid before backup (#29991) * *: replace compareDatum by compare, PR 11 (#30465) * dumpling: fix default column collation with upstream when dump table (#30531) * server: fix prepared cursor select (#30285) * executor: HashJoinExec checks the buildError even if the probeSide is empty (#30471) * parser, expression: follow mysql, increase interval precedence (#30528) * makefile: set timeout 25m for make race (#30555) * planner: fix the unstable test TestAnalyzeGlobalStatsWithOpts/2 (#30576) * expression,types: Adjusts UNIX_TIMESTAMP() for non-existing DST values (#28739) (#30405) * br: add res.Body.close to avoid leak (#30545) * lightning: add back integration test lightning_error_summary (#30547) * sessionctx/variable: small refactor (split large file) (#30511) * ddl: let `admin cancel ddl jobs` run in a new transaction (#30549) * *: Retry when placement PutBundles failed (#30590) * dumpling: delete unit test in github actions (#30562) * *: support trace plan target='estimation' statement (#30491) * expression: migrate test-infra to testify for integration_test.go (#30548) * planner: support trace for min/max eliminate (#30441) * support min/max trace Signed-off-by: yisaer <disxiaofei@163.com> * address the comment Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * br: remove cdclog in br (#30573) * *: show cmd to check if all needed histograms are loaded (#29672) * expression: clone repertoire when clone the scalar function (#30602) * *: use the real StateRemote interface implementation for cached table (#30066) * *: query failed after add index / timestamp out-of-range (#28424) (#29323) * planner: implement collecting predicate columns from logical plan (#29878) * *: show PK name when decoding the clustered index row key (#30623) * ddl/callback_test.go: migrate test-infra to testify (#30317) * *: Rename some names of placement ddl operation (#30622) * executor: fix data race in the index_lookup_hash_join (#30619) * ddl: remove unnecessary locking when adding an index (#29772) * server: try to make `TidbTestSuite` more stable (#30643) * *: Add some PD tests for placement and fix some bug found (#30621) * *: migrate sync.WaitGroup to util.WaitGroupWrapper (#30644) * planner: add trace for join eliminate rule (#30343) * executor: migrate test-infra to testify for executor/shuffle_test.go (#30514) * planner: make (*AccessPath).OnlyPointRange more succinct (#30520) * planner: add trace for join reorder (#30394) * executor: migrate test-infra to testify for executor/union_scan_test.go (#30525) * expression: make cast return error if cast binary literal to another character set (#30537) * *: update tikv client (#30670) * *: update sysutil in go.mod to fix panic when search log (#30523) * topsql: shouldn't evict the SQL meta, since the evicted SQL can be appear on Other components (TiKV) TopN records (#27050) * testify: migrate test-infra to testify for analyze_test.go (#30640) * util: replace compareDatum by compare, point part (#30575) * test: make all the tests run in serial (#30692) * statistics: add mutex for Handle.globalMap and Handle.feedback (#30550) * executor: fix regular expression in json so that it could match identifer start with '$' (#29750) * util/testkit/testkit.go: fix typo (#30638) * planner: Introduce a new global variable to control the historical statistics feature (#30646) * topsql: introduce datasink interface (#30662) * planner: unify the argument of stats functions to use SessionCtx instead of StatementContext (#30668) * metrics: fix the Max SafeTS Gap metrics (#30689) * lightning: Add source dir existence check for s3 (#30674) * golangci-lint: support durationcheck (#30027) * executor: fix data race on IndexHashJoin.cancelFunc (#30701) * sessionctx/variable: change tidb_store_limit to global only (#30522) * statistics: remove reassignment of Handle.pool in NewHandle (#30675) * br: fix some unstable unit test cases. (#30716) * bindinfo: fix the comment typo (#30616) * server: support decoding prepared string args to character_set_client (#30723) * expression: fix enum type join binary get wrong result (#30445) * cmd/explaintest: fix wrong result comparison for explain test (#30717) * parallel create tables in br * metrics: fix copr-cache metrics (#30712) * test: merge executor's serial tests to other tests (#30711) * statistics: avoid deadlock when create/drop extended stats and analyze at the same time (#30566) * ddl: add batch create table api Signed-off-by: xhe <xw897002528@gmail.com> * ddl: add unit tests Signed-off-by: xhe <xw897002528@gmail.com> * ddl: fix fmt Signed-off-by: xhe <xw897002528@gmail.com> * ddl: typo Co-authored-by: Arenatlx <ailinsilence4@gmail.com> * ddl: fix tests Signed-off-by: xhe <xw897002528@gmail.com> * ddl: rename to BatchCreateTableWithInfo Signed-off-by: xhe <xw897002528@gmail.com> * ddl: trace the error Signed-off-by: xhe <xw897002528@gmail.com> * ddl: comments Signed-off-by: xhe <xw897002528@gmail.com> * ddl: cancle the job right Signed-off-by: xhe <xw897002528@gmail.com> * ddl: cancel the job right 2 Signed-off-by: xhe <xw897002528@gmail.com> * ddl: report error if entry too large Signed-off-by: xhe <xw897002528@gmail.com> * ddl: report error when table is duplicated Signed-off-by: xhe <xw897002528@gmail.com> * ddl: go fmt Signed-off-by: xhe <xw897002528@gmail.com> * infoschema: improve batch memory perf Signed-off-by: xhe <xw897002528@gmail.com> * ddl: retain ID Signed-off-by: xhe <xw897002528@gmail.com> * sessionctx: fix the value of analyze_version when upgrading 4.x to 5.… (#30743) * ddl: reduce log frequency Signed-off-by: xhe <xw897002528@gmail.com> * ddl: fix tests Signed-off-by: xhe <xw897002528@gmail.com> * server: disable socket listener for `basicHTTPHandlerTestSuite` (#30680) * planner: support the plan cache aware of bindings (#30169) * planner: fix early set of plan's statisticsTable (#30754) * *: implement renew write lock lease for cached table (#30206) * *: Modify placement rule index to reserve some indexes for future work (#30737) * executor: add an unit test case for unreasonable invoking Close (#30696) * planner: fix wrong subquery's coercibility (#30750) * executor: add more testcases for index merge (#30497) * server: add grpc server config for a suitable behavior (#30774) * config, charset: make charset config not affected by collation config (#30572) * lightning: emit tidb log by change FilterCore to only allow matched packages (#30700) * topsql: a centralized place to generate tipb report data (#30781) * planner: add trace for partition pruning (#30546) * planner: refine collation handling for between (#30793) * test: merge serial tests in bindinfo, expression, parser and statistics (#30749) * br: update log description for split check (#30763) * *: replace compareDatum by compare, range part (#30710) * *: placement policy ref will be converted to direct options when recover or flashback table (#30705) * ddl: handle the incorrect number of placement followers (#30715) * ddl: revert "ddl: remove unnecessary locking when adding an index" (#30667) * br/pkg/task: migrate test-infra to testify (#30605) * *: fix the flen type datetime for union/case-when/control-funcs (#30588) * types, util: clean up compareDatum (#30815) * ddl: add helper function to set and query TiFlash's sync status (#30473) * dumpling: fix more dumpling log level query template (#30176) * parser: support `admin flush plan_cache` (#30747) * topsql: support multiple datasinks (#30808) * br: update permission, so tikv can write to folder when backup to local (#30396) * session: fix bootstrap to only persist global variables (#30593) close #28667 * docs/design: update collation compatibility issues in charsets doc (#30806) * executor: improve SET sysvar=DEFAULT handling (#29680) close #29670 * br: add error handling for group context cancel when restore file is corrupted (#30190) close #30135 * executor: buildWindow cannot call typeInfer twice (#30773) close #30402 * *: refactor encoding and uniform usages (#30288) * lightning: optimize region split check logic (#30428) close #30018 * br: ignore mock directory when gcov in br (#30586) * *: forbid set tiflash replica count for a placement table (#30844) close #30741 * execute: don't transform charset in internal sql (#30843) close #30789 * planner: update PlanBuilder.windowSpecs when building subquery (#30878) close #30804 * br: fix S3 backup endpoint suffix (#30530) close #30104 * lightning: make pre-check output message clearer (#30439) close #30395 * expression: wrap to_binary and from_binary for cast function's argument (#30706) * executor: fix bug when using IndexMerge in transaction (#30719) close #30685 * ddl: migrate test-infra to testify for ddl/foreign_key_test.go (#30853) close #29103 * expression: fix wrong retType for reverse function (#30829) close #30809 * planner: support trace topn push down (#30800) ref #29661 * github: add issue requirement to pull request template (#30817) close #30814 * fix merge issue * topsql: introduce stmtstats and sql execution count (#30277) * topsql: add pubsub datasink (#30860) * executor: fix the incorrect untouch used in optimistic transactions (#30447) close #30410 * expression, cmd: let crc32() support gbk (#30900) close #30898 * server: Add uptime status var and statistics (#29790) close #8842 * br: error log optimization (#29640) close #27015 * planner: fix wrong collation when rewrite in condition (#30492) close #30486 * planner: add extractor for tikv_region_peers (#30656) * fix issue that loss table restore * lightning: add back table empty check and add a switch config (#30887) close #27919 * br: improve backoff unit test (#30892) * *: add TxnManager to manage txn in session (#30574) * *: add TxnManager to manage txn in session * modify * add tests * move failpoint content to a single file * Makefile: add `t.Parallel` check to ensure tests are run in serial (#30869) * refactoring code * refactoring code * placement: remove isolationlevel (#30859) close #30858 * planner: revise the optimize trace output (#30882) * table: set the datum collation correctly in CastValue() (#30931) close #30930 * *: Use TxnManager.GetTxnInfoSchema() to get the txn infoschema (#30934) close #30933 * parser: add IsValid() to Encoding to speed up string validation for UTF-8 (#30937) close #30936 * planner: rename pstmtPlanCacheXX to PlanCacheXX (#30909) * table/tables: make CI TestCacheTableBasicReadAndWrite more stable (#30924) close #30922 * restore: use new ScatterRegions API (#30899) close #30425 * *: when placement conflicts with tiflash, cancel the job (#30945) * Makefile,tools: make CI great again! (#30828) close #30822 * br/pkg/membuf: remove global buffer pool (#29934) * ddl: add format error for incorrect dict syntax in the placement rule (#30919) close #30454 * planner: fix index merge plan when expr cannot be pushed to tikv (#30341) close #30200 * executor: display 'show create table' and INFOSCHEMA for cached table correctly (#30951) close #30950 * br: extend the timeout for scan region since 3 seconds is not enough (#30889) close #30720 * planner: remove bindSQL from planCacheKey to planCacheValue (#30916) * execution: refine precision of cast as decimal in agg func (#30805) * *: fix data race in the tikv_client (#30964) close #30658 * ddl: migrate test-infra to testify for ddl/db_partition_test.go (#30952) close #28635 * planner: fix `AccessPath.TableFilters` got modified unexpectedly (#30966) close #30965 * test: merge serial tests in ddl, infoschema, session, store, table, telemetry and types (#30874) * executor: fix the returned field count of the prepare statement (#30981) close #30971 * binlog: allow multiple ddl targets (#30904) * planner: trace predicate push down (#30902) ref #29661 * placement: give default 2 followers for non-sugar syntax (#31000) * flatten the json output (#30905) Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * test: control log level with environment variables (#30871) * planner: add usage of paging copr in optimizer (#30536) close #30578 * test: merge serial tests in cmd, planner, server, util (#31003) * planner: change predicateColumnCollector to columnStatsUsageCollector and collect histogram-needed columns (#30671) * executor: migrate test-infra to testify for distsql_test.go (#31023) close #28574 * remote uncessary package errors * reused the retry code from lightning * refactoring retryable * ddl: add batch create table api Signed-off-by: xhe <xw897002528@gmail.com> * ddl: add unit tests Signed-off-by: xhe <xw897002528@gmail.com> * ddl: fix fmt Signed-off-by: xhe <xw897002528@gmail.com> * br ddl code * parallel create tables in br * ddl: add batch create table api Signed-off-by: xhe <xw897002528@gmail.com> * ddl: add unit tests Signed-off-by: xhe <xw897002528@gmail.com> * ddl: fix fmt Signed-off-by: xhe <xw897002528@gmail.com> * ddl: typo Co-authored-by: Arenatlx <ailinsilence4@gmail.com> * ddl: fix tests Signed-off-by: xhe <xw897002528@gmail.com> * ddl: rename to BatchCreateTableWithInfo Signed-off-by: xhe <xw897002528@gmail.com> * ddl: trace the error Signed-off-by: xhe <xw897002528@gmail.com> * ddl: comments Signed-off-by: xhe <xw897002528@gmail.com> * ddl: cancle the job right Signed-off-by: xhe <xw897002528@gmail.com> * ddl: cancel the job right 2 Signed-off-by: xhe <xw897002528@gmail.com> * ddl: report error if entry too large Signed-off-by: xhe <xw897002528@gmail.com> * ddl: report error when table is duplicated Signed-off-by: xhe <xw897002528@gmail.com> * ddl: go fmt Signed-off-by: xhe <xw897002528@gmail.com> * infoschema: improve batch memory perf Signed-off-by: xhe <xw897002528@gmail.com> * ddl: retain ID Signed-off-by: xhe <xw897002528@gmail.com> * ddl: reduce log frequency Signed-off-by: xhe <xw897002528@gmail.com> * ddl: fix tests Signed-off-by: xhe <xw897002528@gmail.com> * ddl: remove retainID from the interface Signed-off-by: xhe <xw897002528@gmail.com> * ddl: fix tests Signed-off-by: xhe <xw897002528@gmail.com> * executor: fix rebasing problem Signed-off-by: xhe <xw897002528@gmail.com> * sessionctx: enable IndexMerge by default (#30650) close #29597 * br: Enable lint `gosec` in br (#30895) close #30699 * planner: support 'admin flush plan cache' (#30370) * merge from tidb batch_1 Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: wjHuang <huangwenjun1997@gmail.com> Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> Co-authored-by: Daniël van Eeden <git@myname.nl> Co-authored-by: sylzd <liuzidong@xiaomi.com> Co-authored-by: Jianjun Liao <36503113+Leavrth@users.noreply.github.com> Co-authored-by: unconsolable <chenzhipeng2012@gmail.com> Co-authored-by: tison <wander4096@gmail.com> Co-authored-by: Shenghui Wu <793703860@qq.com> Co-authored-by: tangenta <tangenta@126.com> Co-authored-by: Weizhen Wang <wangweizhen@pingcap.com> Co-authored-by: guo-shaoge <shaoge1994@163.com> Co-authored-by: Ryan Leung <rleungx@gmail.com> Co-authored-by: xufei <xufei@pingcap.com> Co-authored-by: Morgan Tocker <tocker@gmail.com> Co-authored-by: Zhou Kunqin <25057648+time-and-fate@users.noreply.github.com> Co-authored-by: 王超 <cclcwangchao@hotmail.com> Co-authored-by: TonsnakeLin <87681388+TonsnakeLin@users.noreply.github.com> Co-authored-by: Ehco <zh19960202@gmail.com> Co-authored-by: Mattias Jonsson <mjonss@users.noreply.github.com> Co-authored-by: HuaiyuXu <391585975@qq.com> Co-authored-by: Zak Zhao <57036248+joccau@users.noreply.github.com> Co-authored-by: WizardXiao <89761062+WizardXiao@users.noreply.github.com> Co-authored-by: xhe <xw897002528@gmail.com> Co-authored-by: Hangjie Mo <mohangjie1995@gmail.com> Co-authored-by: Yuanjia Zhang <zhangyuanjia@pingcap.com> Co-authored-by: glorv <glorvs@163.com> Co-authored-by: djshow832 <zhangming@pingcap.com> Co-authored-by: Chunzhu Li <lichunzhu@stu.xjtu.edu.cn> Co-authored-by: Song Gao <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Co-authored-by: Xiaoju Wu <wuxiaoju@pingcap.com> Co-authored-by: xiongjiwei <xiongjiwei1996@outlook.com> Co-authored-by: tiancaiamao <tiancaiamao@gmail.com> Co-authored-by: Yifan Xu <30385241+xuyifangreeneyes@users.noreply.github.com> Co-authored-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Zach <51114270+zach030@users.noreply.github.com> Co-authored-by: bb7133 <bb7133@gmail.com> Co-authored-by: lvtu <37565148+tongtongyin@users.noreply.github.com> Co-authored-by: crazycs <crazycs520@gmail.com> Co-authored-by: znhh6018 <44599853+znhh6018@users.noreply.github.com> Co-authored-by: eddie lin <eddievim@foxmail.com> Co-authored-by: dongjunduo <andj4cn@gmail.com> Co-authored-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: wangggong <793160615@qq.com> Co-authored-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> Co-authored-by: Jack Yu <yusp@pingcap.com> Co-authored-by: Arenatlx <ailinsilence4@gmail.com> Co-authored-by: Yiding Cui <winoros@gmail.com> Co-authored-by: Chengpeng Yan <41809508+Reminiscent@users.noreply.github.com> Co-authored-by: bestwoody <89765764+bestwoody@users.noreply.github.com> Co-authored-by: Calvin Neo <CalvinNeo@users.noreply.github.com> Co-authored-by: Lynn <zimu_xia@126.com> Co-authored-by: Zhuhe Fang <fzhedu@gmail.com> Co-authored-by: Mini256 <minianter@foxmail.com> Co-authored-by: Xiang Zhang <angwerzx@126.com> Co-authored-by: Yexiang Zhang <mornyx.z@gmail.com> Co-authored-by: cfzjywxk <lsswxrxr@163.com> Co-authored-by: db <39407623+IcePigZDB@users.noreply.github.com> Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com> Co-authored-by: Yujie Xia <xiayjchn@gmail.com> Co-authored-by: Yilong Li <liyilongko@gmail.com> Co-authored-by: tuuuuuu <83738345+MiaoMiaoGarden@users.noreply.github.com> Co-authored-by: qupeng <qupeng@pingcap.com> Co-authored-by: you06 <you1474600@gmail.com>

topsql: support multiple datasinks

aa14874

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 16, 2021

ti-chi-bot requested review from breezewish and crazycs520 December 16, 2021 09:35

crazycs520 approved these changes Dec 16, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 16, 2021

Merge branch 'master' into topsql-datasinks

5702e74

breezewish reviewed Dec 16, 2021

View reviewed changes

zhongzc added 4 commits December 16, 2021 19:35

address comments

1bf3c02

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

fix

68d1bc8

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Close may be called multiple times, fix potential data race

05c0558

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Merge branch 'master' into topsql-datasinks

49d4a32

breezewish approved these changes Dec 16, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 16, 2021

breezewish reviewed Dec 16, 2021

View reviewed changes

zhongzc added 2 commits December 16, 2021 21:12

add ctx

cda796d

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

add register&deregister, impl for RemoteTopSQLReporter using mutex

be968a6

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

zhongzc requested a review from a team as a code owner December 17, 2021 08:23

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 17, 2021

polish

bf4dcc9

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

simplify SingleTargetDataSink.run

a19d7b6

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

breezewish reviewed Dec 17, 2021

View reviewed changes

zhongzc added 2 commits December 17, 2021 17:44

lock stuff

e215847

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

fix lint

6a90db5

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

breezewish reviewed Dec 17, 2021

View reviewed changes

util/topsql/reporter/reporter.go Outdated Show resolved Hide resolved

util/topsql/reporter/reporter.go Outdated Show resolved Hide resolved

util/topsql/reporter/reporter.go Outdated Show resolved Hide resolved

util/topsql/reporter/datasink.go Outdated Show resolved Hide resolved

address comments

553a691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

breezewish approved these changes Dec 17, 2021

View reviewed changes

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 17, 2021

Merge branch 'master' into topsql-datasinks

0421eca

dont touch tidb_vars.go

dba5396

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Dec 17, 2021

breezewish approved these changes Dec 17, 2021

View reviewed changes

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 17, 2021

ti-chi-bot merged commit c0888f5 into pingcap:master Dec 17, 2021

zhongzc deleted the topsql-datasinks branch December 17, 2021 16:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

topsql: support multiple datasinks #30808

topsql: support multiple datasinks #30808

zhongzc commented Dec 16, 2021

ti-chi-bot commented Dec 16, 2021 •

edited

Loading

zhongzc commented Dec 16, 2021

zhongzc commented Dec 16, 2021

breezewish Dec 16, 2021 •

edited

Loading

breezewish left a comment

breezewish Dec 16, 2021

breezewish Dec 16, 2021

zhongzc Dec 16, 2021

zhongzc Dec 16, 2021

breezewish left a comment

breezewish Dec 16, 2021 •

edited

Loading

zhongzc Dec 16, 2021

zhongzc Dec 16, 2021

breezewish left a comment

breezewish Dec 16, 2021

breezewish Dec 16, 2021 •

edited

Loading

zhongzc Dec 16, 2021

zhongzc commented Dec 16, 2021

sre-bot commented Dec 17, 2021 •

edited

Loading

breezewish left a comment

breezewish Dec 17, 2021

zhongzc Dec 17, 2021

breezewish Dec 17, 2021

zhongzc Dec 17, 2021

zhongzc commented Dec 17, 2021

ti-chi-bot commented Dec 17, 2021

tangenta commented Dec 17, 2021

ti-chi-bot commented Dec 17, 2021

ti-chi-bot commented Dec 17, 2021

zhongzc commented Dec 17, 2021

breezewish commented Dec 17, 2021

ti-chi-bot commented Dec 17, 2021

topsql: support multiple datasinks #30808

topsql: support multiple datasinks #30808

Conversation

zhongzc commented Dec 16, 2021

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Dec 16, 2021 • edited Loading

zhongzc commented Dec 16, 2021

zhongzc commented Dec 16, 2021

breezewish Dec 16, 2021 • edited Loading

Choose a reason for hiding this comment

breezewish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

breezewish left a comment

Choose a reason for hiding this comment

breezewish Dec 16, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

breezewish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

breezewish Dec 16, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhongzc commented Dec 16, 2021

sre-bot commented Dec 17, 2021 • edited Loading

breezewish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhongzc commented Dec 17, 2021

ti-chi-bot commented Dec 17, 2021

tangenta commented Dec 17, 2021

ti-chi-bot commented Dec 17, 2021

ti-chi-bot commented Dec 17, 2021

zhongzc commented Dec 17, 2021

breezewish commented Dec 17, 2021

ti-chi-bot commented Dec 17, 2021

ti-chi-bot commented Dec 16, 2021 •

edited

Loading

breezewish Dec 16, 2021 •

edited

Loading

breezewish Dec 16, 2021 •

edited

Loading

breezewish Dec 16, 2021 •

edited

Loading

sre-bot commented Dec 17, 2021 •

edited

Loading