txn: break dependency from store/tikv to tidb/kv cause by TransactionOption #24656

longfangsong · 2021-05-14T04:12:05Z

What problem does this PR solve?

Problem Summary:

For a standalone client-go, store/tikv should not depend on tidb/kv. But TransactionOption break this promise.

What is changed and how it works?

What's Changed:

Just move TransactionOption to the tikv package, and break the dependency.

Also rename TransactionOption, which IMO is easy to cause misunderstanding, to StartTSOption.

Check List

Tests

Unit test

Release note

No release note.

longfangsong · 2021-05-14T04:34:50Z

/cc @disksing @tisonkun

longfangsong · 2021-05-14T04:36:02Z

PS. Can we add a check to CI to prevent this from happening again?

tisonkun

LGTM

ti-chi-bot · 2021-05-14T04:50:23Z

@tisonkun: /lgtm is only allowed for the reviewers in list.

In response to this:

LGTM

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

tisonkun · 2021-05-14T04:50:59Z

PS. Can we add a check to CI to prevent this from happening again?

what do you mean by this? I think after we extract Go client from TiDB it should naturally not happen again.

longfangsong · 2021-05-14T04:57:06Z

PS. Can we add a check to CI to prevent this from happening again?

what do you mean by this? I think after we extract Go client from TiDB it should naturally not happen again.

I mean to prevent a new PR accidently make store/tikv depend on tidb/kv.

tisonkun · 2021-05-14T05:08:38Z

@disksing is there a timely bound for #22513 getting down? If we can get it down in weeks, @longfangsong we can avoid add a temporary check.

To be clear, all code under store/tikv will be moved to https://github.com/tikv/client-go and replace it. We can add a checker there to prevent depend on pingcap/tidb.

disksing · 2021-05-14T08:15:54Z

@disksing is there a timely bound for #22513 getting down? If we can get it down in weeks, @longfangsong we can avoid add a temporary check.

To be clear, all code under store/tikv will be moved to https://github.com/tikv/client-go and replace it. We can add a checker there to prevent depend on pingcap/tidb.

I'll add a script in makefile after this issue is fixed.

store/tikv/kv.go

store/tikv/txn.go

ti-chi-bot · 2021-05-15T04:06:11Z

@disksing: /lgtm is only allowed for the reviewers in list.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

tisonkun · 2021-05-15T04:28:46Z

FAIL: pessimistic_test.go:2184: testPessimisticSuite.TestAmendForUniqueIndex

Please fix test failure.

tisonkun

LGTM

ti-chi-bot · 2021-05-17T03:45:28Z

@tisonkun: /lgtm is only allowed for the reviewers in list.

In response to this:

LGTM

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

tisonkun · 2021-05-17T06:24:02Z

/cc @morgo @xhebox @crazycs520 @XuHuaiyu

xhebox · 2021-05-17T06:44:28Z

/lgtm

…erstanding

AndreMouche

I think it is not a good idea to import tikv package in the interface kv.Storage since the Storage maybe not tikv. However, the logic looks good to me, and maybe we can do the above thing in another PR.

AndreMouche · 2021-05-19T03:43:43Z

kv/kv.go

 // Storage defines the interface for storage.
 // Isolation should be at least SI(SNAPSHOT ISOLATION)
 type Storage interface {
 	// Begin a global transaction
 	Begin() (Transaction, error)
 	// Begin a transaction with given option
-	BeginWithOption(option TransactionOption) (Transaction, error)
+	BeginWithOption(option tikv.StartTSOption) (Transaction, error)


I think it is not a good idea to import tikv package in the interface Storage, Maybe define the option as an interface{} will be better since the Storage maybe not tikv, and we can convert it to tikv.StartTSOption in tikv driver.

ti-chi-bot · 2021-05-19T03:45:22Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

AndreMouche
xhebox

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

disksing · 2021-05-19T03:47:31Z

/merge

ti-chi-bot · 2021-05-19T03:47:34Z

@disksing: /merge is only allowed for the committers, you can assign this pull request to the committer in list by filling /assign @committer in the comment to help merge this pull request.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

AndreMouche · 2021-05-19T03:51:58Z

/merge

ti-chi-bot · 2021-05-19T03:52:00Z

This pull request has been accepted and is ready to merge.

Commit hash: 1b9ab55

ti-chi-bot · 2021-05-19T06:10:55Z

/run-all-tests

This bot automatically retries jobs that failed on can merge PRs (send feedback to hi-rustin).

Silence the bot with the /merge cancel comment for consistent failures.

ti-chi-bot · 2021-05-19T06:22:08Z

@longfangsong: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-05-19T07:12:53Z

/run-all-tests

This bot automatically retries jobs that failed on can merge PRs (send feedback to hi-rustin).

Silence the bot with the /merge cancel comment for consistent failures.

…ingcap#24052) * *: fix revoke statement for CURRENT_USER() and refine error message planner: support set tidb_allow_mpp to `2` or `ENFORCE` to enforce use mpp mode. (pingcap#24516) store/tikv: remove use of SchemaAmender option in store/tikv (pingcap#24408) *: the value of tikv-client.store-liveness-timeout should not less than 0 (pingcap#24244) store/tikv: remove use of EnableAsyncCommit option in store/tikv (pingcap#24462) txn: Add txn state's view (pingcap#22908) planner: ignore lock for temporary table of PointGet and BatchPointGet (pingcap#24540) store/tikv: remove use of ReplicaRead transaction option in store/tikv (pingcap#24409) store/driver: move error to single package (pingcap#24549) ddl: add check table compatibility for temporary table (pingcap#24501) store/tikv: remove use of IsStatenessReadOnly option in store/tikv (pingcap#24464) store/tikv: change backoff type for missed tiflash peer. (pingcap#24577) store/tikv: remove use of MatchStoreLabels transaction option in store/tikv (pingcap#24465) executor, meta: Allocate auto id for global temporary tables (pingcap#24506) store/tikv: remove use of SampleStep option in store/tikv (pingcap#24461) executor: add partition pruning tests for adding and dropping partition operations (pingcap#24573) ddl: forbid partition on temporary mode before put into queue (pingcap#24565) ddl: speedup test case TestIndexOnMultipleGeneratedColumn (pingcap#24487) execution: Fix issue 24439 Inconsistent error with MySQL for GRANT CREATE USER ON <specific db>.* (pingcap#24485) *: fix errcheck (pingcap#24463) test: make TestExtractStartTs stable (pingcap#24585) ddl: forbid recover/flashback temporary tables (pingcap#24518) executor: fix point_get result on clustered index when new-row-format disabled but new-collation enabled (pingcap#24544) executor: Improve the performance of appending not fixed columns (pingcap#20969) *: typo fix (pingcap#24564) planner/core: refresh stale regions in cache for batch cop response (pingcap#24457) binlog: DML on temporary tables do not write binlog (pingcap#24570) store/tikv: remove use of GuaranteeLinearizability option in store/tikv (pingcap#24605) store/tikv: remove use of CollectRuntimeStats option in store/tikv (pingcap#24604) store/tikv: move Backoffer into a single package (pingcap#24525) variables: init cte max recursive deeps in a new session (pingcap#24609) store/tikv: move transaction options out to /kv (pingcap#24619) store/driver: move backoff driver into single package so we can use i… (pingcap#24624) server: close the temporary session in HTTP API to avoid memory leak (pingcap#24339) store/tikv: use latest PD TS plus one as min commit ts (pingcap#24579) planner: fix incorrect TableDual plan built from nulleq (pingcap#24596) ranger: fix the case which could have duplicate ranges (pingcap#24590) executor, store: Pass the SQL digest down to pessimistic lock request (pingcap#24380) kv: remove UnionStore interface (pingcap#24625) *: enable gosimple linter (pingcap#24617) txn: avoid the gc resolving pessimistic locks of ongoing transactions (pingcap#24601) util: fix wrong enum building for index range (pingcap#24632) sessionctx: change innodb large prefix default (pingcap#24555) store: fix data race about KVStore.tikvClient (pingcap#24655) executor, privileges: Add dynamic privileges to SHOW PRIVILEGES (pingcap#24646) ddl: refactor rule [4/6] (pingcap#24007) cmd: ddl_test modify retryCnt from 5 to 20 (pingcap#24662) executor: add correctness tests about direct reading with ORDER BY and LIMIT (pingcap#24455) store/tikv: remove options from unionstore (pingcap#24629) planner: fix wrongly check for update statement (pingcap#24614) store/tikv: remove CompareTS (pingcap#24657) planner, privilege: Add security enhanced mode part 4 (pingcap#24416) executor: add some test cases about partition table dynamic-mode with split-region (pingcap#24665) planner: fix wrong column offsets when processing dynamic pruning for IndexJoin (pingcap#24659) *: Add security enhanced mode part 3 (pingcap#24412) store/tikv: resolve ReplicaReadType dependencies (pingcap#24653) executor: add test cases about partition table with `expression` (pingcap#24628) tablecodec: fix write wrong prefix index value when collation is ascii_bin/latin1_bin (pingcap#24578) *: compatibility with staleread (pingcap#24285) session: test that temporary tables will also be retried (pingcap#24505) domain, session: Add new sysvarcache to replace global values cache (pingcap#24359) ddl, transaction: DDL on temporary tables won't affect transactions (pingcap#24534) *: implement tidb_bounded_staleness built-in function (pingcap#24328) executor: add correctness tests for partition table with different joins (pingcap#24673) expression: fix the spelling of word arithmetical (pingcap#24713) store/copr: balance region for batch cop task (pingcap#24521) store, metrics: Add metrics for safetTS updating (pingcap#24687) sem: add tidbredact log to restricted variables (pingcap#24701) session: fix dml_batch_size doesn't load the global variable (pingcap#24710) store/tikv: retry TSO RPC (pingcap#24682) expression, planner: push cast down to control function with enum type. (pingcap#24542) executor: add correctness tests about IndexMerge (pingcap#24674) variable: change default for DefDMLBatchSize tidbOptInt64 call (pingcap#24697) planner: add partitioning pruning tests for range partitioning (pingcap#24554) *: add option for enum push down (pingcap#24685) txn: break dependency from store/tikv to tidb/kv cause by TransactionOption (pingcap#24656) executor: enhancement for ListInDisk(support writing after reading) (pingcap#24379) kv: move TxnScope into kv (pingcap#24715) execution: fix the incorrect use of cached plan for point get (pingcap#24749) executor: add correctness tests about direct reading with indexJoin (pingcap#24497) variable: fix sysvar datarace with deep copy (pingcap#24732) *: Implementing RENAME USER (pingcap#24413) infoschema, executor: Add the deadlock table (pingcap#24524) docs: Some proposal for renaming and configurations for Lock View (pingcap#24718) planner: add range partition boundaries tests with BETWEEN expression (pingcap#24598) oracle: simplify timestamp utilities (pingcap#24688) executor: fix wrong enum key in point get (pingcap#24618) ranger: fix incorrect enum range for xxx_ci collation (pingcap#24661) executor: add some test cases about dynamic-mode and apply operator (pingcap#24683) store/tikv: remove Variables.Hook (pingcap#24758) ddl: speed up the execution time of `TestBackwardCompatibility`. (pingcap#24704) *: prepare errors for CTE (pingcap#24763) expression: support cast real/int as real (pingcap#24670) executor: add table name in log (pingcap#24666) expression: add builtin function ``json_pretty`` (pingcap#24675) ddl: make `TestDropLastVisibleColumns` stable (pingcap#24790) * ddl: make `TestDropLastVisibleColumns` stable *: support AS OF TIMESTAMP read-only begin statement (pingcap#24740) executor: Fix unstable TestTiDBLastTxnInfoCommitMode (pingcap#24779) planner: add tests for partition range boundaries for LT/GT (pingcap#24574) test: record random seed in TestIssue20658 (pingcap#24782) store/tikv/retry: define Config instead of BackoffType (pingcap#24692) config: ignore tiflash when show config (pingcap#24770) privileges: improve dynamic privs registration and tests (pingcap#24773) README: remove the link of TiDB Monthly Update (pingcap#24791) region_cache: filter peers on tombstone or dropped stores (pingcap#24726) util/stmtsummary: remove import package tikv (pingcap#24776) ddl: grammar check for create unsupported temporary table (pingcap#24723) *: update go.etcd.io/bbolt (pingcap#24799) ddl: speed up the execution time of `ddl test` and `Test Chunk pingcap#7 ddl-other` (pingcap#24780) executor: remove the unnecessary use of fmt.Sprintf (pingcap#24815) executor: fix index join panic on prefix index on some cases (pingcap#24568)

break dependency

7a2b3c4

ti-chi-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 14, 2021

Move unit tests

d5f3e08

longfangsong force-pushed the break-dependency branch from a23fc0f to d5f3e08 Compare May 14, 2021 04:23

github-actions bot added the sig/sql-infra SIG: SQL Infra label May 14, 2021

ti-chi-bot requested review from disksing and tisonkun May 14, 2021 04:34

tisonkun approved these changes May 14, 2021

View reviewed changes

disksing reviewed May 14, 2021

View reviewed changes

store/tikv/kv.go Show resolved Hide resolved

store/tikv/kv.go Outdated Show resolved Hide resolved

store/tikv/txn.go Outdated Show resolved Hide resolved

longfangsong added 2 commits May 14, 2021 18:54

Address some comments

72cb181

Update comments

ee60d0e

longfangsong requested a review from disksing May 14, 2021 10:57

Merge branch 'master' into break-dependency

8955b4b

disksing approved these changes May 15, 2021

View reviewed changes

Merge branch 'master' into break-dependency

abe633a

longfangsong requested a review from tisonkun May 17, 2021 02:35

tisonkun approved these changes May 17, 2021

View reviewed changes

Merge branch 'master' into break-dependency

f92910f

ti-chi-bot requested a review from crazycs520 May 17, 2021 06:24

ti-chi-bot requested review from morgo, xhebox and XuHuaiyu May 17, 2021 06:24

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 17, 2021

Merge remote-tracking branch 'upstream/master' into break-dependency

4e96013

longfangsong force-pushed the break-dependency branch from 0eea2cd to 4e96013 Compare May 19, 2021 02:34

Rename TransactionOption to StartTSOption, which might be less misund…

a52ba04

…erstanding

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 19, 2021

AndreMouche approved these changes May 19, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 19, 2021

AndreMouche added the sig/transaction SIG:Transaction label May 19, 2021

Merge branch 'master' into break-dependency

1b9ab55

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 19, 2021

Merge branch 'master' into break-dependency

5d627bb

ti-chi-bot merged commit 49d1eaa into pingcap:master May 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

txn: break dependency from store/tikv to tidb/kv cause by TransactionOption #24656

txn: break dependency from store/tikv to tidb/kv cause by TransactionOption #24656

longfangsong commented May 14, 2021 •

edited

Loading

longfangsong commented May 14, 2021

longfangsong commented May 14, 2021

tisonkun left a comment

ti-chi-bot commented May 14, 2021

tisonkun commented May 14, 2021

longfangsong commented May 14, 2021

tisonkun commented May 14, 2021

disksing commented May 14, 2021

ti-chi-bot commented May 15, 2021

tisonkun commented May 15, 2021

tisonkun left a comment

ti-chi-bot commented May 17, 2021

tisonkun commented May 17, 2021

xhebox commented May 17, 2021

AndreMouche left a comment

AndreMouche May 19, 2021

ti-chi-bot commented May 19, 2021

disksing commented May 19, 2021

ti-chi-bot commented May 19, 2021

AndreMouche commented May 19, 2021

ti-chi-bot commented May 19, 2021

ti-chi-bot commented May 19, 2021

ti-chi-bot commented May 19, 2021

ti-chi-bot commented May 19, 2021

txn: break dependency from store/tikv to tidb/kv cause by TransactionOption #24656

txn: break dependency from store/tikv to tidb/kv cause by TransactionOption #24656

Conversation

longfangsong commented May 14, 2021 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

longfangsong commented May 14, 2021

longfangsong commented May 14, 2021

tisonkun left a comment

Choose a reason for hiding this comment

ti-chi-bot commented May 14, 2021

tisonkun commented May 14, 2021

longfangsong commented May 14, 2021

tisonkun commented May 14, 2021

disksing commented May 14, 2021

ti-chi-bot commented May 15, 2021

tisonkun commented May 15, 2021

tisonkun left a comment

Choose a reason for hiding this comment

ti-chi-bot commented May 17, 2021

tisonkun commented May 17, 2021

xhebox commented May 17, 2021

AndreMouche left a comment

Choose a reason for hiding this comment

AndreMouche May 19, 2021

Choose a reason for hiding this comment

ti-chi-bot commented May 19, 2021

disksing commented May 19, 2021

ti-chi-bot commented May 19, 2021

AndreMouche commented May 19, 2021

ti-chi-bot commented May 19, 2021

ti-chi-bot commented May 19, 2021

ti-chi-bot commented May 19, 2021

ti-chi-bot commented May 19, 2021

longfangsong commented May 14, 2021 •

edited

Loading