Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

topsql: distinguish the row and index operation type #29044

Merged
merged 97 commits into from
Nov 24, 2021
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
1a8573d
init
crazycs520 Aug 12, 2021
f59d00b
init
crazycs520 Aug 13, 2021
8c049fe
Merge branch 'master' into topsql-scan-row
mornyx Sep 1, 2021
e832bfd
Merge branch 'master' into topsql-scan-row
mornyx Sep 2, 2021
5478ab9
Merge branch 'master' into topsql-scan-row
mornyx Sep 27, 2021
3cee76c
Merge branch 'master' into resource-metering
mornyx Oct 13, 2021
7ae5bbd
Extend label of ResourceGroupTag
mornyx Oct 24, 2021
d4b5fe6
Merge branch 'master' into resource-metering
mornyx Oct 24, 2021
e8dd9d3
Fix unit tests
mornyx Oct 24, 2021
b6866b7
Merge branch 'master' into resource-metering
mornyx Oct 27, 2021
9d750ad
Remove mornyx/tipb
mornyx Oct 28, 2021
26a03b1
Merge branch 'master' into resource-metering
mornyx Oct 28, 2021
e2b0099
Run go mod tidy
mornyx Oct 28, 2021
dccdef5
Merge branch 'master' into resource-metering
mornyx Oct 28, 2021
1e66caa
Merge branch 'master' into resource-metering
mornyx Oct 28, 2021
9267abb
Add test cases for resource metering tag label
mornyx Oct 28, 2021
99380ba
Merge branch 'master' into resource-metering
mornyx Oct 28, 2021
35942bc
Fix tests
mornyx Oct 28, 2021
1b04965
Fix tests
mornyx Oct 28, 2021
af63673
Fix tests
mornyx Oct 28, 2021
e67f408
Distinguish row/index based on first key
mornyx Nov 3, 2021
941a53b
Merge branch 'master' into resource-metering
mornyx Nov 3, 2021
1777d41
Repair circular dependencies; Add unit-tests
mornyx Nov 3, 2021
1259c8b
Upgrade client-go
mornyx Nov 3, 2021
d90985d
tidy
mornyx Nov 3, 2021
2dd79e8
Merge branch 'master' into resource-metering
mornyx Nov 3, 2021
a6caf5b
Complete executor_test
mornyx Nov 3, 2021
7537bca
Use ResourceGroupTagFactory
mornyx Nov 3, 2021
40a09dc
Merge branch 'master' into resource-metering
mornyx Nov 3, 2021
ad1d725
Fix type conversion
mornyx Nov 3, 2021
a703a8a
Merge branch 'master' into resource-metering
mornyx Nov 4, 2021
4781623
Upgrade to ResourceGroupTagger
mornyx Nov 9, 2021
02f535e
Merge remote-tracking branch 'mornyx/resource-metering' into resource…
mornyx Nov 9, 2021
9ce5bca
Merge branch 'master' into resource-metering
mornyx Nov 9, 2021
0b7be5d
Update kv/option.go
mornyx Nov 9, 2021
e747587
Fix go.sum
mornyx Nov 10, 2021
eea3cda
Merge branch 'master' into resource-metering
mornyx Nov 10, 2021
5c960c3
Merge branch 'master' into resource-metering
mornyx Nov 10, 2021
eeaeb16
Merge branch 'master' into resource-metering
mornyx Nov 11, 2021
155348e
Merge branch 'master' into resource-metering
mornyx Nov 15, 2021
0fcd445
Upgrade client-go
mornyx Nov 15, 2021
e37e8a2
Fix resource group tag test assertion
mornyx Nov 15, 2021
70634be
Merge branch 'master' into resource-metering
mornyx Nov 15, 2021
f5d269e
Merge branch 'master' into resource-metering
mornyx Nov 15, 2021
ac6afb5
Merge branch 'master' into resource-metering
mornyx Nov 15, 2021
41f978a
Optimize resource group tagger for DAG request
mornyx Nov 15, 2021
46c6626
Remove print in test
mornyx Nov 15, 2021
50e829c
Merge branch 'master' into resource-metering
mornyx Nov 15, 2021
74c2b82
Merge branch 'master' into resource-metering
mornyx Nov 16, 2021
5489fa2
Update comment for SetFromSessionVars
mornyx Nov 16, 2021
06395ce
optimize
mornyx Nov 16, 2021
3251c73
Merge branch 'master' into resource-metering
mornyx Nov 16, 2021
2c41863
Add rowindexcodec
mornyx Nov 17, 2021
a2d214b
Add comments
mornyx Nov 17, 2021
7b70cab
Merge branch 'master' into resource-metering
mornyx Nov 17, 2021
cc352e1
Add license comment; Add main_test
mornyx Nov 17, 2021
3b6703e
Merge branch 'master' into resource-metering
mornyx Nov 17, 2021
7b026b2
Update executor/executor.go
mornyx Nov 18, 2021
d59055e
Rename setResourceGroupTagForTxn
mornyx Nov 18, 2021
23d5eaf
Merge branch 'master' into resource-metering
crazycs520 Nov 18, 2021
fd2ba1f
Set ResourceGroupTagger for LockCtx
mornyx Nov 18, 2021
cc25883
Fix executor test
mornyx Nov 19, 2021
df05935
Merge branch 'master' into resource-metering
mornyx Nov 19, 2021
c118a2e
Merge remote-tracking branch 'mornyx/resource-metering' into resource…
mornyx Nov 19, 2021
f341a3e
Fix executor test
mornyx Nov 19, 2021
74d4da8
Merge branch 'master' into resource-metering
mornyx Nov 19, 2021
3370349
Fix tests
mornyx Nov 19, 2021
e757948
Merge branch 'master' into resource-metering
mornyx Nov 19, 2021
e0e2d86
Merge branch 'master' into resource-metering
mornyx Nov 19, 2021
23f6e50
Merge branch 'master' into resource-metering
mornyx Nov 22, 2021
88e4bfe
Merge branch 'master' into resource-metering
breezewish Nov 22, 2021
d689ee1
Generate digest explicitly
mornyx Nov 22, 2021
6132d4b
Merge branch 'master' into resource-metering
mornyx Nov 22, 2021
d90c86f
Remove unnecessary judgment whether TopSQL is enabled
mornyx Nov 22, 2021
ae33f1a
Merge branch 'master' into resource-metering
mornyx Nov 22, 2021
09d5b98
Merge branch 'master' into resource-metering
mornyx Nov 22, 2021
77c6ced
Merge branch 'master' into resource-metering
mornyx Nov 22, 2021
9c32941
Merge branch 'master' into resource-metering
mornyx Nov 22, 2021
804ab5e
Merge branch 'master' into resource-metering
mornyx Nov 22, 2021
ff2fbd8
Merge branch 'master' into resource-metering
mornyx Nov 22, 2021
f63e2d9
Merge branch 'master' into resource-metering
mornyx Nov 22, 2021
9bea4a7
Fix plan digest
mornyx Nov 23, 2021
0a9516f
Merge branch 'master' into resource-metering
mornyx Nov 23, 2021
1c0bcc6
Fix
mornyx Nov 23, 2021
8443600
Merge branch 'master' into resource-metering
mornyx Nov 23, 2021
33780a2
Merge branch 'master' into resource-metering
mornyx Nov 23, 2021
abc4d09
Merge branch 'master' into resource-metering
mornyx Nov 23, 2021
754380d
Merge branch 'master' into resource-metering
mornyx Nov 23, 2021
82b0156
Merge branch 'master' into resource-metering
mornyx Nov 23, 2021
2113719
Merge branch 'master' into resource-metering
mornyx Nov 23, 2021
ce014d1
Merge branch 'master' into resource-metering
mornyx Nov 23, 2021
79cee96
Merge branch 'master' into resource-metering
mornyx Nov 23, 2021
e0668d8
Merge branch 'master' into resource-metering
mornyx Nov 23, 2021
110a222
Merge branch 'master' into resource-metering
mornyx Nov 23, 2021
37a684d
Merge branch 'master' into resource-metering
mornyx Nov 23, 2021
a0be0df
Merge branch 'master' into resource-metering
ti-chi-bot Nov 23, 2021
2a74c95
Merge branch 'master' into resource-metering
ti-chi-bot Nov 24, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion distsql/request_builder.go
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,16 @@ func (builder *RequestBuilder) SetDAGRequest(dag *tipb.DAGRequest) *RequestBuild
builder.Request.Concurrency = 1
}
}
if variable.TopSQLEnabled() && len(dag.Executors) > 0 {
breezewish marked this conversation as resolved.
Show resolved Hide resolved
switch dag.Executors[0].Tp {
case tipb.ExecType_TypeIndexScan:
builder.SetResourceGroupTagLabel(tipb.ResourceGroupTagLabel_ResourceGroupTagLabelIndex)
breezewish marked this conversation as resolved.
Show resolved Hide resolved
case tipb.ExecType_TypeTableScan:
builder.SetResourceGroupTagLabel(tipb.ResourceGroupTagLabel_ResourceGroupTagLabelRow)
default:
builder.SetResourceGroupTagLabel(tipb.ResourceGroupTagLabel_ResourceGroupTagLabelUnknown)
}
}
return builder
}

Expand Down Expand Up @@ -214,6 +224,12 @@ func (builder *RequestBuilder) SetAllowBatchCop(batchCop bool) *RequestBuilder {
return builder
}

// SetResourceGroupTagLabel sets `ResourceGroupTagLabel` property.
func (builder *RequestBuilder) SetResourceGroupTagLabel(label tipb.ResourceGroupTagLabel) *RequestBuilder {
builder.Request.ResourceGroupTagLabel = label
return builder
}

func (builder *RequestBuilder) getIsolationLevel() kv.IsoLevel {
switch builder.Tp {
case kv.ReqTypeAnalyze:
Expand Down Expand Up @@ -285,7 +301,7 @@ func (builder *RequestBuilder) SetFromInfoSchema(pis interface{}) *RequestBuilde
// SetResourceGroupTag sets the request resource group tag.
func (builder *RequestBuilder) SetResourceGroupTag(sc *stmtctx.StatementContext) *RequestBuilder {
if variable.TopSQLEnabled() {
builder.Request.ResourceGroupTag = sc.GetResourceGroupTag()
builder.Request.ResourceGroupTag = sc.GetResourceGroupTag(builder.Request.ResourceGroupTagLabel)
}
return builder
}
Expand Down
4 changes: 4 additions & 0 deletions executor/batch_point_get.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ import (
"sort"
"sync/atomic"

"github.com/pingcap/tipb/go-tipb"

mornyx marked this conversation as resolved.
Show resolved Hide resolved
"github.com/pingcap/errors"
"github.com/pingcap/failpoint"
"github.com/pingcap/kvproto/pkg/metapb"
Expand Down Expand Up @@ -275,6 +277,7 @@ func (e *BatchPointGetExec) initialize(ctx context.Context) error {
}

// Fetch all handles.
setResourceGroupTagForSnapshot(e.ctx.GetSessionVars().StmtCtx, e.snapshot, tipb.ResourceGroupTagLabel_ResourceGroupTagLabelIndex)
handleVals, err = batchGetter.BatchGet(ctx, toFetchIndexKeys)
if err != nil {
return err
Expand Down Expand Up @@ -414,6 +417,7 @@ func (e *BatchPointGetExec) initialize(ctx context.Context) error {
}
}
// Fetch all values.
setResourceGroupTagForSnapshot(e.ctx.GetSessionVars().StmtCtx, e.snapshot, tipb.ResourceGroupTagLabel_ResourceGroupTagLabelRow)
values, err = batchGetter.BatchGet(ctx, keys)
if err != nil {
return err
Expand Down
11 changes: 9 additions & 2 deletions executor/executor.go
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ import (
"github.com/pingcap/tidb/util/memory"
"github.com/pingcap/tidb/util/resourcegrouptag"
"github.com/pingcap/tidb/util/topsql"
"github.com/pingcap/tipb/go-tipb"
tikverr "github.com/tikv/client-go/v2/error"
tikvstore "github.com/tikv/client-go/v2/kv"
tikvutil "github.com/tikv/client-go/v2/util"
Expand Down Expand Up @@ -979,7 +980,7 @@ func newLockCtx(seVars *variable.SessionVars, lockWaitTime int64) *tikvstore.Loc
lockCtx.LockKeysDuration = &seVars.StmtCtx.LockKeysDuration
lockCtx.LockKeysCount = &seVars.StmtCtx.LockKeysCount
lockCtx.LockExpired = &seVars.TxnCtx.LockExpire
lockCtx.ResourceGroupTag = resourcegrouptag.EncodeResourceGroupTag(sqlDigest, planDigest)
lockCtx.ResourceGroupTag = resourcegrouptag.EncodeResourceGroupTag(sqlDigest, planDigest, tipb.ResourceGroupTagLabel_ResourceGroupTagLabelUnknown)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not also use SetResourceGroupTagger too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to be missed here, but does it make sense to distinguish row/index on PessimisticLockRequest? If so, I will push an update.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it makes sense.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tikv/client-go#381
PTAL, thanks. I will update this PR after the PR of client-go is merged.
cc @zhongzc @crazycs520

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lockCtx.OnDeadlock = func(deadlock *tikverr.ErrDeadlock) {
cfg := config.GetGlobalConfig()
if deadlock.IsRetryable && !cfg.PessimisticTxn.DeadlockHistoryCollectRetryable {
Expand Down Expand Up @@ -1896,6 +1897,12 @@ func FillVirtualColumnValue(virtualRetTypes []*types.FieldType, virtualColumnInd

func setResourceGroupTagForTxn(sc *stmtctx.StatementContext, snapshot kv.Snapshot) {
mornyx marked this conversation as resolved.
Show resolved Hide resolved
if snapshot != nil && variable.TopSQLEnabled() {
snapshot.SetOption(kv.ResourceGroupTag, sc.GetResourceGroupTag())
snapshot.SetOption(kv.ResourceGroupTag, sc.GetResourceGroupTag(tipb.ResourceGroupTagLabel_ResourceGroupTagLabelUnknown))
}
}

func setResourceGroupTagForSnapshot(sc *stmtctx.StatementContext, snapshot kv.Snapshot, label tipb.ResourceGroupTagLabel) {
if snapshot != nil && variable.TopSQLEnabled() {
snapshot.SetOption(kv.ResourceGroupTag, sc.GetResourceGroupTag(label))
}
}
4 changes: 4 additions & 0 deletions executor/point_get.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ import (
"context"
"fmt"

"github.com/pingcap/tipb/go-tipb"

"github.com/pingcap/errors"
"github.com/pingcap/failpoint"
"github.com/pingcap/kvproto/pkg/metapb"
Expand Down Expand Up @@ -239,6 +241,7 @@ func (e *PointGetExecutor) Next(ctx context.Context, req *chunk.Chunk) error {
return err
}

setResourceGroupTagForSnapshot(e.ctx.GetSessionVars().StmtCtx, e.snapshot, tipb.ResourceGroupTagLabel_ResourceGroupTagLabelIndex)
e.handleVal, err = e.get(ctx, e.idxKey)
if err != nil {
if !kv.ErrNotExist.Equal(err) {
Expand Down Expand Up @@ -292,6 +295,7 @@ func (e *PointGetExecutor) Next(ctx context.Context, req *chunk.Chunk) error {
}
}

setResourceGroupTagForSnapshot(e.ctx.GetSessionVars().StmtCtx, e.snapshot, tipb.ResourceGroupTagLabel_ResourceGroupTagLabelRow)
key := tablecodec.EncodeRowKeyWithHandle(tblID, e.handle)
val, err := e.getAndLock(ctx, key)
if err != nil {
Expand Down
3 changes: 2 additions & 1 deletion executor/update.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ import (
"github.com/pingcap/tidb/util/chunk"
"github.com/pingcap/tidb/util/execdetails"
"github.com/pingcap/tidb/util/memory"
"github.com/pingcap/tipb/go-tipb"
"github.com/tikv/client-go/v2/txnkv/txnsnapshot"
)

Expand Down Expand Up @@ -274,7 +275,7 @@ func (e *UpdateExec) updateRows(ctx context.Context) (int, error) {
if variable.TopSQLEnabled() {
txn, err := e.ctx.Txn(true)
if err == nil {
txn.SetOption(kv.ResourceGroupTag, e.ctx.GetSessionVars().StmtCtx.GetResourceGroupTag())
txn.SetOption(kv.ResourceGroupTag, e.ctx.GetSessionVars().StmtCtx.GetResourceGroupTag(tipb.ResourceGroupTagLabel_ResourceGroupTagLabelUnknown))
}
}
for rowIdx := 0; rowIdx < chk.NumRows(); rowIdx++ {
Expand Down
8 changes: 4 additions & 4 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ require (
github.com/pingcap/sysutil v0.0.0-20210730114356-fcd8a63f68c5
github.com/pingcap/tidb-tools v5.2.2-0.20211019062242-37a8bef2fa17+incompatible
github.com/pingcap/tidb/parser v0.0.0-20211011031125-9b13dc409c5e
github.com/pingcap/tipb v0.0.0-20211026080602-ec68283c1735
github.com/pingcap/tipb v0.0.0-20211028032111-4de152204fb8
github.com/prometheus/client_golang v1.5.1
github.com/prometheus/client_model v0.2.0
github.com/prometheus/common v0.9.1
Expand All @@ -79,12 +79,12 @@ require (
go.uber.org/goleak v1.1.11-0.20210813005559-691160354723
go.uber.org/multierr v1.7.0
go.uber.org/zap v1.19.1
golang.org/x/net v0.0.0-20210503060351-7fd8e65b6420
golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d
golang.org/x/oauth2 v0.0.0-20210805134026-6f1e6394065a
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
golang.org/x/sys v0.0.0-20210806184541-e5e7981a1069
golang.org/x/sys v0.0.0-20211025201205-69cdffdb9359
golang.org/x/text v0.3.7
golang.org/x/tools v0.1.5
golang.org/x/tools v0.1.7
google.golang.org/api v0.54.0
google.golang.org/grpc v1.40.0
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
Expand Down
15 changes: 10 additions & 5 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -614,8 +614,8 @@ github.com/pingcap/tidb-dashboard v0.0.0-20210312062513-eef5d6404638/go.mod h1:O
github.com/pingcap/tidb-dashboard v0.0.0-20210716172320-2226872e3296/go.mod h1:OCXbZTBTIMRcIt0jFsuCakZP+goYRv6IjawKbwLS2TQ=
github.com/pingcap/tidb-tools v5.2.2-0.20211019062242-37a8bef2fa17+incompatible h1:c7+izmker91NkjkZ6FgTlmD4k1A5FLOAq+li6Ki2/GY=
github.com/pingcap/tidb-tools v5.2.2-0.20211019062242-37a8bef2fa17+incompatible/go.mod h1:XGdcy9+yqlDSEMTpOXnwf3hiTeqrV6MN/u1se9N8yIM=
github.com/pingcap/tipb v0.0.0-20211026080602-ec68283c1735 h1:kS8pJNUnF3ENkjtBcJeMe/W8+9RtrChcortoyljCwwc=
github.com/pingcap/tipb v0.0.0-20211026080602-ec68283c1735/go.mod h1:A7mrd7WHBl1o63LE2bIBGEJMTNWXqhgmYiOvMLxozfs=
github.com/pingcap/tipb v0.0.0-20211028032111-4de152204fb8 h1:aVxoDUuWW4PmsCWHL+KAkyPU280+HDsTm+0p4/KPwZM=
github.com/pingcap/tipb v0.0.0-20211028032111-4de152204fb8/go.mod h1:A7mrd7WHBl1o63LE2bIBGEJMTNWXqhgmYiOvMLxozfs=
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
Expand Down Expand Up @@ -796,6 +796,7 @@ github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9de
github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
github.com/yuin/goldmark v1.4.0/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.etcd.io/bbolt v1.3.3/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.etcd.io/bbolt v1.3.5 h1:XAzx9gjCb0Rxj7EoqcClPD1d5ZBxZJk0jbuoPHenBt0=
Expand Down Expand Up @@ -949,8 +950,9 @@ golang.org/x/net v0.0.0-20210119194325-5f4716e94777/go.mod h1:m0MpNAwzfU5UDzcl9v
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210316092652-d523dce5a7f4/go.mod h1:RBQZq4jEuRlivfhVLdyRGr576XBO4/greRjx4P4O3yc=
golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
golang.org/x/net v0.0.0-20210503060351-7fd8e65b6420 h1:a8jGStKg0XqKDlKqjLrXn0ioF5MH36pT7Z0BRTqLhbk=
golang.org/x/net v0.0.0-20210503060351-7fd8e65b6420/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d h1:20cMwl2fHAzkJMEA+8J4JgqBQcQGzbisXo31MIeenXI=
golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
Expand Down Expand Up @@ -1043,8 +1045,10 @@ golang.org/x/sys v0.0.0-20210514084401-e8d321eab015/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20210603125802-9665404d3644/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210806184541-e5e7981a1069 h1:siQdpVirKtzPhKl3lZWozZraCFObP8S1v6PRp0bLrtU=
golang.org/x/sys v0.0.0-20210806184541-e5e7981a1069/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211025201205-69cdffdb9359 h1:2B5p2L5IfGiD7+b9BOoRMC6DgObAVZV+Fsp050NqXik=
golang.org/x/sys v0.0.0-20211025201205-69cdffdb9359/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
Expand Down Expand Up @@ -1133,8 +1137,9 @@ golang.org/x/tools v0.1.1/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/tools v0.1.2/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/tools v0.1.3/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/tools v0.1.4/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/tools v0.1.5 h1:ouewzE6p+/VEB31YYnTbEJdi8pFqKp4P4n85vwo3DHA=
golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/tools v0.1.7 h1:6j8CgantCy3yc8JGBqkDLMKWqZ0RDU2g1HVgacojGWQ=
golang.org/x/tools v0.1.7/go.mod h1:LGqMHiF4EqQNHR1JncWGqT5BVaXmza+X+BDGol+dOxo=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
Expand Down
7 changes: 4 additions & 3 deletions infoschema/cluster_tables_serial_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ import (
"github.com/pingcap/tidb/util/resourcegrouptag"
"github.com/pingcap/tidb/util/set"
"github.com/pingcap/tidb/util/testutil"
"github.com/pingcap/tipb/go-tipb"
"github.com/stretchr/testify/require"
"google.golang.org/grpc"
)
Expand Down Expand Up @@ -160,10 +161,10 @@ func SubTestTestDataLockWaits(s *clusterTablesSuite) func(*testing.T) {
_, digest1 := parser.NormalizeDigest("select * from test_data_lock_waits for update")
_, digest2 := parser.NormalizeDigest("update test_data_lock_waits set f1=1 where id=2")
s.store.(mockstorage.MockLockWaitSetter).SetMockLockWaits([]*deadlock.WaitForEntry{
{Txn: 1, WaitForTxn: 2, Key: []byte("key1"), ResourceGroupTag: resourcegrouptag.EncodeResourceGroupTag(digest1, nil)},
{Txn: 3, WaitForTxn: 4, Key: []byte("key2"), ResourceGroupTag: resourcegrouptag.EncodeResourceGroupTag(digest2, nil)},
{Txn: 1, WaitForTxn: 2, Key: []byte("key1"), ResourceGroupTag: resourcegrouptag.EncodeResourceGroupTag(digest1, nil, tipb.ResourceGroupTagLabel_ResourceGroupTagLabelUnknown)},
{Txn: 3, WaitForTxn: 4, Key: []byte("key2"), ResourceGroupTag: resourcegrouptag.EncodeResourceGroupTag(digest2, nil, tipb.ResourceGroupTagLabel_ResourceGroupTagLabelUnknown)},
// Invalid digests
{Txn: 5, WaitForTxn: 6, Key: []byte("key3"), ResourceGroupTag: resourcegrouptag.EncodeResourceGroupTag(nil, nil)},
{Txn: 5, WaitForTxn: 6, Key: []byte("key3"), ResourceGroupTag: resourcegrouptag.EncodeResourceGroupTag(nil, nil, tipb.ResourceGroupTagLabel_ResourceGroupTagLabelUnknown)},
{Txn: 7, WaitForTxn: 8, Key: []byte("key4"), ResourceGroupTag: []byte("asdfghjkl")},
})

Expand Down
3 changes: 3 additions & 0 deletions kv/kv.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import (
"github.com/pingcap/tidb/parser/model"
"github.com/pingcap/tidb/util/memory"
"github.com/pingcap/tidb/util/trxevents"
"github.com/pingcap/tipb/go-tipb"
tikvstore "github.com/tikv/client-go/v2/kv"
"github.com/tikv/client-go/v2/oracle"
"github.com/tikv/client-go/v2/tikv"
Expand Down Expand Up @@ -337,6 +338,8 @@ type Request struct {
MatchStoreLabels []*metapb.StoreLabel
// ResourceGroupTag indicates the kv request task group.
ResourceGroupTag []byte
// ResourceGroupTagLabel indicates the key label of the kv request.
ResourceGroupTagLabel tipb.ResourceGroupTagLabel
}

const (
Expand Down
1 change: 1 addition & 0 deletions kv/option.go
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ const (
ResourceGroupTag
// KVFilter indicates the filter to ignore key-values in the transaction's memory buffer.
KVFilter

// SnapInterceptor is used for setting the interceptor for snapshot
SnapInterceptor
)
Expand Down
3 changes: 2 additions & 1 deletion session/session.go
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ import (
"github.com/pingcap/tidb/table/temptable"
"github.com/pingcap/tidb/util/topsql"
"github.com/pingcap/tipb/go-binlog"
"github.com/pingcap/tipb/go-tipb"
"go.uber.org/zap"

"github.com/pingcap/tidb/bindinfo"
Expand Down Expand Up @@ -546,7 +547,7 @@ func (s *session) doCommit(ctx context.Context) error {
}
s.txn.SetOption(kv.EnableAsyncCommit, sessVars.EnableAsyncCommit)
s.txn.SetOption(kv.Enable1PC, sessVars.Enable1PC)
s.txn.SetOption(kv.ResourceGroupTag, sessVars.StmtCtx.GetResourceGroupTag())
s.txn.SetOption(kv.ResourceGroupTag, sessVars.StmtCtx.GetResourceGroupTag(tipb.ResourceGroupTagLabel_ResourceGroupTagLabelUnknown))
// priority of the sysvar is lower than `start transaction with causal consistency only`
if val := s.txn.GetOption(kv.GuaranteeLinearizability); val == nil || val.(bool) {
// We needn't ask the TiKV client to guarantee linearizability for auto-commit transactions
Expand Down
19 changes: 10 additions & 9 deletions sessionctx/stmtctx/stmtctx.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ import (
"github.com/pingcap/tidb/util/execdetails"
"github.com/pingcap/tidb/util/memory"
"github.com/pingcap/tidb/util/resourcegrouptag"
"github.com/pingcap/tipb/go-tipb"
"github.com/tikv/client-go/v2/util"
atomic2 "go.uber.org/atomic"
"go.uber.org/zap"
Expand Down Expand Up @@ -168,8 +169,8 @@ type StatementContext struct {

// stmtCache is used to store some statement-related values.
stmtCache map[StmtCacheKey]interface{}
// resourceGroupTag cache for the current statement resource group tag.
resourceGroupTag atomic.Value
// resourceGroupTags cache for the current statement resource group tags.
resourceGroupTags sync.Map
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

atomic.Value is lock free but sync.Map is not. This may introduce high contention.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since label is enumerable, I used independent atomic.Value to avoid lock contention. code here

// Map to store all CTE storages of current SQL.
// Will clean up at the end of the execution.
CTEStorageMap interface{}
Expand Down Expand Up @@ -270,18 +271,18 @@ func (sc *StatementContext) GetPlanDigest() (normalized string, planDigest *pars
}

// GetResourceGroupTag gets the resource group of the statement.
func (sc *StatementContext) GetResourceGroupTag() []byte {
tag, _ := sc.resourceGroupTag.Load().([]byte)
if len(tag) > 0 {
return tag
func (sc *StatementContext) GetResourceGroupTag(label tipb.ResourceGroupTagLabel) []byte {
tag, ok := sc.resourceGroupTags.Load(label)
mornyx marked this conversation as resolved.
Show resolved Hide resolved
if ok {
return tag.([]byte)
}
normalized, sqlDigest := sc.SQLDigest()
if len(normalized) == 0 {
return nil
}
tag = resourcegrouptag.EncodeResourceGroupTag(sqlDigest, sc.planDigest)
sc.resourceGroupTag.Store(tag)
return tag
newTag := resourcegrouptag.EncodeResourceGroupTag(sqlDigest, sc.planDigest, label)
sc.resourceGroupTags.Store(label, newTag)
return newTag
}

// SetPlanDigest sets the normalized plan and plan digest.
Expand Down
23 changes: 23 additions & 0 deletions sessionctx/stmtctx/stmtctx_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import (

"github.com/pingcap/tidb/sessionctx/stmtctx"
"github.com/pingcap/tidb/util/execdetails"
"github.com/pingcap/tipb/go-tipb"
"github.com/stretchr/testify/require"
"github.com/tikv/client-go/v2/util"
)
Expand Down Expand Up @@ -90,3 +91,25 @@ func TestStatementContextPushDownFLags(t *testing.T) {
require.Equal(t, tt.out, got)
}
}

func TestGetResourceGroupTag(t *testing.T) {
ctx := stmtctx.StatementContext{OriginalSQL: "SELECT * FROM t"}
tagRow := ctx.GetResourceGroupTag(tipb.ResourceGroupTagLabel_ResourceGroupTagLabelRow)
tagIndex := ctx.GetResourceGroupTag(tipb.ResourceGroupTagLabel_ResourceGroupTagLabelIndex)
tagUnknown := ctx.GetResourceGroupTag(tipb.ResourceGroupTagLabel_ResourceGroupTagLabelUnknown)
tagRow2 := ctx.GetResourceGroupTag(tipb.ResourceGroupTagLabel_ResourceGroupTagLabelRow)
tagIndex2 := ctx.GetResourceGroupTag(tipb.ResourceGroupTagLabel_ResourceGroupTagLabelIndex)
tagUnknown2 := ctx.GetResourceGroupTag(tipb.ResourceGroupTagLabel_ResourceGroupTagLabelUnknown)
require.NotEmpty(t, tagRow)
require.NotEmpty(t, tagIndex)
require.NotEmpty(t, tagUnknown)
require.NotEmpty(t, tagRow2)
require.NotEmpty(t, tagIndex2)
require.NotEmpty(t, tagUnknown2)
require.Equal(t, &tagRow, &tagRow2) // mem addr
require.Equal(t, &tagIndex, &tagIndex2)
require.Equal(t, &tagUnknown, &tagUnknown2)
require.NotEqual(t, &tagRow, &tagIndex)
require.NotEqual(t, &tagRow, &tagUnknown)
require.NotEqual(t, &tagIndex, &tagUnknown)
}
4 changes: 2 additions & 2 deletions util/resourcegrouptag/resource_group_tag.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@ import (
)

// EncodeResourceGroupTag encodes sql digest and plan digest into resource group tag.
func EncodeResourceGroupTag(sqlDigest, planDigest *parser.Digest) []byte {
func EncodeResourceGroupTag(sqlDigest, planDigest *parser.Digest, label tipb.ResourceGroupTagLabel) []byte {
if sqlDigest == nil && planDigest == nil {
return nil
}

tag := &tipb.ResourceGroupTag{}
tag := &tipb.ResourceGroupTag{Label: &label}
if sqlDigest != nil {
tag.SqlDigest = sqlDigest.Bytes()
}
Expand Down
Loading