Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lightning: retry when write tikv is slow and rate limiter context timeout #51383

Closed
D3Hunter opened this issue Feb 28, 2024 · 1 comment · Fixed by #51425
Closed

lightning: retry when write tikv is slow and rate limiter context timeout #51383

D3Hunter opened this issue Feb 28, 2024 · 1 comment · Fixed by #51425
Assignees
@lance6716
Labels
affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. component/ddl This issue is related to DDL of TiDB. component/lightning This issue is related to Lightning of TiDB. severity/moderate type/bug The issue is confirmed as a bug.

Comments

@D3Hunter
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

with store-write-bwlimit set, if write tikv is slow(might due to tikv io overload), and we cannot wait enough token to write to tikv within 15 minutes, it will report this error, we should retry on this

[errorVerbose="rate: Wait(n=28406) would exceed context deadline
github.com/pingcap/errors.AddStack
    /go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174
github.com/pingcap/errors.Trace
    /go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doWrite.func5
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/region_job.go:354
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doWrite
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/region_job.go:395
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).writeToTiKV
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/region_job.go:190
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).executeJob
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1436
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).startWorker
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1345
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.func5
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1678
golang.org/x/sync/errgroup.(*Group).Go.func1
    /go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1650"

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)
@D3Hunter D3Hunter added type/bug The issue is confirmed as a bug. severity/moderate component/ddl This issue is related to DDL of TiDB. labels Feb 28, 2024
@D3Hunter D3Hunter changed the title retry when write tikv is slow and rate limiter context timeout lightning: retry when write tikv is slow and rate limiter context timeout Feb 28, 2024
@D3Hunter D3Hunter added the component/lightning This issue is related to Lightning of TiDB. label Feb 28, 2024
@lance6716
Copy link
Contributor

Apart from retry, I can change the hard-coded 15 minutes timeout to a computed value from store-write-bwlimit

@lance6716 lance6716 self-assigned this Feb 29, 2024
@lance6716 lance6716 added affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. labels Feb 29, 2024
@lance6716 lance6716 mentioned this issue Feb 29, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Mar 1, 2024
@lance6716
lightning: retry for WaitN error caused by store-write-bwlimit (#51425)
8e29115 
close #51383
ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this issue Mar 25, 2024
@lance6716 @ti-chi-bot
lightning: retry for WaitN error caused by store-write-bwlimit (pingc…
b852e2b 
…ap#51425)

close pingcap#51383
@ti-chi-bot ti-chi-bot mentioned this issue Mar 25, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Mar 27, 2024
@ti-chi-bot
lightning: retry for WaitN error caused by store-write-bwlimit (#51425)…
6b8952b 
… (#52064)

close #51383
ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this issue Apr 12, 2024
@lance6716 @ti-chi-bot
lightning: retry for WaitN error caused by store-write-bwlimit (pingc…
bbd8cf6 
…ap#51425)

close pingcap#51383
@ti-chi-bot ti-chi-bot mentioned this issue Apr 12, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Apr 15, 2024
@ti-chi-bot
lightning: retry for WaitN error caused by store-write-bwlimit (#51425)…
a1b73d0 
… (#52581)

close #51383
@ti-chi-bot ti-chi-bot mentioned this issue May 7, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue May 7, 2024
@ti-chi-bot
lightning: retry for WaitN error caused by store-write-bwlimit (#51425)…
7e26e65 
… (#53067)

close #51383
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lance6716 lance6716

Labels
affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. component/ddl This issue is related to DDL of TiDB. component/lightning This issue is related to Lightning of TiDB. severity/moderate type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

lightning: retry for WaitN error caused by store-write-bwlimit lance6716/tidb
2 participants
@lance6716 @D3Hunter