Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

v7.0.0 br restore hit error "[BR:KV:ErrKVEpochNotMatch]epoch not match" #42924

Closed
mayjiang0203 opened this issue Apr 11, 2023 · 5 comments · Fixed by #42972
Closed

v7.0.0 br restore hit error "[BR:KV:ErrKVEpochNotMatch]epoch not match" #42924

mayjiang0203 opened this issue Apr 11, 2023 · 5 comments · Fixed by #42972
Assignees
@YuJuncen
Labels
affects-7.1 affects-7.5 component/br This issue is related to BR of TiDB. severity/major type/bug The issue is confirmed as a bug. wontfix This issue will not be fixed.

Comments

@mayjiang0203
Copy link

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. backup full via br in v6.6.0
  2. restore the backup in v7.0.0

2. What did you expect to see? (Required)

restore succeeded.

3. What did you see instead (Required)

..........................> 21.81%Full Restore <-----------------------------------------------------------------------------> 100.00%[2023/04/11 06:18:34.548 +00:00] [INFO] [collector.go:73] ["Full Restore failed summary"] [total-ranges=6242] [ranges-succeed=6241] [ranges-failed=1] [restore-ranges=53400] [unit-name=file] [error="[BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match"] [errorVerbose="the following errors occurred:\n -  [BR:KV:ErrKVEpochNotMatch]epoch not match\n    github.com/pingcap/errors.AddStack\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\n    github.com/pingcap/errors.Trace\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ingest\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:847\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:53\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1206\n    github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go1.20.2/src/runtime/asm_arm64.s:1172\n -  [BR:KV:ErrKVEpochNotMatch]epoch not match\n    github.com/pingcap/errors.AddStack\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\n    github.com/pingcap/errors.Trace\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ingest\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:847\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:53\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1206\n    github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go1.20.2/src/runtime/asm_arm64.s:1172\n -  [BR:KV:ErrKVEpochNotMatch]epoch not match\n    github.com/pingcap/errors.AddStack\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\n    github.com/pingcap/errors.Trace\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ingest\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:847\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:53\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1206\n    github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go1.20.2/src/runtime/asm_arm64.s:1172\n -  [BR:KV:ErrKVEpochNotMatch]epoch not match\n    github.com/pingcap/errors.AddStack\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\n    github.com/pingcap/errors.Trace\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ingest\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:847\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:53\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1206\n    github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go1.20.2/src/runtime/asm_arm64.s:1172\n -  [BR:KV:ErrKVEpochNotMatch]epoch not match\n    github.com/pingcap/errors.AddStack\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\n    github.com/pingcap/errors.Trace\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ingest\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:847\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:53\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1206\n    github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go1.20.2/src/runtime/asm_arm64.s:1172\n -  [BR:KV:ErrKVEpochNotMatch]epoch not match\n    github.com/pingcap/errors.AddStack\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\n    github.com/pingcap/errors.Trace\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ingest\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:847\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:53\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1206\n    github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go1.20.2/src/runtime/asm_arm64.s:1172\n -  [BR:KV:ErrKVEpochNotMatch]epoch not match\n    github.com/pingcap/errors.AddStack\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\n    github.com/pingcap/errors.Trace\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ingest\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:847\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:53\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1206\n    github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go1.20.2/src/runtime/asm_arm64.s:1172\n -  [BR:KV:ErrKVEpochNotMatch]epoch not match\n    github.com/pingcap/errors.AddStack\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\n    github.com/pingcap/errors.Trace\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ingest\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:847\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:53\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1206\n    github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go1.20.2/src/runtime/asm_arm64.s:1172\n -  [BR:KV:ErrKVEpochNotMatch]epoch not match\n    github.com/pingcap/errors.AddStack\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\n    github.com/pingcap/errors.Trace\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ingest\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:847\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:53\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1206\n    github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go1.20.2/src/runtime/asm_arm64.s:1172\n -  [BR:KV:ErrKVEpochNotMatch]epoch not match\n    github.com/pingcap/errors.AddStack\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\n    github.com/pingcap/errors.Trace\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ingest\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:847\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:53\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1206\n    github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go1.20.2/src/runtime/asm_arm64.s:1172\n -  [BR:KV:ErrKVEpochNotMatch]epoch not match\n    github.com/pingcap/errors.AddStack\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\n    github.com/pingcap/errors.Trace\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ingest\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:847\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:53\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1206\n    github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go1.20.2/src/runtime/asm_arm64.s:1172\n -  [BR:KV:ErrKVEpochNotMatch]epoch not match\n    github.com/pingcap/errors.AddStack\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\n    github.com/pingcap/errors.Trace\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ingest\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:847\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:53\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1206\n    github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go1.20.2/src/runtime/asm_arm64.s:1172\n -  [BR:KV:ErrKVEpochNotMatch]epoch not match\n    github.com/pingcap/errors.AddStack\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\n    github.com/pingcap/errors.Trace\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ingest\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:847\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:53\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1206\n    github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go1.20.2/src/runtime/asm_arm64.s:1172\n -  [BR:KV:ErrKVEpochNotMatch]epoch not match\n    github.com/pingcap/errors.AddStack\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\n    github.com/pingcap/errors.Trace\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ingest\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:847\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:53\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1206\n    github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go1.20.2/src/runtime/asm_arm64.s:1172\n -  [BR:KV:ErrKVEpochNotMatch]epoch not match\n    github.com/pingcap/errors.AddStack\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\n    github.com/pingcap/errors.Trace\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ingest\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:847\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:53\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1206\n    github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go1.20.2/src/runtime/asm_arm64.s:1172\n -  [BR:KV:ErrKVEpochNotMatch]epoch not match\n    github.com/pingcap/errors.AddStack\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\n    github.com/pingcap/errors.Trace\n    \t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ingest\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:847\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:551\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:53\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n    github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1206\n    github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n    \t/var/lib/docker/jenkins/workspace/build-common@4/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go1.20.2/src/runtime/asm_arm64.s:1172"]
Error: [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match; [BR:KV:ErrKVEpochNotMatch]epoch not match

4. What is your TiDB version? (Required)

sh-5.1# /br --version
Release Version: v7.0.0
Git Commit Hash: 7376954
Git Branch: heads/refs/tags/v7.0.0
Go Version: go1.20.2
UTC Build Time: 2023-03-29 13:21:57
Race Enabled: false
@mayjiang0203 mayjiang0203 added the type/bug The issue is confirmed as a bug. label Apr 11, 2023
@mayjiang0203
Copy link
Author

mayjiang0203 commented Apr 11, 2023 •
edited
Loading

/remove-severity critical
/severity major

@ti-chi-bot ti-chi-bot added severity/critical may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 labels Apr 11, 2023
@YuJuncen YuJuncen mentioned this issue Apr 12, 2023
Merged
4 tasks
@YuJuncen YuJuncen added affects-6.1 affects-6.5 affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects 5.4.x versions. and removed may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-6.1 may-affects-6.5 labels Apr 12, 2023
@ti-chi-bot ti-chi-bot added severity/major may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. and removed severity/critical labels Apr 12, 2023
@YuJuncen YuJuncen removed may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. labels Apr 17, 2023
ti-chi-bot pushed a commit that referenced this issue Apr 18, 2023
@YuJuncen
restore: split between tables (#42972)
36c59c7 
close #42924
This was referenced Apr 18, 2023
Open
Open
Closed
Closed
@YuJuncen YuJuncen mentioned this issue Apr 26, 2023
Merged
@YuJuncen YuJuncen removed affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects 5.4.x versions. affects-6.1 affects-6.5 labels Apr 26, 2023
@YuJuncen
Copy link
Contributor

This change may broke the PD server. We are going to revert that.

ti-chi-bot bot pushed a commit that referenced this issue May 5, 2023
@YuJuncen
restore: Revert "split between tables" (#43435)
8728abc 
ref tikv/tikv#14641, ref #42924
@ti-chi-bot ti-chi-bot mentioned this issue May 5, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this issue May 8, 2023
@ti-chi-bot
restore: Revert "split between tables" (#43435) (#43547)
8b2d294 
ref tikv/tikv#14641, ref #42924
@YuJuncen YuJuncen reopened this Jul 20, 2023
@jebter jebter added the component/br This issue is related to BR of TiDB. label Jul 27, 2023
@BornChanger
Copy link
Contributor

We don't have proper solution to this, but turning on restore checkpoint can get mitigation.

@BornChanger
Copy link
Contributor

From BR 7.6 and above, full restore sst download/ingest phase is after region split/scatter, such issue won't reoccur.

@BornChanger BornChanger added the wontfix This issue will not be fixed. label Oct 21, 2024
@BornChanger
Copy link
Contributor

given current situation, we won't fix it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@YuJuncen YuJuncen

Labels
affects-7.1 affects-7.5 component/br This issue is related to BR of TiDB. severity/major type/bug The issue is confirmed as a bug. wontfix This issue will not be fixed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

restore: split between tables YuJuncen/tidb
5 participants
@jebter @mayjiang0203 @YuJuncen @ti-chi-bot @BornChanger