DBaaS v5.0.0 restore failed

[apollodafoni]

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
   If possible, provide a recipe for reproducing the error.

2. What did you expect to see?

3. What did you see instead?

I0607 02:17:20.643371 11 restore.go:90] [2021/06/07 02:17:20.642 +00:00] [ERROR] [import.go:267] ["download file failed"] [file="{name=1_584_32_77269f908c550594852b6adda5f616af55b7e38d973e3eb9681919c123dae892_1622622232312_write.sst,CF=write,sha256=08b8e4032cbbd54d816c37a5bcb59adb18f86894a4cd0eb5cd629b432530b4b6,startKey=?,endKey=?,startVersion=0,endVersion=425360675683958785,totalKvs=960000,totalBytes=34560000,CRC64Xor=9602817452552570455}"] [region="{ID=1756,startKey=?,endKey=?,epoch="conf_ver:5 version:327 ",peers="id:1757 store_id:1 ,id:1758 store_id:4 ,id:1759 store_id:5 "}"] [startKey=?] [endKey=?] [error="file already exists, /var/lib/tikv/import/2937a1b6-0161-4c36-a3f9-cdf6d4b363eb_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/01fed403-f53c-430e-98d5-a55da9de56e7_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/75d08c69-a99d-4a30-9763-6edcbef0aef0_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/bb9cfa9d-9788-4c55-b5cc-a153637848f1_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/243fabf6-610b-4385-9ad8-f4175fba4b43_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/042fe8c9-f6af-406b-af98-6467a4da52b9_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/f40d0451-7c1a-427a-ae32-9a98162bc359_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/bfd402c3-0744-4bb0-9ac1-219f51d42987_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed"] [stack="github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\n\tgithub.com/pingcap/br@/pkg/restore/import.go:267\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\tgithub.com/pingcap/br@/pkg/utils/retry.go:46\ngithub.com/pingcap/br/pkg/restore.(*FileImporter).Import\n\tgithub.com/pingcap/br@/pkg/restore/import.go:222\ngithub.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\n\tgithub.com/pingcap/br@/pkg/restore/client.go:584\ngithub.com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n\tgithub.com/pingcap/br@/pkg/utils/worker.go:63\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
I0607 02:17:20.643438 11 restore.go:90] [2021/06/07 02:17:20.642 +00:00] [WARN] [backoff.go:77] ["unexcepted error, stop to retry"] [error="file already exists, /var/lib/tikv/import/2937a1b6-0161-4c36-a3f9-cdf6d4b363eb_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/01fed403-f53c-430e-98d5-a55da9de56e7_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/75d08c69-a99d-4a30-9763-6edcbef0aef0_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/bb9cfa9d-9788-4c55-b5cc-a153637848f1_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/243fabf6-610b-4385-9ad8-f4175fba4b43_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/042fe8c9-f6af-406b-af98-6467a4da52b9_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/f40d0451-7c1a-427a-ae32-9a98162bc359_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed; file already exists, /var/lib/tikv/import/bfd402c3-0744-4bb0-9ac1-219f51d42987_1756_5_327_write.sst: [BR:KV:ErrKVDownloadFailed]download sst failed"] [errorVerbose="the following errors occurred:\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n file already exists, /var/lib/tikv/import/2937a1b6-0161-4c36-a3f9-cdf6d4b363eb_1756_5_327_write.sst\n github.com/pingcap/br/pkg/restore.(*FileImporter).downloadSST\n \tgithub.com/pingcap/br@/pkg/restore/import.go:397\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1.1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:244\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:239\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import\n \tgithub.com/pingcap/br@/pkg/restore/import.go:222\n github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\n \tgithub.com/pingcap/br@/pkg/restore/client.go:584\n github.com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n \tgithub.com/pingcap/br@/pkg/utils/worker.go:63\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\n runtime.goexit\n \truntime/asm_amd64.s:1357\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n file already exists, /var/lib/tikv/import/01fed403-f53c-430e-98d5-a55da9de56e7_1756_5_327_write.sst\n github.com/pingcap/br/pkg/restore.(*FileImporter).downloadSST\n \tgithub.com/pingcap/br@/pkg/restore/import.go:397\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1.1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:244\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:239\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import\n \tgithub.com/pingcap/br@/pkg/restore/import.go:222\n github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\n \tgithub.com/pingcap/br@/pkg/restore/client.go:584\n github.com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n \tgithub.com/pingcap/br@/pkg/utils/worker.go:63\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\n runtime.goexit\n \truntime/asm_amd64.s:1357\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n file already exists, /var/lib/tikv/import/75d08c69-a99d-4a30-9763-6edcbef0aef0_1756_5_327_write.sst\n github.com/pingcap/br/pkg/restore.(*FileImporter).downloadSST\n \tgithub.com/pingcap/br@/pkg/restore/import.go:397\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1.1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:244\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:239\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import\n \tgithub.com/pingcap/br@/pkg/restore/import.go:222\n github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\n \tgithub.com/pingcap/br@/pkg/restore/client.go:584\n github.com/pingcap/br/pkg/utils.(*Worke
rPool).ApplyOnErrorGroup.func1\n \tgithub.com/pingcap/br@/pkg/utils/worker.go:63\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15b
c9/errgroup/errgroup.go:57\n runtime.goexit\n \truntime/asm_amd64.s:1357\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n file already exists, /var/lib/tikv/import/bb9cfa9d-9788
-4c55-b5cc-a153637848f1_1756_5_327_write.sst\n github.com/pingcap/br/pkg/restore.(*FileImporter).downloadSST\n \tgithub.com/pingcap/br@/pkg/restore/import.go:397\n github.com/pingcap
/br/pkg/restore.(*FileImporter).Import.func1.1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:244\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/r
etry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:239\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgi
thub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import\n \tgithub.com/pingcap/br@/pkg/restore/import.go:222\n github.com/pingcap/br/pkg
/restore.(*Client).RestoreFiles.func2\n \tgithub.com/pingcap/br@/pkg/restore/client.go:584\n github.com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n \tgithub.com/ping
cap/br@/pkg/utils/worker.go:63\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\n runtime.goexit\n
truntime/asm_amd64.s:1357\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n file already exists, /var/lib/tikv/import/243fabf6-610b-4385-9ad8-f4175fba4b43_1756_5_327_write.sst\n git
hub.com/pingcap/br/pkg/restore.(*FileImporter).downloadSST\n \tgithub.com/pingcap/br@/pkg/restore/import.go:397\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1.1\n
tgithub.com/pingcap/br@/pkg/restore/import.go:244\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*F
ileImporter).Import.func1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:239\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n githu
b.com/pingcap/br/pkg/restore.(*FileImporter).Import\n \tgithub.com/pingcap/br@/pkg/restore/import.go:222\n github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\n \tgithub.c
om/pingcap/br@/pkg/restore/client.go:584\n github.com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n \tgithub.com/pingcap/br@/pkg/utils/worker.go:63\n golang.org/x/sync
/errgroup.(*Group).Go.func1\n \tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\n runtime.goexit\n \truntime/asm_amd64.s:1357\n - [BR:KV:ErrKVDownloadFa
iled]download sst failed\n file already exists, /var/lib/tikv/import/042fe8c9-f6af-406b-af98-6467a4da52b9_1756_5_327_write.sst\n github.com/pingcap/br/pkg/restore.(*FileImporter).downlo
adSST\n \tgithub.com/pingcap/br@/pkg/restore/import.go:397\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1.1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:244\n
github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\n \tgithub.com/pingcap/b
r@/pkg/restore/import.go:239\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import\n
\tgithub.com/pingcap/br@/pkg/restore/import.go:222\n github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\n \tgithub.com/pingcap/br@/pkg/restore/client.go:584\n github.
com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n \tgithub.com/pingcap/br@/pkg/utils/worker.go:63\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \tgolang.org/x/sync@
v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\n runtime.goexit\n \truntime/asm_amd64.s:1357\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n file already exists, /v
ar/lib/tikv/import/f40d0451-7c1a-427a-ae32-9a98162bc359_1756_5_327_write.sst\n github.com/pingcap/br/pkg/restore.(*FileImporter).downloadSST\n \tgithub.com/pingcap/br@/pkg/restore/impor
t.go:397\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1.1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:244\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgi
thub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:239\n github.com/pingcap/
br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import\n \tgithub.com/pingcap/br@/pkg/restore/import.go:22
2\n github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\n \tgithub.com/pingcap/br@/pkg/restore/client.go:584\n github.com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGr
oup.func1\n \tgithub.com/pingcap/br@/pkg/utils/worker.go:63\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.
go:57\n runtime.goexit\n \truntime/asm_amd64.s:1357\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n file already exists, /var/lib/tikv/import/bfd402c3-0744-4bb0-9ac1-219f51d429
87_1756_5_327_write.sst\n github.com/pingcap/br/pkg/restore.(*FileImporter).downloadSST\n \tgithub.com/pingcap/br@/pkg/restore/import.go:397\n github.com/pingcap/br/pkg/restore.(*Fil
eImporter).Import.func1.1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:244\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/pkg/utils/retry.go:46\n githu
b.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\n \tgithub.com/pingcap/br@/pkg/restore/import.go:239\n github.com/pingcap/br/pkg/utils.WithRetry\n \tgithub.com/pingcap/br@/
pkg/utils/retry.go:46\n github.com/pingcap/br/pkg/restore.(*FileImporter).Import\n \tgithub.com/pingcap/br@/pkg/restore/import.go:222\n github.com/pingcap/br/pkg/restore.(*Client).Re
storeFiles.func2\n \tgithub.com/pingcap/br@/pkg/restore/client.go:584\n github.com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n \tgithub.com/pingcap/br@/pkg/utils/wor
ker.go:63\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\n runtime.goexit\n \truntime/asm_amd64.s:
1357"]
I0607 02:17:20.644808 11 restore.go:90] [2021/06/07 02:17:20.643 +00:00] [INFO] [client.go:580] ["import file done"] [file="

{name=1_584_32_77269f908c550594852b6adda5f616af55b7e38d973e3eb 9681919c123dae892_1622622232312_write.sst,CF=write,sha256=08b8e4032cbbd54d816c37a5bcb59adb18f86894a4cd0eb5cd629b432530b4b6,startKey=?,endKey=?,startVersion=0,endVersion=425360675683958785,tot alKvs=960000,totalBytes=34560000,CRC64Xor=9602817452552570455}
"] [take=54.605267947s]

4. What version of BR and TiDB/TiKV/PD are you using?

   v5.0.0

5. Operation logs

   • Please upload br.log for BR if possible
   • Please upload tidb-lightning.log for TiDB-Lightning if possible
   • Please upload tikv-importer.log from TiKV-Importer if possible
   • Other interesting logs

6. Configuration of the cluster and the task

   • tidb-lightning.toml for TiDB-Lightning if possible
   • tikv-importer.toml for TiKV-Importer if possible
   • topology.yml if deployed by TiUP

dbaas使用配置参看：
https://docs.google.com/spreadsheets/d/1KFaqPTNN9BHq1sUaJOaous4DbWUo4nz5ZBDM5kQimCQ/edit#gid=0

7. Screenshot/exported-PDF of Grafana dashboard or metrics' graph in Prometheus if possible