importer: Check whether file exist before importing (#9522) #9544

ti-srebot · 2021-01-22T05:54:04Z

cherry-pick #9522 to release-4.0
You can switch your code base to this Pull Request by using git-extras:

# In tikv repo:
git pr https://github.com/tikv/tikv/pull/9544

After apply modifications, you can push your change to this PR via:

git push git@github.com:ti-srebot/tikv.git pr/9544:release-4.0-f4be3a0bf2fa

What problem does this PR solve?

Issue Number: close #9496

Problem Summary:
If TiKV has applied one command of ingest but tidb-lightning(import-client) thinks this request has failed, it would send the same ingest-request to TiKV again, which may cause TiKV panic because the file has been ingested and could not be found.
There is many case which may cause request failed while it has been applied. Such as:

Network lost some packets of ingest-response.
The command had been copied to more than half of the members of raft-group but the original leader crashed before the entry of this command committed, then the new leader committed this entry but the client has received fail messages by the response from the original leader.

What is changed and how it works?

We can propose one ingest-request when all the following conditions are met:

There is not any pending ingest-request for this sst.
The current leader has applied to current term, and we can ensure that the file which we want to ingest exist.
The ingest file must be removed before we return response to client, so that the new request will never see the same file again.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
PR to update pingcap/tidb-ansible:
Need to cherry-pick to the release branch

Check List

Tests

Unit test

Side effects

Performance regression
- It may increase the duration of ingest request.
Breaking backward compatibility

Release note

No release note.

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2021-01-22T05:54:08Z

/run-all-tests

ti-srebot · 2021-01-22T05:54:18Z

@Little-Wallace you're already a collaborator in bot's repo.

kennytm · 2021-01-24T12:48:23Z

components/raftstore/src/store/fsm/apply.rs

@@ -361,8 +376,14 @@ impl<W: WriteBatch + WriteBatchVecExt<RocksEngine>> ApplyContext<W> {
            exec_ctx: None,
            use_delete_range: cfg.use_delete_range,
            yield_duration: cfg.apply_yield_duration.0,
+<<<<<<< HEAD


Merge conflict.

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

gengliqi · 2021-01-25T10:36:14Z

/lgtm

Little-Wallace · 2021-01-25T10:37:36Z

/merge

ti-chi-bot · 2021-01-25T10:37:37Z

@Little-Wallace: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-01-25T10:37:39Z

@Little-Wallace: /merge is only allowed for the committers in list.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

NingLin-P

/LGTM

ti-chi-bot · 2021-01-25T12:46:12Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

NingLin-P
gengliqi

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

NingLin-P · 2021-01-26T05:42:03Z

/merge

ti-chi-bot · 2021-01-26T05:42:04Z

@NingLin-P: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-01-26T05:42:07Z

This pull request has been accepted and is ready to merge.

Commit hash: e848c43

Little-Wallace · 2021-01-26T06:00:13Z

/test

Little-Wallace · 2021-01-26T06:07:38Z

/test

Little-Wallace · 2021-01-26T06:20:10Z

/test

Little-Wallace · 2021-01-26T06:45:13Z

/test

…#9544) cherry-pick tikv#9522 to release-4.0 You can switch your code base to this Pull Request by using [git-extras](https://github.com/tj/git-extras): ```bash # In tikv repo: git pr tikv#9544 ``` After apply modifications, you can push your change to this PR via: ```bash git push git@github.com:ti-srebot/tikv.git pr/9544:release-4.0-f4be3a0bf2fa ``` ---  ### What problem does this PR solve? Issue Number: close tikv#9496 Problem Summary: If TiKV has applied one command of ingest but tidb-lightning(import-client) thinks this request has failed, it would send the same ingest-request to TiKV again, which may cause TiKV panic because the file has been ingested and could not be found. There is many case which may cause request failed while it has been applied. Such as: 1. Network lost some packets of ingest-response. 2. The command had been copied to more than half of the members of raft-group but the original leader crashed before the entry of this command committed, then the new leader committed this entry but the client has received fail messages by the response from the original leader. ### What is changed and how it works? We can propose one ingest-request when all the following conditions are met: 1. There is not any pending ingest-request for this sst. 2. The current leader has applied to current term, and we can ensure that the file which we want to ingest exist. 3. The ingest file must be removed before we return response to client, so that the new request will never see the same file again. ### Related changes - PR to update `pingcap/docs`/`pingcap/docs-cn`: - PR to update `pingcap/tidb-ansible`: - Need to cherry-pick to the release branch ### Check List  Tests  - Unit test Side effects - Performance regression - It may increase the duration of ingest request. - Breaking backward compatibility ### Release note  - No release note.

cherry pick tikv#9522 to release-4.0

770ca7d

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-chi-bot added the do-not-merge/cherry-pick-not-approved label Jan 22, 2021

ti-srebot mentioned this pull request Jan 22, 2021

importer: Check whether file exist before importing #9522

Merged

ti-srebot added component/backup-restore Component: backup, import, external_storage sig/raft Component: Raft, RaftStore, etc. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. type/cherry-pick Type: PR - Cherry pick labels Jan 22, 2021

ti-srebot requested review from gengliqi and kennytm January 22, 2021 05:54

ti-srebot added this to the v4.0.10 milestone Jan 22, 2021

ti-srebot assigned Little-Wallace Jan 22, 2021

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 22, 2021

kennytm reviewed Jan 24, 2021

View reviewed changes

fix conflict and upgrade future of sst_import

a92a061

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 24, 2021

jebter modified the milestones: v4.0.10, v4.0.11 Jan 25, 2021

jebter removed the do-not-merge/cherry-pick-not-approved label Jan 25, 2021

ti-chi-bot added the do-not-merge/cherry-pick-not-approved label Jan 25, 2021

jebter removed the do-not-merge/cherry-pick-not-approved label Jan 25, 2021

ti-chi-bot added the do-not-merge/cherry-pick-not-approved label Jan 25, 2021

jebter removed the do-not-merge/cherry-pick-not-approved label Jan 25, 2021

ti-chi-bot added the do-not-merge/cherry-pick-not-approved label Jan 25, 2021

fix conflict type

baff35e

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

ti-chi-bot removed the do-not-merge/cherry-pick-not-approved label Jan 25, 2021

Merge branch 'release-4.0' into release-4.0-f4be3a0bf2fa

55575f5

Little-Wallace requested a review from kennytm January 25, 2021 08:47

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jan 25, 2021

NingLin-P added the sig/migrate label Jan 25, 2021

NingLin-P approved these changes Jan 25, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jan 25, 2021

Little-Wallace added 2 commits January 26, 2021 10:41

Merge branch 'release-4.0' into release-4.0-f4be3a0bf2fa

fc1416a

Merge branch 'release-4.0' into release-4.0-f4be3a0bf2fa

e848c43

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jan 26, 2021

ti-chi-bot merged commit 9a98501 into tikv:release-4.0 Jan 26, 2021

NingLin-P mentioned this pull request Feb 1, 2021

releases: add tidb v4.0.11 release notes pingcap/docs#4761

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

importer: Check whether file exist before importing (#9522) #9544

importer: Check whether file exist before importing (#9522) #9544

ti-srebot commented Jan 22, 2021 •

edited

Loading

ti-srebot commented Jan 22, 2021

ti-srebot commented Jan 22, 2021

kennytm Jan 24, 2021

Little-Wallace Jan 25, 2021

gengliqi commented Jan 25, 2021

Little-Wallace commented Jan 25, 2021

ti-chi-bot commented Jan 25, 2021

ti-chi-bot commented Jan 25, 2021

NingLin-P left a comment

ti-chi-bot commented Jan 25, 2021

NingLin-P commented Jan 26, 2021

ti-chi-bot commented Jan 26, 2021

ti-chi-bot commented Jan 26, 2021

Little-Wallace commented Jan 26, 2021

Little-Wallace commented Jan 26, 2021

Little-Wallace commented Jan 26, 2021

Little-Wallace commented Jan 26, 2021

importer: Check whether file exist before importing (#9522) #9544

importer: Check whether file exist before importing (#9522) #9544

Conversation

ti-srebot commented Jan 22, 2021 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

ti-srebot commented Jan 22, 2021

ti-srebot commented Jan 22, 2021

kennytm Jan 24, 2021

Choose a reason for hiding this comment

Little-Wallace Jan 25, 2021

Choose a reason for hiding this comment

gengliqi commented Jan 25, 2021

Little-Wallace commented Jan 25, 2021

ti-chi-bot commented Jan 25, 2021

ti-chi-bot commented Jan 25, 2021

NingLin-P left a comment

Choose a reason for hiding this comment

ti-chi-bot commented Jan 25, 2021

NingLin-P commented Jan 26, 2021

ti-chi-bot commented Jan 26, 2021

ti-chi-bot commented Jan 26, 2021

Little-Wallace commented Jan 26, 2021

Little-Wallace commented Jan 26, 2021

Little-Wallace commented Jan 26, 2021

Little-Wallace commented Jan 26, 2021

ti-srebot commented Jan 22, 2021 •

edited

Loading