Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TiFlash can't gc totally after set tiflash replica 0 #4414

Closed
lilinghai opened this issue Mar 24, 2022 · 3 comments · Fixed by #4668
Closed

TiFlash can't gc totally after set tiflash replica 0 #4414

lilinghai opened this issue Mar 24, 2022 · 3 comments · Fixed by #4668

Comments

@lilinghai
Copy link

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

udmASBWGW8
PCj4V4N2Od

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

v6.0

@lidezhu
Copy link
Contributor

lidezhu commented Mar 28, 2022

Reproduce it locally. The remaining data doesn't correspond to any region range in the tiflash node. And this is a possible case in the cluster, and tikv handle this case by remove garbage ranges at restart: https://github.com/tikv/tikv/blob/b5559844ce26a453900ca64c86902647dc807e28/components/raftstore/src/store/fsm/store.rs.

So this is a normal behavior and is not a bug.

image
image

The remaining data is in the range: [216172782114912251, 216172782117644718]
The two neighbor region ranges ever on this node are: [144115188085470701, 216172782114912251) and [216172782117644721, 216172782121487120).

tiflash log:
https://drive.google.com/file/d/1PY5F-STVXPDSqj2alhvNC-PaMyH7Zl-e/view?usp=sharing
https://drive.google.com/file/d/11vVeoR_rlpoduxAKFvVZkhtHeGRqkh3O/view?usp=sharing
https://drive.google.com/file/d/1JP9-Mi0YcfGtLeye9sRCg2232DltDWza/view?usp=sharing

@lidezhu
Copy link
Contributor

lidezhu commented Mar 30, 2022

This pr #4500 just fix the following two problems:

  1. some empty segments cannot be merged;
  2. the stable size is not 0 for empty dmfile because there is a '0' in every index file in such case;

The problem that some data left on tiflash node when all region is removed will be fixed later. @CalvinNeo

@CalvinNeo
Copy link
Member

Sending snapshots may change start_key and end_key, TiKV will delete range regularly, or at least before shutdown. However, these messages will not be delivered to TiFlash.

ti-chi-bot pushed a commit that referenced this issue Apr 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants