Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dropping table data is not atomic #1192

Closed
JaySon-Huang opened this issue Nov 2, 2020 · 3 comments · Fixed by #4428
Closed

Dropping table data is not atomic #1192

JaySon-Huang opened this issue Nov 2, 2020 · 3 comments · Fixed by #4428
Assignees
Labels
severity/major type/bug The issue is confirmed as a bug.

Comments

@JaySon-Huang
Copy link
Contributor

JaySon-Huang commented Nov 2, 2020

Now table's data will be dropped after it has passed the GC time. (See SchemaSyncService::gc())
But dropping table data is not atomic. When there is a large amount of data / deployed with multi-disks, dropping a table data could cost tens of seconds. If TiFlash crashes during dropping table data, some broken data may be left on disks and make TiFlash can NOT restart normally.

Workaround:
We could see which table is broken by tracking the logfile. Ensure whether that table id is able to remove data from disks by querying TiDB. Manually remove those files left on disks.

@JaySon-Huang JaySon-Huang added the type/bug The issue is confirmed as a bug. label Nov 2, 2020
@ilovesoup
Copy link
Contributor

What is the status of this one? @JaySon-Huang @flowbehappy

@flowbehappy flowbehappy self-assigned this Mar 20, 2021
@flowbehappy
Copy link
Contributor

@ilovesoup We will discuss it next week.

@lidezhu
Copy link
Contributor

lidezhu commented Apr 20, 2022

Table is regarded to be dropped after its metafile has been removed which is atomic. And metadata is removed before remove data.
So there is just some potential that left some data on disk but tiflash can still work normally even it crashes in the process of dropping a table. No need to fix it on previous version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants