Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] [Remote State] Garbage collect dangling metadata files in remote state repo #10686

Open
linuxpi opened this issue Oct 18, 2023 · 0 comments
Open

[BUG] [Remote State] Garbage collect dangling metadata files in remote state repo #10686

linuxpi opened this issue Oct 18, 2023 · 0 comments
Assignees
@linuxpi
Labels
bug Something isn't working Cluster Manager Storage:Remote Storage Issues and PRs relating to data and metadata storage

Comments

@linuxpi
Copy link
Collaborator

linuxpi commented Oct 18, 2023

Describe the bug

  • During remote state upload to remote, we upload multiple IndexMetadata files, single global metadata file and then upload the manifest.
  • Manifest contains reference to IndexMetadata and GlobalMetadata files
  • If manifest upload fails and metadata upload succeeds, we end up with dangling metadata files which are not referenced by any manifest.
  • Since current GC logic only considers stale metadata files referenced by stale manifest files. The referenced metadata files would never be cleaned up.

Expected behavior
GC should also cleanup any dangling/un-referenced files in remote state repo
@linuxpi linuxpi added bug Something isn't working untriaged labels Oct 18, 2023
This was referenced Oct 18, 2023
Closed
Merged
@linuxpi linuxpi self-assigned this Oct 23, 2023
@linuxpi linuxpi added Cluster Manager Storage Issues and PRs relating to data and metadata storage Storage:Remote v2.12.0 Issues and PRs related to version 2.12.0 and removed untriaged v2.12.0 Issues and PRs related to version 2.12.0 labels Oct 23, 2023
@linuxpi linuxpi moved this from 🆕 New to Later (6 months plus) in Storage Project Board Mar 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@linuxpi linuxpi

Labels
bug Something isn't working Cluster Manager Storage:Remote Storage Issues and PRs relating to data and metadata storage
Projects
Cluster Manager Project Board
Status: 🆕 New
Milestone
No milestone
Development

No branches or pull requests
1 participant
@linuxpi