Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] Optimise clone operation for incremental full cluster snapshots #16295

Closed
ashking94 opened this issue Oct 12, 2024 · 0 comments · Fixed by #16296
Closed

[Feature Request] Optimise clone operation for incremental full cluster snapshots #16295

ashking94 opened this issue Oct 12, 2024 · 0 comments · Fixed by #16296
Labels
enhancement Enhancement or improvement to existing feature or request Storage:Snapshots

Comments

@ashking94
Copy link
Member

ashking94 commented Oct 12, 2024

Is your feature request related to a problem? Please describe

During the development of shallow v1 snapshots, the clone operation code was modified to handle both shallow and existing full cluster incremental snapshots with the PR - #7496. However, in this PR, we have inadvertently introduced degraded experience for doc-rep cluster's clone snapshot operation where for each shard, there is dependency on repository data (index-N file for repository). If the index-N file size is above 512KB, then data needs to be downloaded from snapshot repository each time. This makes the clone snapshot operation slow. This can be reproduced by creating a lot of snapshots in a repo so that the index-N file size is above 512 KB.

Describe the solution you'd like

For doc-rep clusters, we don't need to depend on the repository data to decide which type of snapshot was taken. For docrep clusters, we only support full cluster incremental snapshots. In this case, we will check whether the cluster is remote store enabled or not, and if not, then directly jump to the part of cloning without seeking the repository data for any shard.

Related component

Storage:Snapshots

Describe alternatives you've considered

No response

Additional context

No response

@ashking94 ashking94 added enhancement Enhancement or improvement to existing feature or request untriaged labels Oct 12, 2024
@ashking94 ashking94 changed the title [Feature Request] Optimised snapshot clone operation for doc-rep clusters [Feature Request] Optimise snapshot clone operation for doc-rep clusters Oct 12, 2024
@ashking94 ashking94 changed the title [Feature Request] Optimise snapshot clone operation for doc-rep clusters [Feature Request] Optimise clone operation for incremental full cluster snapshots Oct 12, 2024
@github-project-automation github-project-automation bot moved this from 🆕 New to ✅ Done in Storage Project Board Oct 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancement or improvement to existing feature or request Storage:Snapshots
Projects
Status: ✅ Done
Development

Successfully merging a pull request may close this issue.

1 participant