Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create documentation for snapshots with hashed prefix path type #8196

Merged
merged 7 commits into from
Sep 11, 2024

Conversation

ashking94
Copy link
Member

@ashking94 ashking94 commented Sep 9, 2024

Description

This PR aims to add documentation for the following features -

  1. RFC proposed here - [RFC] Optimized Prefix Pattern for Shard-Level Files for Efficient Snapshots OpenSearch#15146.
  2. Add path prefix support to hashed prefix snapshots OpenSearch#15664
  3. Add prefix support to hashed prefix & infix path types on remote store OpenSearch#15557

Issues Resolved

Closes #8143

Version

This PR applies to version 2.17 and later.

Frontend features

If you're submitting documentation for an OpenSearch Dashboards feature, add a video that shows how a user will interact with the UI step by step. A voiceover is optional.
NONE

Checklist

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
    For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Copy link

github-actions bot commented Sep 9, 2024

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Copy link
Collaborator

@Naarcha-AWS Naarcha-AWS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of follow up questions for you @gbbafna and @ashking94.

@Naarcha-AWS Naarcha-AWS added 4 - Doc review PR: Doc review in progress v2.17.0 release-notes PR: Include this PR in the automated release notes labels Sep 10, 2024
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
@ashking94
Copy link
Member Author

A couple of follow up questions for you @gbbafna and @ashking94.

Thanks for working on this PR @Naarcha-AWS. I have responded to your question.

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
@Naarcha-AWS Naarcha-AWS added 5 - Editorial review PR: Editorial review in progress 6 - Done but waiting to merge PR: The work is done and ready to merge and removed 4 - Doc review PR: Doc review in progress labels Sep 10, 2024
Copy link
Collaborator

@natebower natebower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashking94 @Naarcha-AWS Please see my comments and changes and let me know if you have any questions. Thanks!

_api-reference/snapshots/create-repository.md Outdated Show resolved Hide resolved
_api-reference/snapshots/create-repository.md Show resolved Hide resolved
_api-reference/snapshots/create-repository.md Outdated Show resolved Hide resolved
_api-reference/snapshots/create-repository.md Outdated Show resolved Hide resolved
_api-reference/snapshots/create-repository.md Outdated Show resolved Hide resolved

- `FIXED`: Keeps the path structure in the existing hierarchical manner, such as, `<ROOT>/<BASE_PATH>/indices/<index-id>/0/<SHARD_BLOBS>`
- `HASHED_PREFIX`: Prepends a hashed prefix at the start of path for each unique shard ID, for example, `<ROOT>/<HASH-OF-INDEX-ID-AND-SHARD-ID>/<BASE_PATH>/indices/<index-id>/0/<SHARD_BLOBS>`.
- `HASHED_INFIX`: Appends a hashed prefix after the base path for each unique shard ID, for example, `<ROOT>/<BASE-PATH>/<HASH-OF-INDEX-ID-AND-SHARD-ID>/indices/<index-id>/0/<SHARD_BLOBS>`. The hash method used is `fnv_1a_composite_1, which uses the `FNV1a` hash function and generates a custom-encoded 64-bit hash value that scales well with most remote store options. `FNV1a` takes the most significant 6 bits to create a url-safe base64 character and the next 14 bits to create a binary string.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line above: Same question re: fnv_1a_composite_1. I believe it may have been in all caps in a previous PR. Also, it looks like you have an unclosed tic mark here.

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
@Naarcha-AWS Naarcha-AWS merged commit 9bc06e4 into opensearch-project:main Sep 11, 2024
6 checks passed
epugh pushed a commit to o19s/documentation-website that referenced this pull request Sep 12, 2024
…search-project#8196)

* Create documentation for snapshots with hashed prefix path type

Signed-off-by: Ashish Singh <ssashish@amazon.com>

* Add documentation on new cluster settings for fixed prefix

Signed-off-by: Ashish Singh <ssashish@amazon.com>

* Update create-repository.md

* Update create-repository.md

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

---------

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
5 - Editorial review PR: Editorial review in progress 6 - Done but waiting to merge PR: The work is done and ready to merge release-notes PR: Include this PR in the automated release notes v2.17.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[DOC] Create documentation for snapshots with hashed prefix path type
5 participants