Enhance searchable snapshots to enable a read-only view of older snapshots #5429

kartg · 2022-12-01T19:10:03Z

Description

This change removes the guardrails around N-1 backward compatibility and uses Lucene's "expert" APIs to read snapshots (Lucene segments) older than N-1 on a best-effort basis. The functionality is gated by an additional feature flag, separate from the searchable snapshots flag. Note that the Lucene integration is rather inefficient because the necessary "expert" Lucene APIs are still package-private.

Signed-off-by: Kartik Ganesh gkart@amazon.com

Issues Resolved

closes #5451

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2022-12-01T19:49:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/7516/
CommitID: dfe549a

codecov-commenter · 2022-12-01T19:53:25Z

Codecov Report

Merging #5429 (b95e49c) into 2.x (6ac361f) will increase coverage by 0.16%.
The diff coverage is 61.42%.

@@             Coverage Diff              @@
##                2.x    #5429      +/-   ##
============================================
+ Coverage     70.52%   70.68%   +0.16%     
- Complexity    58450    58500      +50     
============================================
  Files          4724     4725       +1     
  Lines        279961   280011      +50     
  Branches      40833    40838       +5     
============================================
+ Hits         197435   197935     +500     
+ Misses        66064    65555     -509     
- Partials      16462    16521      +59

Impacted Files	Coverage Δ
...va/org/opensearch/cluster/block/ClusterBlocks.java	`84.61% <0.00%> (-0.08%)`	⬇️
.../java/org/opensearch/common/util/FeatureFlags.java	`50.00% <ø> (ø)`
...in/java/org/opensearch/indices/IndicesService.java	`63.49% <0.00%> (-7.31%)`	⬇️
...rg/opensearch/indices/recovery/RecoveryTarget.java	`75.29% <0.00%> (+1.76%)`	⬆️
...rch/repositories/blobstore/FileRestoreContext.java	`66.27% <0.00%> (-6.34%)`	⬇️
...rc/main/java/org/opensearch/index/store/Store.java	`79.76% <30.00%> (-1.35%)`	⬇️
...in/java/org/opensearch/index/shard/IndexShard.java	`70.85% <33.33%> (-0.06%)`	⬇️
.../java/org/opensearch/snapshots/RestoreService.java	`53.85% <33.33%> (-0.48%)`	⬇️
...org/opensearch/cluster/metadata/IndexMetadata.java	`83.95% <50.00%> (+0.60%)`	⬆️
...rc/main/java/org/opensearch/index/IndexModule.java	`81.81% <50.00%> (-0.47%)`	⬇️
... and 536 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

server/src/main/java/org/opensearch/Version.java

server/src/main/java/org/opensearch/index/engine/ReadOnlyEngine.java

server/src/main/java/org/opensearch/index/store/Store.java

server/src/main/java/org/opensearch/index/shard/IndexShard.java

server/src/main/java/org/opensearch/common/lucene/Lucene.java

server/src/main/java/org/opensearch/cluster/metadata/IndexMetadata.java

server/src/main/java/org/opensearch/common/lucene/Lucene.java

github-actions · 2022-12-06T03:25:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/7629/
CommitID: 8888c54

github-actions · 2022-12-06T05:17:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/7631/
CommitID: f3e909b

github-actions · 2022-12-06T22:59:52Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/7707/
CommitID: 54a8ebf

github-actions · 2022-12-07T01:59:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/7717/
CommitID: 8fd8800

gregschohn

Please add a comment - maybe where you use the testIndex file or a readme file alongside it - to show how you generated that file. A set of commands and specific versions would be great.

server/src/main/java/org/opensearch/common/lucene/Lucene.java

server/src/main/java/org/opensearch/common/util/FeatureFlags.java

server/src/main/java/org/opensearch/index/IndexSettings.java

server/src/main/java/org/opensearch/index/engine/ReadOnlyEngine.java

server/src/main/java/org/opensearch/indices/IndicesService.java

server/src/main/java/org/opensearch/snapshots/RestoreService.java

server/src/test/java/org/opensearch/common/lucene/LuceneTests.java

server/src/test/java/org/opensearch/index/IndexSettingsTests.java

server/src/test/java/org/opensearch/index/engine/ReadOnlyEngineTests.java

server/src/test/java/org/opensearch/index/store/StoreTests.java

github-actions · 2022-12-08T23:07:06Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/7812/
CommitID: cce19a3

kartg · 2022-12-09T00:16:06Z

Please add a comment - maybe where you use the testIndex file or a readme file alongside it - to show how you generated that file. A set of commands and specific versions would be great.

Good callout, @gregschohn. I've added a README file in the latest commit 😄

github-actions · 2022-12-09T19:37:03Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/7850/
CommitID: 7531e14

server/src/main/java/org/opensearch/common/lucene/Lucene.java

server/src/main/java/org/opensearch/index/shard/IndexShard.java

server/src/main/java/org/opensearch/index/store/Store.java

…shots This change removes the guardrails around N-1 backward compatibility and uses Lucene's "expert" APIs to read snapshots (Lucene segments) older than N-1 on a best-effort basis. The functionality is gated by an additional feature flag, separate from the searchable snapshots flag. Note that the Lucene integration is rather inefficient because the necessary "expert" Lucene APIs are still package-private. Signed-off-by: Kartik Ganesh <gkart@amazon.com>

This change also includes a test index ZIP file for the unit tests. The change also introduces a bug fix in the readAnySegmentsInfo method to close the reader before returning the SegmentInfos object - this avoids dangling/open file handles. Signed-off-by: Kartik Ganesh <gkart@amazon.com>

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

…s compatibility Signed-off-by: Kartik Ganesh <gkart@amazon.com>

This change also includes an update to the file name to clarify that it is an ES index, and changing the associated markdown file name to just README.md. All tests that reference this ZIP file have corresponding changes to the path they reference. Signed-off-by: Kartik Ganesh <gkart@amazon.com>

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

Also refactored unit test classes to use the helper class. Signed-off-by: Kartik Ganesh <gkart@amazon.com>

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

Updated the asserts in IndexSettingsTests to account for the new defaulting behavior. Signed-off-by: Kartik Ganesh <gkart@amazon.com>

github-actions · 2023-01-04T20:17:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

URL: https://build.ci.opensearch.org/job/gradle-check/8815/
CommitID: 99b81ae
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

kartg · 2023-01-04T20:21:51Z

Looks like I clicked approve a bit too soon...there are some test failures :)

@andrross oops, my mistake - i changed the defaulting behavior in IndexSettings but forgot to update the related test in IndexSettingsTests. Fixed now, and rebased to resolve the CHANGELOG merge conflict.

Also the unstable result above seems to be coming from a flaky test failure - #3603

github-actions · 2023-01-04T23:12:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/8832/
CommitID: 99b81ae

@mch2

…shots (opensearch-project#5429) * Enhance searchable snapshots to enable a read-only view of older snapshots This change removes the guardrails around N-1 backward compatibility and uses Lucene's "expert" APIs to read snapshots (Lucene segments) older than N-1 on a best-effort basis. The functionality is gated by an additional feature flag, separate from the searchable snapshots flag. Note that the Lucene integration is rather inefficient because the necessary "expert" Lucene APIs are still package-private. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added some unit tests This change also includes a test index ZIP file for the unit tests. The change also introduces a bug fix in the readAnySegmentsInfo method to close the reader before returning the SegmentInfos object - this avoids dangling/open file handles. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporate PR comments from andrross Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Remove use of IndexSetting for minimum version for snapshots backwards compatibility Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Moved ES 6.3.0 test data to a subdirectory This change also includes an update to the file name to clarify that it is an ES index, and changing the associated markdown file name to just README.md. All tests that reference this ZIP file have corresponding changes to the path they reference. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Update unit tests to use try-with-resources Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added FeatureFlagSetter helper class Also refactored unit test classes to use the helper class. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback from @mch2 Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Fix IndexSettingsTests Updated the asserts in IndexSettingsTests to account for the new defaulting behavior. Signed-off-by: Kartik Ganesh <gkart@amazon.com> Signed-off-by: Kartik Ganesh <gkart@amazon.com>

@mch2

…shots (opensearch-project#5429) * Enhance searchable snapshots to enable a read-only view of older snapshots This change removes the guardrails around N-1 backward compatibility and uses Lucene's "expert" APIs to read snapshots (Lucene segments) older than N-1 on a best-effort basis. The functionality is gated by an additional feature flag, separate from the searchable snapshots flag. Note that the Lucene integration is rather inefficient because the necessary "expert" Lucene APIs are still package-private. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added some unit tests This change also includes a test index ZIP file for the unit tests. The change also introduces a bug fix in the readAnySegmentsInfo method to close the reader before returning the SegmentInfos object - this avoids dangling/open file handles. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporate PR comments from andrross Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Remove use of IndexSetting for minimum version for snapshots backwards compatibility Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Moved ES 6.3.0 test data to a subdirectory This change also includes an update to the file name to clarify that it is an ES index, and changing the associated markdown file name to just README.md. All tests that reference this ZIP file have corresponding changes to the path they reference. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Update unit tests to use try-with-resources Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added FeatureFlagSetter helper class Also refactored unit test classes to use the helper class. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback from @mch2 Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Fix IndexSettingsTests Updated the asserts in IndexSettingsTests to account for the new defaulting behavior. Signed-off-by: Kartik Ganesh <gkart@amazon.com> Signed-off-by: Kartik Ganesh <gkart@amazon.com>

@mch2

…shots (opensearch-project#5429) * Enhance searchable snapshots to enable a read-only view of older snapshots This change removes the guardrails around N-1 backward compatibility and uses Lucene's "expert" APIs to read snapshots (Lucene segments) older than N-1 on a best-effort basis. The functionality is gated by an additional feature flag, separate from the searchable snapshots flag. Note that the Lucene integration is rather inefficient because the necessary "expert" Lucene APIs are still package-private. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added some unit tests This change also includes a test index ZIP file for the unit tests. The change also introduces a bug fix in the readAnySegmentsInfo method to close the reader before returning the SegmentInfos object - this avoids dangling/open file handles. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporate PR comments from andrross Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Remove use of IndexSetting for minimum version for snapshots backwards compatibility Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Moved ES 6.3.0 test data to a subdirectory This change also includes an update to the file name to clarify that it is an ES index, and changing the associated markdown file name to just README.md. All tests that reference this ZIP file have corresponding changes to the path they reference. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Update unit tests to use try-with-resources Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added FeatureFlagSetter helper class Also refactored unit test classes to use the helper class. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback from @mch2 Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Fix IndexSettingsTests Updated the asserts in IndexSettingsTests to account for the new defaulting behavior. Signed-off-by: Kartik Ganesh <gkart@amazon.com> Signed-off-by: Kartik Ganesh <gkart@amazon.com>

@mch2

…shots (opensearch-project#5429) * Enhance searchable snapshots to enable a read-only view of older snapshots This change removes the guardrails around N-1 backward compatibility and uses Lucene's "expert" APIs to read snapshots (Lucene segments) older than N-1 on a best-effort basis. The functionality is gated by an additional feature flag, separate from the searchable snapshots flag. Note that the Lucene integration is rather inefficient because the necessary "expert" Lucene APIs are still package-private. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added some unit tests This change also includes a test index ZIP file for the unit tests. The change also introduces a bug fix in the readAnySegmentsInfo method to close the reader before returning the SegmentInfos object - this avoids dangling/open file handles. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporate PR comments from andrross Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Remove use of IndexSetting for minimum version for snapshots backwards compatibility Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Moved ES 6.3.0 test data to a subdirectory This change also includes an update to the file name to clarify that it is an ES index, and changing the associated markdown file name to just README.md. All tests that reference this ZIP file have corresponding changes to the path they reference. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Update unit tests to use try-with-resources Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added FeatureFlagSetter helper class Also refactored unit test classes to use the helper class. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback from @mch2 Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Fix IndexSettingsTests Updated the asserts in IndexSettingsTests to account for the new defaulting behavior. Signed-off-by: Kartik Ganesh <gkart@amazon.com> Signed-off-by: Kartik Ganesh <gkart@amazon.com>

@mch2

…shots (opensearch-project#5429) * Enhance searchable snapshots to enable a read-only view of older snapshots This change removes the guardrails around N-1 backward compatibility and uses Lucene's "expert" APIs to read snapshots (Lucene segments) older than N-1 on a best-effort basis. The functionality is gated by an additional feature flag, separate from the searchable snapshots flag. Note that the Lucene integration is rather inefficient because the necessary "expert" Lucene APIs are still package-private. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added some unit tests This change also includes a test index ZIP file for the unit tests. The change also introduces a bug fix in the readAnySegmentsInfo method to close the reader before returning the SegmentInfos object - this avoids dangling/open file handles. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporate PR comments from andrross Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Remove use of IndexSetting for minimum version for snapshots backwards compatibility Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Moved ES 6.3.0 test data to a subdirectory This change also includes an update to the file name to clarify that it is an ES index, and changing the associated markdown file name to just README.md. All tests that reference this ZIP file have corresponding changes to the path they reference. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Update unit tests to use try-with-resources Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added FeatureFlagSetter helper class Also refactored unit test classes to use the helper class. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback from @mch2 Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Fix IndexSettingsTests Updated the asserts in IndexSettingsTests to account for the new defaulting behavior. Signed-off-by: Kartik Ganesh <gkart@amazon.com> Signed-off-by: Kartik Ganesh <gkart@amazon.com>

@mch2

…shots (opensearch-project#5429) * Enhance searchable snapshots to enable a read-only view of older snapshots This change removes the guardrails around N-1 backward compatibility and uses Lucene's "expert" APIs to read snapshots (Lucene segments) older than N-1 on a best-effort basis. The functionality is gated by an additional feature flag, separate from the searchable snapshots flag. Note that the Lucene integration is rather inefficient because the necessary "expert" Lucene APIs are still package-private. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added some unit tests This change also includes a test index ZIP file for the unit tests. The change also introduces a bug fix in the readAnySegmentsInfo method to close the reader before returning the SegmentInfos object - this avoids dangling/open file handles. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporate PR comments from andrross Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Remove use of IndexSetting for minimum version for snapshots backwards compatibility Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Moved ES 6.3.0 test data to a subdirectory This change also includes an update to the file name to clarify that it is an ES index, and changing the associated markdown file name to just README.md. All tests that reference this ZIP file have corresponding changes to the path they reference. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Update unit tests to use try-with-resources Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added FeatureFlagSetter helper class Also refactored unit test classes to use the helper class. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback from @mch2 Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Fix IndexSettingsTests Updated the asserts in IndexSettingsTests to account for the new defaulting behavior. Signed-off-by: Kartik Ganesh <gkart@amazon.com> Signed-off-by: Kartik Ganesh <gkart@amazon.com>

@mch2

…shots (opensearch-project#5429) * Enhance searchable snapshots to enable a read-only view of older snapshots This change removes the guardrails around N-1 backward compatibility and uses Lucene's "expert" APIs to read snapshots (Lucene segments) older than N-1 on a best-effort basis. The functionality is gated by an additional feature flag, separate from the searchable snapshots flag. Note that the Lucene integration is rather inefficient because the necessary "expert" Lucene APIs are still package-private. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added some unit tests This change also includes a test index ZIP file for the unit tests. The change also introduces a bug fix in the readAnySegmentsInfo method to close the reader before returning the SegmentInfos object - this avoids dangling/open file handles. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporate PR comments from andrross Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Remove use of IndexSetting for minimum version for snapshots backwards compatibility Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Moved ES 6.3.0 test data to a subdirectory This change also includes an update to the file name to clarify that it is an ES index, and changing the associated markdown file name to just README.md. All tests that reference this ZIP file have corresponding changes to the path they reference. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Update unit tests to use try-with-resources Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added FeatureFlagSetter helper class Also refactored unit test classes to use the helper class. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback from @mch2 Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Fix IndexSettingsTests Updated the asserts in IndexSettingsTests to account for the new defaulting behavior. Signed-off-by: Kartik Ganesh <gkart@amazon.com> Signed-off-by: Kartik Ganesh <gkart@amazon.com>

@mch2

…shots (#5812) * Enhance searchable snapshots to enable a read-only view of older snapshots (#5429) * Enhance searchable snapshots to enable a read-only view of older snapshots This change removes the guardrails around N-1 backward compatibility and uses Lucene's "expert" APIs to read snapshots (Lucene segments) older than N-1 on a best-effort basis. The functionality is gated by an additional feature flag, separate from the searchable snapshots flag. Note that the Lucene integration is rather inefficient because the necessary "expert" Lucene APIs are still package-private. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added some unit tests This change also includes a test index ZIP file for the unit tests. The change also introduces a bug fix in the readAnySegmentsInfo method to close the reader before returning the SegmentInfos object - this avoids dangling/open file handles. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporate PR comments from andrross Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Remove use of IndexSetting for minimum version for snapshots backwards compatibility Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Moved ES 6.3.0 test data to a subdirectory This change also includes an update to the file name to clarify that it is an ES index, and changing the associated markdown file name to just README.md. All tests that reference this ZIP file have corresponding changes to the path they reference. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Update unit tests to use try-with-resources Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added FeatureFlagSetter helper class Also refactored unit test classes to use the helper class. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback from @mch2 Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Fix IndexSettingsTests Updated the asserts in IndexSettingsTests to account for the new defaulting behavior. Signed-off-by: Kartik Ganesh <gkart@amazon.com> Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Fixed compile issues after cherry-pick Note that the unit tests are still failing at this commit since the Lucene 9 libraries no longer hold constants for Lucene 7 and below, so the fromId logic resolves the Lucene version to 8. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Fix multiple aspects of version resolution This change fixes resolution of the Lucene version for legacy versions since the Lucene 9 libraries no longer hold constants for Lucene 7 and below. The change also updates DECLARED_VERSIONS to derive from the Versions class rather than LegacyESVersions (thereby ignoring legacy versions). This in turn required a change to the minimumIndexCompatibleVersion logic for LegacyESVersion. Finally, the testMinimumIndexCompatibilityVersion unit test was updated to use accurate version identifiers. All unit tests pass and the code compiles, but actual functionality is still broken because some backwards compatibility logic was removed in the current branch that is retained in 2.x Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Reintroducing backwards compatibility logic in certain classes This reverts changes made in #4728 and #4702. These were only made in main and not backported to 2.x This change also adds unit tests for IndexMetadataGenerations Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Declared LegacyESVersion constants for better readability This commit also includes a correction to documentation, and removes the unnecessary "afterWriteSnapBlob" runnable from BlobStoreRepository Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Fixing PR reference in CHANGELOG Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Revert CHANGELOG update This feature was released in 2.5.0 so it no longer needs to be listed in the changelog. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Remove outdated comment Signed-off-by: Kartik Ganesh <gkart@amazon.com> Signed-off-by: Kartik Ganesh <gkart@amazon.com>

@mch2

…shots (#5429) * Enhance searchable snapshots to enable a read-only view of older snapshots This change removes the guardrails around N-1 backward compatibility and uses Lucene's "expert" APIs to read snapshots (Lucene segments) older than N-1 on a best-effort basis. The functionality is gated by an additional feature flag, separate from the searchable snapshots flag. Note that the Lucene integration is rather inefficient because the necessary "expert" Lucene APIs are still package-private. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added some unit tests This change also includes a test index ZIP file for the unit tests. The change also introduces a bug fix in the readAnySegmentsInfo method to close the reader before returning the SegmentInfos object - this avoids dangling/open file handles. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporate PR comments from andrross Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Remove use of IndexSetting for minimum version for snapshots backwards compatibility Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Moved ES 6.3.0 test data to a subdirectory This change also includes an update to the file name to clarify that it is an ES index, and changing the associated markdown file name to just README.md. All tests that reference this ZIP file have corresponding changes to the path they reference. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Update unit tests to use try-with-resources Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added FeatureFlagSetter helper class Also refactored unit test classes to use the helper class. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback from @mch2 Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Fix IndexSettingsTests Updated the asserts in IndexSettingsTests to account for the new defaulting behavior. Signed-off-by: Kartik Ganesh <gkart@amazon.com> Signed-off-by: Kartik Ganesh <gkart@amazon.com>

kartg changed the title ~~Enhance searchable snapshots to enable a read-only view of older snapshots~~ [DRAFT] Enhance searchable snapshots to enable a read-only view of older snapshots Dec 1, 2022

mch2 reviewed Dec 1, 2022

View reviewed changes

andrross reviewed Dec 6, 2022

View reviewed changes

server/src/main/java/org/opensearch/cluster/metadata/IndexMetadata.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/common/lucene/Lucene.java Outdated Show resolved Hide resolved

kartg changed the title ~~[DRAFT] Enhance searchable snapshots to enable a read-only view of older snapshots~~ Enhance searchable snapshots to enable a read-only view of older snapshots Dec 6, 2022

kartg marked this pull request as ready for review December 6, 2022 22:29

kartg requested review from a team and reta as code owners December 6, 2022 22:29

gregschohn reviewed Dec 7, 2022

View reviewed changes

kartg force-pushed the 2x-backwardCompat branch from cce19a3 to 7531e14 Compare December 9, 2022 19:00

gregschohn reviewed Dec 13, 2022

View reviewed changes

server/src/main/java/org/opensearch/common/lucene/Lucene.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/index/shard/IndexShard.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/index/store/Store.java Outdated Show resolved Hide resolved

kartg force-pushed the 2x-backwardCompat branch from 7531e14 to 58a23b2 Compare December 15, 2022 20:10

kartg requested review from anasalkouz, Bukhtawar, CEHENKLE, dblock, setiah, kotwanikunal, nknize, owaiskazi19, adnapibar and Rishikesh1159 as code owners December 15, 2022 20:10

kartg added 10 commits January 4, 2023 10:58

Incorporating PR feedback

d85174f

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

Incorporate PR comments from andrross

e5d8880

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

Remove use of IndexSetting for minimum version for snapshots backward…

a0eea5d

…s compatibility Signed-off-by: Kartik Ganesh <gkart@amazon.com>

Update unit tests to use try-with-resources

73a1429

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

Added FeatureFlagSetter helper class

2006683

Also refactored unit test classes to use the helper class. Signed-off-by: Kartik Ganesh <gkart@amazon.com>

Incorporating PR feedback from @mch2

4339244

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

Fix IndexSettingsTests

99b81ae

Updated the asserts in IndexSettingsTests to account for the new defaulting behavior. Signed-off-by: Kartik Ganesh <gkart@amazon.com>

kartg force-pushed the 2x-backwardCompat branch from 74c25bd to 99b81ae Compare January 4, 2023 19:42

kartg mentioned this pull request Jan 4, 2023

[Proposal] Enable searchable snapshots to read legacy versions #5451

Closed

andrross approved these changes Jan 4, 2023

View reviewed changes

mch2 approved these changes Jan 5, 2023

View reviewed changes

kartg merged commit 85b85b1 into opensearch-project:2.x Jan 5, 2023

kartg deleted the 2x-backwardCompat branch January 10, 2023 17:58

kartg mentioned this pull request Jan 11, 2023

Enhance searchable snapshots to enable a read-only view of older snapshots #5812

Merged

6 tasks

kartg mentioned this pull request Jan 20, 2023

[Meta] GA for searchable snapshots reading legacy versions #5948

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhance searchable snapshots to enable a read-only view of older snapshots #5429

Enhance searchable snapshots to enable a read-only view of older snapshots #5429

kartg commented Dec 1, 2022 •

edited

Loading

github-actions bot commented Dec 1, 2022

codecov-commenter commented Dec 1, 2022 •

edited

Loading

github-actions bot commented Dec 6, 2022

github-actions bot commented Dec 6, 2022

github-actions bot commented Dec 6, 2022

github-actions bot commented Dec 7, 2022

gregschohn left a comment

github-actions bot commented Dec 8, 2022

kartg commented Dec 9, 2022

github-actions bot commented Dec 9, 2022

github-actions bot commented Jan 4, 2023

kartg commented Jan 4, 2023

github-actions bot commented Jan 4, 2023

Enhance searchable snapshots to enable a read-only view of older snapshots #5429

Enhance searchable snapshots to enable a read-only view of older snapshots #5429

Conversation

kartg commented Dec 1, 2022 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Dec 1, 2022

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Dec 1, 2022 • edited Loading

Codecov Report

github-actions bot commented Dec 6, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Dec 6, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Dec 6, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Dec 7, 2022

Gradle Check (Jenkins) Run Completed with:

gregschohn left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 8, 2022

Gradle Check (Jenkins) Run Completed with:

kartg commented Dec 9, 2022

github-actions bot commented Dec 9, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 4, 2023

Gradle Check (Jenkins) Run Completed with:

kartg commented Jan 4, 2023

github-actions bot commented Jan 4, 2023

Gradle Check (Jenkins) Run Completed with:

kartg commented Dec 1, 2022 •

edited

Loading

codecov-commenter commented Dec 1, 2022 •

edited

Loading