Enhance searchable snapshots to enable a read-only view of older snap…

…shots (opensearch-project#5429) * Enhance searchable snapshots to enable a read-only view of older snapshots This change removes the guardrails around N-1 backward compatibility and uses Lucene's "expert" APIs to read snapshots (Lucene segments) older than N-1 on a best-effort basis. The functionality is gated by an additional feature flag, separate from the searchable snapshots flag. Note that the Lucene integration is rather inefficient because the necessary "expert" Lucene APIs are still package-private. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added some unit tests This change also includes a test index ZIP file for the unit tests. The change also introduces a bug fix in the readAnySegmentsInfo method to close the reader before returning the SegmentInfos object - this avoids dangling/open file handles. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporate PR comments from andrross Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Remove use of IndexSetting for minimum version for snapshots backwards compatibility Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Moved ES 6.3.0 test data to a subdirectory This change also includes an update to the file name to clarify that it is an ES index, and changing the associated markdown file name to just README.md. All tests that reference this ZIP file have corresponding changes to the path they reference. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Update unit tests to use try-with-resources Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added FeatureFlagSetter helper class Also refactored unit test classes to use the helper class. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback from @mch2 Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Fix IndexSettingsTests Updated the asserts in IndexSettingsTests to account for the new defaulting behavior. Signed-off-by: Kartik Ganesh <gkart@amazon.com> Signed-off-by: Kartik Ganesh <gkart@amazon.com>
kartg · Jan 9, 2023 · fe8844b · fe8844b
1 parent 094c1d8
commit fe8844b
Show file tree

Hide file tree

Showing 2 changed files with 3 additions and 1 deletion.
diff --git a/server/src/main/java/org/opensearch/common/lucene/Lucene.java b/server/src/main/java/org/opensearch/common/lucene/Lucene.java
@@ -57,6 +57,7 @@
 import org.apache.lucene.index.SegmentReader;
 import org.apache.lucene.index.StandardDirectoryReader;
 import org.apache.lucene.index.StandardDirectoryReader;
+import org.apache.lucene.index.StandardDirectoryReader;
 import org.apache.lucene.search.DocIdSetIterator;
 import org.apache.lucene.search.Explanation;
 import org.apache.lucene.search.FieldDoc;

diff --git a/server/src/main/java/org/opensearch/index/IndexSettings.java b/server/src/main/java/org/opensearch/index/IndexSettings.java
@@ -589,7 +589,7 @@ public final class IndexSettings {
  private final String remoteStoreTranslogRepository;
  private final String remoteStoreRepository;
  private final boolean isRemoteSnapshot;
- private Version extendedCompatibilitySnapshotVersion;
+ private final Version extendedCompatibilitySnapshotVersion;
 
  // volatile fields are updated via #updateIndexMetadata(IndexMetadata) under lock
  private volatile Settings settings;
@@ -755,6 +755,7 @@ public IndexSettings(final IndexMetadata indexMetadata, final Settings nodeSetti
  remoteStoreTranslogRepository = settings.get(IndexMetadata.SETTING_REMOTE_TRANSLOG_STORE_REPOSITORY);
  remoteStoreRepository = settings.get(IndexMetadata.SETTING_REMOTE_STORE_REPOSITORY);
  isRemoteSnapshot = IndexModule.Type.REMOTE_SNAPSHOT.match(this.settings);
+
  if (isRemoteSnapshot && FeatureFlags.isEnabled(SEARCHABLE_SNAPSHOT_EXTENDED_COMPATIBILITY)) {
  extendedCompatibilitySnapshotVersion = SEARCHABLE_SNAPSHOT_EXTENDED_COMPATIBILITY_MINIMUM_VERSION;
  } else {