Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix compiler warning and base blob path case for searchable snapshots #6558

Merged
merged 1 commit into from
Mar 8, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Add GeoTile and GeoHash Grid aggregations on GeoShapes. ([#5589](https://github.com/opensearch-project/OpenSearch/pull/5589))
- Disallow multiple data paths for search nodes ([#6427](https://github.com/opensearch-project/OpenSearch/pull/6427))
- [Segment Replication] Allocation and rebalancing based on average primary shard count per index ([#6422](https://github.com/opensearch-project/OpenSearch/pull/6422))
- Add 'base_path' setting to File System Repository ([#6558](https://github.com/opensearch-project/OpenSearch/pull/6558))

### Dependencies
- Bump `org.apache.logging.log4j:log4j-core` from 2.18.0 to 2.20.0 ([#6490](https://github.com/opensearch-project/OpenSearch/pull/6490))
Expand All @@ -94,8 +95,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),

### Fixed
- Added equals/hashcode for named DocValueFormat.DateTime inner class ([#6357](https://github.com/opensearch-project/OpenSearch/pull/6357))
- Fixed bug for searchable snapshot to take 'base_path' of blob into account ([#6558](https://github.com/opensearch-project/OpenSearch/pull/6558))

### Security

[Unreleased 3.0]: https://github.com/opensearch-project/OpenSearch/compare/2.x...HEAD
[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.5...2.x
[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.5...2.x
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ public void testAllocationWithDisruption() throws Exception {

/**
* Utility method which ensures cluster has balanced primary shard distribution across a single index.
* @throws Exception
* @throws Exception exception
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just an FYI, if you've been getting javadoc warnings/failures in the tests, it might be due to building with JDK 14. The javadoc rules are supposed to be ignored for tests, but JDK-14 doesn't seem to do the right thing, though it has been end-of-life for quite awhile now.

*/
private void verifyPerIndexPrimaryBalance() throws Exception {
assertBusy(() -> {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ protected Settings featureFlagSettings() {
protected Settings.Builder randomRepositorySettings() {
final Settings.Builder settings = Settings.builder();
settings.put("location", randomRepoPath()).put("compress", randomBoolean());
settings.put(FsRepository.BASE_PATH_SETTING.getKey(), "my_base_path");
return settings;
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,8 @@ private Future<RemoteSnapshotDirectory> createRemoteSnapshotDirectoryFromSnapsho
ShardPath localShardPath,
BlobStoreRepository blobStoreRepository
) throws IOException {
final BlobPath blobPath = new BlobPath().add("indices")
final BlobPath blobPath = blobStoreRepository.basePath()
.add("indices")
.add(IndexSettings.SEARCHABLE_SNAPSHOT_INDEX_ID.get(indexSettings.getSettings()))
.add(Integer.toString(localShardPath.getShardId().getId()));
final SnapshotId snapshotId = new SnapshotId(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
import org.apache.logging.log4j.Logger;
import org.opensearch.cluster.metadata.RepositoryMetadata;
import org.opensearch.cluster.service.ClusterService;
import org.opensearch.common.Strings;
import org.opensearch.common.blobstore.BlobPath;
import org.opensearch.common.blobstore.BlobStore;
import org.opensearch.common.blobstore.fs.FsBlobStore;
Expand Down Expand Up @@ -98,6 +99,9 @@ public class FsRepository extends BlobStoreRepository {
Property.NodeScope,
Property.Deprecated
);

public static final Setting<String> BASE_PATH_SETTING = Setting.simpleString("base_path");

private final Environment environment;

private ByteSizeValue chunkSize;
Expand Down Expand Up @@ -154,7 +158,12 @@ public FsRepository(
} else {
this.chunkSize = REPOSITORIES_CHUNK_SIZE_SETTING.get(environment.settings());
}
this.basePath = BlobPath.cleanPath();
final String basePath = BASE_PATH_SETTING.get(metadata.settings());
if (Strings.hasLength(basePath)) {
this.basePath = new BlobPath().add(basePath);
} else {
this.basePath = BlobPath.cleanPath();
}
}

private static boolean calculateCompress(RepositoryMetadata metadata, Environment environment) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ private Map<String, String> getDummyMetadata(String prefix, int commitGeneration
* Prepares metadata file bytes with header and footer
* @param segmentFilesMap: actual metadata content
* @return ByteArrayIndexInput: metadata file bytes with header and footer
* @throws IOException
* @throws IOException IOException
*/
private ByteArrayIndexInput createMetadataFileBytes(Map<String, String> segmentFilesMap) throws IOException {
BytesStreamOutput output = new BytesStreamOutput();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@ public void testSnapshotAndRestore() throws IOException, InterruptedException {
.put("location", repo)
.put("compress", randomBoolean())
.put("chunk_size", randomIntBetween(100, 1000), ByteSizeUnit.BYTES)
.put(FsRepository.BASE_PATH_SETTING.getKey(), "my_base_path")
.build();

int numDocs = indexDocs(directory);
Expand Down