Add support for custom remote store segment path prefix to support clusterless configurations

[rgsriram]

Description

This commit adds support for injecting a unique writer node identifier into remote store paths to support clusterless configurations where multiple writers may write to the same shard.

Problem:
In clusterless configurations, multiple writers can write to the same shard simultaneously, which would cause conflicts with the current remote store path structure that doesn't include writer identification.

Solution:

• Add a new index setting index.remote_store.segment.path_prefix that allows injecting a writer node ID into the remote store path
• The setting is index-scoped, dynamic, and backwards compatible
• Only explicitly set, non-empty values are used as path components
• Empty, null, or whitespace-only values are ignored (no path component added)

Implementation Details:

• Add INDEX_REMOTE_STORE_SEGMENT_PATH_PREFIX setting in IndexMetadata with proper validation that ensures:
  • Remote store must be enabled when the setting is used
  • Path prefix cannot contain invalid characters (/, , :)
• Update RemoteStorePathStrategy to include writer node ID in path generation between shard ID and data category
• Modify IndexSettings to handle the new setting with proper null/empty checks
• Update RemoteSegmentStoreDirectoryFactory and RemoteStoreLockManagerFactory to pass writer node ID through the path generation chain
• Add comprehensive tests for path generation and validation

Path Structure:
Before: remote_repository/hash/index-uuid/shard-id/segments/data/file
After: remote_repository/hash/index-uuid/shard-id/writer-node-id/segments/data/file

Usage:

{
  "index.remote_store.segment.path_prefix": "writer-node-1"
}

Validation:

• Setting only works when index.remote_store.enabled is true
• Path prefix cannot contain path separators or drive specifiers
• Empty/whitespace values are allowed and ignored for backward compatibility

Related Issues

Resolves #18750

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

✅ Gradle check result for 40faadd: SUCCESS

[codecov]

Codecov Report

❌ Patch coverage is 78.78788% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.79%. Comparing base (f0e5003) to head (616ff4c).
⚠️ Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
...org/opensearch/cluster/metadata/IndexMetadata.java	86.66%	0 Missing and 2 partials ⚠️
...c/main/java/org/opensearch/index/IndexService.java	0.00%	2 Missing ⚠️
.../main/java/org/opensearch/index/IndexSettings.java	33.33%	1 Missing and 1 partial ⚠️
...ore/lockmanager/RemoteStoreLockManagerFactory.java	66.66%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #18857      +/-   ##
============================================
- Coverage     72.88%   72.79%   -0.10%     
+ Complexity    69160    69110      -50     
============================================
  Files          5633     5633              
  Lines        317978   318008      +30     
  Branches      45988    45993       +5     
============================================
- Hits         231752   231487     -265     
- Misses        67517    67744     +227     
- Partials      18709    18777      +68     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[rgsriram]

@msfroh - Could you please review?

[github-actions]

❌ Gradle check result for 706e564: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 08d63f2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 9102895: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 6aaa438: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 3095f53: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

✅ Gradle check result for a3d3eef: SUCCESS

[github-actions]

✅ Gradle check result for 07bf191: SUCCESS

[msfroh]

Thanks, @rgsriram! This looks really good. I just had two easy-to-fix comments:

1. Use PrivateIndex instead of InternalIndex for the new setting, for consistency with other remote store settings.
2. Rename the parameter in ShardDataPathInput to more generically represent the fact that it's an index-level fixed prefix, without necessarily assuming that it comes from the writer node ID.

Thanks!

[github-actions]

✅ Gradle check result for cdd3a31: SUCCESS

[github-actions]

✅ Gradle check result for 616ff4c: SUCCESS