Fix file cache initialization #14004

bugmakerrrrrr · 2024-06-05T14:15:49Z

Description

When a search node starts, a file cache is initialized. The size of the file cache is determined by available space of the local storage and the user's config (Math.min(capacity, availableCapacity)). However, this approach can cause a issues. For example, if we have a search node with 100GB local storage and config the file cache to be 80GB, after some time, 90% of the file cache is used. if we then restart the node, the available space may be reduced to just 28GB, which means the file cache will also be reduced to 28GB. This can cause further issues as the fs stats will be adjusted based on the file cache size, leading to wired results.

To address this issue, we can use the total space to calculate the size of file cache. User can config the size to be absolute bytes value or a percentage of total space.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
API changes companion pull request created.
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

bugmakerrrrrr · 2024-06-05T14:19:55Z

@andrross @kotwanikunal could you please take a look when you get a chance

server/src/main/java/org/opensearch/node/Node.java

github-actions · 2024-06-05T15:07:59Z

❌ Gradle check result for 328da32: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-06-05T15:14:07Z

✅ Gradle check result for afe886f: SUCCESS

Signed-off-by: panguixin <panguixin@bytedance.com>

codecov · 2024-06-25T17:04:39Z

Codecov Report

Attention: Patch coverage is 76.31579% with 9 lines in your changes missing coverage. Please review.

Project coverage is 71.75%. Comparing base (b15cb0c) to head (bd73b60).
Report is 484 commits behind head on main.

Files	Patch %	Lines
server/src/main/java/org/opensearch/node/Node.java	76.66%	4 Missing and 3 partials ⚠️
...c/main/java/org/opensearch/monitor/fs/FsProbe.java	60.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #14004      +/-   ##
============================================
+ Coverage     71.42%   71.75%   +0.33%     
- Complexity    59978    62165    +2187     
============================================
  Files          4985     5125     +140     
  Lines        282275   292385   +10110     
  Branches      40946    42249    +1303     
============================================
+ Hits         201603   209795    +8192     
- Misses        63999    65306    +1307     
- Partials      16673    17284     +611

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

bugmakerrrrrr · 2024-06-25T17:07:07Z

@andrross friendly ping :)

github-actions · 2024-06-25T17:51:48Z

❕ Gradle check result for bd73b60: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

* fix file cache initialization Signed-off-by: panguixin <panguixin@bytedance.com> * changelog Signed-off-by: panguixin <panguixin@bytedance.com> * add test Signed-off-by: panguixin <panguixin@bytedance.com> --------- Signed-off-by: panguixin <panguixin@bytedance.com> (cherry picked from commit 2be25bb) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* fix file cache initialization Signed-off-by: panguixin <panguixin@bytedance.com> * changelog Signed-off-by: panguixin <panguixin@bytedance.com> * add test Signed-off-by: panguixin <panguixin@bytedance.com> --------- Signed-off-by: panguixin <panguixin@bytedance.com>

* fix file cache initialization * changelog * add test --------- (cherry picked from commit 2be25bb) Signed-off-by: panguixin <panguixin@bytedance.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

@internalapi

* Fix flaky test in range aggregation yaml test (opensearch-project#14486) Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Use CODECOV_TOKEN (opensearch-project#14536) Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> * [Tiered Caching] Moving query recomputation logic outside of write lock (opensearch-project#14187) * Moving query recompute out of write lock Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * [Tiered Caching] Moving query recomputation logic outside of write lock Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Adding java doc for the completable map Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Changes to call future handler only once per key Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing spotless check Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Added changelog Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Addressing comments Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing gradle fail Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Addressing comments to refactor unit test Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * minor UT refactor Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> --------- Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com> * Fix Flaky Test ClusterRerouteIT.testDelayWithALargeAmountOfShards (opensearch-project#14510) Signed-off-by: kkewwei kkewwei@163.com Signed-off-by: kkewwei kkewwei@163.com Signed-off-by: kkewwei <kkewwei@163.com> * Add doc for debugging rest tests (opensearch-project#14491) * add doc for debugging rest tests Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Update TESTING.md Co-authored-by: Marc Handalian <handalm@amazon.com> Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Address comment Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> --------- Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Co-authored-by: Marc Handalian <handalm@amazon.com> * Fix flaky DefaultCacheStatsHolderTests (opensearch-project#14462) Signed-off-by: Peter Alfonsi <petealft@amazon.com> Co-authored-by: Peter Alfonsi <petealft@amazon.com> * [AUTO] [main] Add bwc version 2.15.1. (opensearch-project#14549) * Add bwc version 2.15.1 Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fix auto-generated version Signed-off-by: Andrew Ross <andrross@amazon.com> --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <andrross@amazon.com> Co-authored-by: opensearch-ci-bot <83309141+opensearch-ci-bot@users.noreply.github.com> Co-authored-by: Andrew Ross <andrross@amazon.com> * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently (opensearch-project#14550) * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Addressing comment Signed-off-by: Sagar Upadhyaya <upasagar@amazon.com> --------- Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Sagar Upadhyaya <upasagar@amazon.com> Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com> * Add allowlist setting for ingest-common processors (opensearch-project#14479) Add a new static setting that lets an operator choose specific ingest processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <andrross@amazon.com> * Fix file cache initialization (opensearch-project#14004) * fix file cache initialization Signed-off-by: panguixin <panguixin@bytedance.com> * changelog Signed-off-by: panguixin <panguixin@bytedance.com> * add test Signed-off-by: panguixin <panguixin@bytedance.com> --------- Signed-off-by: panguixin <panguixin@bytedance.com> * Add Ashish Singh as maintainer (opensearch-project#14567) Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> * Allow @internalapi annotation on classes not meant to be constructed outside of the OpenSearch core (opensearch-project#14575) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Bump com.azure:azure-storage-common from 12.21.2 to 12.25.1 in /plugins/repository-azure (opensearch-project#14517) * Bump com.azure:azure-storage-common in /plugins/repository-azure Bumps [com.azure:azure-storage-common](https://github.com/Azure/azure-sdk-for-java) from 12.21.2 to 12.25.1. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-common_12.21.2...azure-storage-blob_12.25.1) --- updated-dependencies: - dependency-name: com.azure:azure-storage-common dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Add allowlist setting for search-pipeline-common processors (opensearch-project#14562) Add a new static setting that lets an operator choose specific search pipeline processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. A distinct setting exists for each of request, response, and search phase results processors. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <andrross@amazon.com> * Bump Apache Lucene to 9.11.1 (opensearch-project#14576) (opensearch-project#14581) (cherry picked from commit 0095fd1) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Add unittests for RemoteClusterStateAttributesManager (opensearch-project#14427) * Add unittests for RemoteClusterStateAttributesManager Signed-off-by: Shivansh Arora <hishiv@amazon.com> * Add Ashish Singh to codeowners (opensearch-project#14592) Signed-off-by: Ashish Singh <ssashish@amazon.com> * Add batching processor base type AbstractBatchingProcessor (opensearch-project#14554) Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Add @internalapi annotation to japicmp exclusions (opensearch-project#14597) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Fix issue 14519:Parsing a GetResult returns NPE if found field is mis… (opensearch-project#14552) * Fix issue 14519:Parsing a GetResult returns NPE if found field is missing. Signed-off-by: Vatsal <vatsal.v.anand@gmail.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> * Fix issue 14519:Parsing a GetResult returns NPE if found field is missing. Signed-off-by: Vatsal <vatsal.v.anand@gmail.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> * Fix issue 14519:Fix wildcart import. Signed-off-by: Vatsal <vatsal.v.anand@gmail.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> * Fix issue 14519:Fix wildcart import. Signed-off-by: Vatsal <vatsal.v.anand@gmail.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> * Fix issue 14519:Fix spotless issues. Signed-off-by: Vatsal <vatsal.v.anand@gmail.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> * Fix issue 14519:update changelog Signed-off-by: vatsal <vatsal.v.anand@gmail.com> --------- Signed-off-by: vatsal <vatsal.v.anand@gmail.com> Signed-off-by: Daniel Widdis <widdis@gmail.com> Co-authored-by: Daniel Widdis <widdis@gmail.com> * Star tree mapping changes (opensearch-project#14605) * Star tree mapping changes with feature flag --------- Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * Bump com.microsoft.azure:msal4j from 1.15.1 to 1.16.0 in /plugins/repository-azure (opensearch-project#14610) * Bump com.microsoft.azure:msal4j in /plugins/repository-azure Bumps [com.microsoft.azure:msal4j](https://github.com/AzureAD/microsoft-authentication-library-for-java) from 1.15.1 to 1.16.0. - [Release notes](https://github.com/AzureAD/microsoft-authentication-library-for-java/releases) - [Changelog](https://github.com/AzureAD/microsoft-authentication-library-for-java/blob/dev/changelog.txt) - [Commits](AzureAD/microsoft-authentication-library-for-java@v1.15.1...v1.16.0) --- updated-dependencies: - dependency-name: com.microsoft.azure:msal4j dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * [Bugfix] Fix ICacheKeySerializerTests flakiness (opensearch-project#14564) * Fix testInvalidInput flakiness Signed-off-by: Peter Alfonsi <petealft@amazon.com> * Addressed andrross's comment Signed-off-by: Peter Alfonsi <petealft@amazon.com> * rerun security check Signed-off-by: Peter Alfonsi <petealft@amazon.com> --------- Signed-off-by: Peter Alfonsi <petealft@amazon.com> Co-authored-by: Peter Alfonsi <petealft@amazon.com> * Correct typo in method name (opensearch-project#14621) Signed-off-by: vatsal <vatsal.v.anand@gmail.com> * Refactoring FilterPath.parse by using an iterative approach instead of recursion. (opensearch-project#14200) * Refactor FilterPath parse function (opensearch-project#12067) Signed-off-by: Robin Friedmann <robinfriedmann.rf@gmail.com> * Implement unit tests for FilterPathTests (opensearch-project#12067) Signed-off-by: Robin Friedmann <robinfriedmann.rf@gmail.com> * Write warn log if Filter is empty; Add comments (opensearch-project#12067) Signed-off-by: Robin Friedmann <robinfriedmann.rf@gmail.com> * Add changelog Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Remove unnecessary log statement Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Remove unused logger Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Spotless apply Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Remove incorrect changelog Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> --------- Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> Co-authored-by: Robin Friedmann <robinfriedmann.rf@gmail.com> * Removing String format in RemoteStoreMigrationAllocationDecider to optimise performance(opensearch-project#14612) Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com> * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata; Correct the check for deciding upload of HashesOfConsistentSettings (opensearch-project#14513) * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata * Correct the check for deciding upload of hashes of consistent settings Signed-off-by: Sooraj Sinha <soosinha@amazon.com> * Improve reroute performance by optimising List.removeAll in LocalShardsBalancer to filter remote search shard from relocation decision (opensearch-project#14613) Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com> * Fix assertion failure while deleting remote backed index (opensearch-project#14601) Signed-off-by: Sachin Kale <kalsac@amazon.com> * OnHeap Star Tree Implementation Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * addressed nits Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * addressed major nits Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * includes Count Aggregator Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * handling for missing doc values Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * addressing review comments Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * rebasing with main Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * support for empty sequential doc values iterator Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * nits Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * min and max star tree aggregators Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * star tree file formats * Star tree codec changes Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * Adding tests Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * Addressing comments Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * addressing review comments Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * Star tree merge changes Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * fix annotations * star-tree file formats reader and javadoc fixes * read for composite index values Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * doc values file format Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> --------- Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: kkewwei kkewwei@163.com Signed-off-by: kkewwei <kkewwei@163.com> Signed-off-by: Peter Alfonsi <petealft@amazon.com> Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Sagar Upadhyaya <upasagar@amazon.com> Signed-off-by: panguixin <panguixin@bytedance.com> Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Liyun Xiu <xiliyun@amazon.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> Signed-off-by: Daniel Widdis <widdis@gmail.com> Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com> Signed-off-by: Sooraj Sinha <soosinha@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> Signed-off-by: Bharathwaj G <bharath78910@gmail.com> Co-authored-by: bowenlan-amzn <bowenlan23@gmail.com> Co-authored-by: Prudhvi Godithi <pgodithi@amazon.com> Co-authored-by: Sagar <99425694+sgup432@users.noreply.github.com> Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com> Co-authored-by: kkewwei <kkewwei@163.com> Co-authored-by: Marc Handalian <handalm@amazon.com> Co-authored-by: Peter Alfonsi <peter.alfonsi@gmail.com> Co-authored-by: Peter Alfonsi <petealft@amazon.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <83309141+opensearch-ci-bot@users.noreply.github.com> Co-authored-by: Andrew Ross <andrross@amazon.com> Co-authored-by: panguixin <panguixin@bytedance.com> Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Shivansh Arora <hishiv@amazon.com> Co-authored-by: Ashish Singh <ssashish@amazon.com> Co-authored-by: Liyun Xiu <xiliyun@amazon.com> Co-authored-by: Vatsal <36672090+imvtsl@users.noreply.github.com> Co-authored-by: Daniel Widdis <widdis@gmail.com> Co-authored-by: Siddhant Deshmukh <deshsid@amazon.com> Co-authored-by: Robin Friedmann <robinfriedmann.rf@gmail.com> Co-authored-by: rishavz_sagar <rishavsagar4b1@gmail.com> Co-authored-by: Sooraj Sinha <81695996+soosinha@users.noreply.github.com> Co-authored-by: Sachin Kale <sachinpkale@gmail.com> Co-authored-by: Sarthak Aggarwal <sarthagg@amazon.com>

@internalapi

* Fix flaky test in range aggregation yaml test (opensearch-project#14486) Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Use CODECOV_TOKEN (opensearch-project#14536) Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> * [Tiered Caching] Moving query recomputation logic outside of write lock (opensearch-project#14187) * Moving query recompute out of write lock Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * [Tiered Caching] Moving query recomputation logic outside of write lock Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Adding java doc for the completable map Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Changes to call future handler only once per key Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing spotless check Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Added changelog Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Addressing comments Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing gradle fail Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Addressing comments to refactor unit test Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * minor UT refactor Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> --------- Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com> * Fix Flaky Test ClusterRerouteIT.testDelayWithALargeAmountOfShards (opensearch-project#14510) Signed-off-by: kkewwei kkewwei@163.com Signed-off-by: kkewwei kkewwei@163.com Signed-off-by: kkewwei <kkewwei@163.com> * Add doc for debugging rest tests (opensearch-project#14491) * add doc for debugging rest tests Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Update TESTING.md Co-authored-by: Marc Handalian <handalm@amazon.com> Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Address comment Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> --------- Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Co-authored-by: Marc Handalian <handalm@amazon.com> * Fix flaky DefaultCacheStatsHolderTests (opensearch-project#14462) Signed-off-by: Peter Alfonsi <petealft@amazon.com> Co-authored-by: Peter Alfonsi <petealft@amazon.com> * [AUTO] [main] Add bwc version 2.15.1. (opensearch-project#14549) * Add bwc version 2.15.1 Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fix auto-generated version Signed-off-by: Andrew Ross <andrross@amazon.com> --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <andrross@amazon.com> Co-authored-by: opensearch-ci-bot <83309141+opensearch-ci-bot@users.noreply.github.com> Co-authored-by: Andrew Ross <andrross@amazon.com> * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently (opensearch-project#14550) * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Addressing comment Signed-off-by: Sagar Upadhyaya <upasagar@amazon.com> --------- Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Sagar Upadhyaya <upasagar@amazon.com> Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com> * Add allowlist setting for ingest-common processors (opensearch-project#14479) Add a new static setting that lets an operator choose specific ingest processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <andrross@amazon.com> * Fix file cache initialization (opensearch-project#14004) * fix file cache initialization Signed-off-by: panguixin <panguixin@bytedance.com> * changelog Signed-off-by: panguixin <panguixin@bytedance.com> * add test Signed-off-by: panguixin <panguixin@bytedance.com> --------- Signed-off-by: panguixin <panguixin@bytedance.com> * Add Ashish Singh as maintainer (opensearch-project#14567) Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> * Allow @internalapi annotation on classes not meant to be constructed outside of the OpenSearch core (opensearch-project#14575) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Bump com.azure:azure-storage-common from 12.21.2 to 12.25.1 in /plugins/repository-azure (opensearch-project#14517) * Bump com.azure:azure-storage-common in /plugins/repository-azure Bumps [com.azure:azure-storage-common](https://github.com/Azure/azure-sdk-for-java) from 12.21.2 to 12.25.1. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-common_12.21.2...azure-storage-blob_12.25.1) --- updated-dependencies: - dependency-name: com.azure:azure-storage-common dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Add allowlist setting for search-pipeline-common processors (opensearch-project#14562) Add a new static setting that lets an operator choose specific search pipeline processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. A distinct setting exists for each of request, response, and search phase results processors. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <andrross@amazon.com> * Bump Apache Lucene to 9.11.1 (opensearch-project#14576) (opensearch-project#14581) (cherry picked from commit 0095fd1) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Add unittests for RemoteClusterStateAttributesManager (opensearch-project#14427) * Add unittests for RemoteClusterStateAttributesManager Signed-off-by: Shivansh Arora <hishiv@amazon.com> * Add Ashish Singh to codeowners (opensearch-project#14592) Signed-off-by: Ashish Singh <ssashish@amazon.com> * Add batching processor base type AbstractBatchingProcessor (opensearch-project#14554) Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Add @internalapi annotation to japicmp exclusions (opensearch-project#14597) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Fix issue 14519:Parsing a GetResult returns NPE if found field is mis… (opensearch-project#14552) * Fix issue 14519:Parsing a GetResult returns NPE if found field is missing. Signed-off-by: Vatsal <vatsal.v.anand@gmail.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> * Fix issue 14519:Parsing a GetResult returns NPE if found field is missing. Signed-off-by: Vatsal <vatsal.v.anand@gmail.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> * Fix issue 14519:Fix wildcart import. Signed-off-by: Vatsal <vatsal.v.anand@gmail.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> * Fix issue 14519:Fix wildcart import. Signed-off-by: Vatsal <vatsal.v.anand@gmail.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> * Fix issue 14519:Fix spotless issues. Signed-off-by: Vatsal <vatsal.v.anand@gmail.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> * Fix issue 14519:update changelog Signed-off-by: vatsal <vatsal.v.anand@gmail.com> --------- Signed-off-by: vatsal <vatsal.v.anand@gmail.com> Signed-off-by: Daniel Widdis <widdis@gmail.com> Co-authored-by: Daniel Widdis <widdis@gmail.com> * Star tree mapping changes (opensearch-project#14605) * Star tree mapping changes with feature flag --------- Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * Bump com.microsoft.azure:msal4j from 1.15.1 to 1.16.0 in /plugins/repository-azure (opensearch-project#14610) * Bump com.microsoft.azure:msal4j in /plugins/repository-azure Bumps [com.microsoft.azure:msal4j](https://github.com/AzureAD/microsoft-authentication-library-for-java) from 1.15.1 to 1.16.0. - [Release notes](https://github.com/AzureAD/microsoft-authentication-library-for-java/releases) - [Changelog](https://github.com/AzureAD/microsoft-authentication-library-for-java/blob/dev/changelog.txt) - [Commits](AzureAD/microsoft-authentication-library-for-java@v1.15.1...v1.16.0) --- updated-dependencies: - dependency-name: com.microsoft.azure:msal4j dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * [Bugfix] Fix ICacheKeySerializerTests flakiness (opensearch-project#14564) * Fix testInvalidInput flakiness Signed-off-by: Peter Alfonsi <petealft@amazon.com> * Addressed andrross's comment Signed-off-by: Peter Alfonsi <petealft@amazon.com> * rerun security check Signed-off-by: Peter Alfonsi <petealft@amazon.com> --------- Signed-off-by: Peter Alfonsi <petealft@amazon.com> Co-authored-by: Peter Alfonsi <petealft@amazon.com> * Correct typo in method name (opensearch-project#14621) Signed-off-by: vatsal <vatsal.v.anand@gmail.com> * Refactoring FilterPath.parse by using an iterative approach instead of recursion. (opensearch-project#14200) * Refactor FilterPath parse function (opensearch-project#12067) Signed-off-by: Robin Friedmann <robinfriedmann.rf@gmail.com> * Implement unit tests for FilterPathTests (opensearch-project#12067) Signed-off-by: Robin Friedmann <robinfriedmann.rf@gmail.com> * Write warn log if Filter is empty; Add comments (opensearch-project#12067) Signed-off-by: Robin Friedmann <robinfriedmann.rf@gmail.com> * Add changelog Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Remove unnecessary log statement Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Remove unused logger Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Spotless apply Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Remove incorrect changelog Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> --------- Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> Co-authored-by: Robin Friedmann <robinfriedmann.rf@gmail.com> * Removing String format in RemoteStoreMigrationAllocationDecider to optimise performance(opensearch-project#14612) Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com> * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata; Correct the check for deciding upload of HashesOfConsistentSettings (opensearch-project#14513) * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata * Correct the check for deciding upload of hashes of consistent settings Signed-off-by: Sooraj Sinha <soosinha@amazon.com> * Improve reroute performance by optimising List.removeAll in LocalShardsBalancer to filter remote search shard from relocation decision (opensearch-project#14613) Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com> * Fix assertion failure while deleting remote backed index (opensearch-project#14601) Signed-off-by: Sachin Kale <kalsac@amazon.com> * OnHeap Star Tree Implementation Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * addressed nits Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * addressed major nits Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * includes Count Aggregator Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * handling for missing doc values Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * addressing review comments Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * rebasing with main Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * support for empty sequential doc values iterator Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * nits Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * min and max star tree aggregators Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * star tree file formats * Star tree codec changes Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * Adding tests Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * Addressing comments Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * addressing review comments Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * Star tree merge changes Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * fix annotations * star-tree file formats reader and javadoc fixes * read for composite index values Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * doc values file format Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> --------- Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: kkewwei kkewwei@163.com Signed-off-by: kkewwei <kkewwei@163.com> Signed-off-by: Peter Alfonsi <petealft@amazon.com> Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Sagar Upadhyaya <upasagar@amazon.com> Signed-off-by: panguixin <panguixin@bytedance.com> Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Liyun Xiu <xiliyun@amazon.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> Signed-off-by: Daniel Widdis <widdis@gmail.com> Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com> Signed-off-by: Sooraj Sinha <soosinha@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> Signed-off-by: Bharathwaj G <bharath78910@gmail.com> Co-authored-by: bowenlan-amzn <bowenlan23@gmail.com> Co-authored-by: Prudhvi Godithi <pgodithi@amazon.com> Co-authored-by: Sagar <99425694+sgup432@users.noreply.github.com> Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com> Co-authored-by: kkewwei <kkewwei@163.com> Co-authored-by: Marc Handalian <handalm@amazon.com> Co-authored-by: Peter Alfonsi <peter.alfonsi@gmail.com> Co-authored-by: Peter Alfonsi <petealft@amazon.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <83309141+opensearch-ci-bot@users.noreply.github.com> Co-authored-by: Andrew Ross <andrross@amazon.com> Co-authored-by: panguixin <panguixin@bytedance.com> Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Shivansh Arora <hishiv@amazon.com> Co-authored-by: Ashish Singh <ssashish@amazon.com> Co-authored-by: Liyun Xiu <xiliyun@amazon.com> Co-authored-by: Vatsal <36672090+imvtsl@users.noreply.github.com> Co-authored-by: Daniel Widdis <widdis@gmail.com> Co-authored-by: Siddhant Deshmukh <deshsid@amazon.com> Co-authored-by: Robin Friedmann <robinfriedmann.rf@gmail.com> Co-authored-by: rishavz_sagar <rishavsagar4b1@gmail.com> Co-authored-by: Sooraj Sinha <81695996+soosinha@users.noreply.github.com> Co-authored-by: Sachin Kale <sachinpkale@gmail.com> Co-authored-by: Sarthak Aggarwal <sarthagg@amazon.com>

@internalapi

* Fix flaky test in range aggregation yaml test (opensearch-project#14486) Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Use CODECOV_TOKEN (opensearch-project#14536) Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> * [Tiered Caching] Moving query recomputation logic outside of write lock (opensearch-project#14187) * Moving query recompute out of write lock Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * [Tiered Caching] Moving query recomputation logic outside of write lock Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Adding java doc for the completable map Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Changes to call future handler only once per key Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing spotless check Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Added changelog Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Addressing comments Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing gradle fail Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Addressing comments to refactor unit test Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * minor UT refactor Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> --------- Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com> * Fix Flaky Test ClusterRerouteIT.testDelayWithALargeAmountOfShards (opensearch-project#14510) Signed-off-by: kkewwei kkewwei@163.com Signed-off-by: kkewwei kkewwei@163.com Signed-off-by: kkewwei <kkewwei@163.com> * Add doc for debugging rest tests (opensearch-project#14491) * add doc for debugging rest tests Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Update TESTING.md Co-authored-by: Marc Handalian <handalm@amazon.com> Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Address comment Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> --------- Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Co-authored-by: Marc Handalian <handalm@amazon.com> * Fix flaky DefaultCacheStatsHolderTests (opensearch-project#14462) Signed-off-by: Peter Alfonsi <petealft@amazon.com> Co-authored-by: Peter Alfonsi <petealft@amazon.com> * [AUTO] [main] Add bwc version 2.15.1. (opensearch-project#14549) * Add bwc version 2.15.1 Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fix auto-generated version Signed-off-by: Andrew Ross <andrross@amazon.com> --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <andrross@amazon.com> Co-authored-by: opensearch-ci-bot <83309141+opensearch-ci-bot@users.noreply.github.com> Co-authored-by: Andrew Ross <andrross@amazon.com> * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently (opensearch-project#14550) * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Addressing comment Signed-off-by: Sagar Upadhyaya <upasagar@amazon.com> --------- Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Sagar Upadhyaya <upasagar@amazon.com> Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com> * Add allowlist setting for ingest-common processors (opensearch-project#14479) Add a new static setting that lets an operator choose specific ingest processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <andrross@amazon.com> * Fix file cache initialization (opensearch-project#14004) * fix file cache initialization Signed-off-by: panguixin <panguixin@bytedance.com> * changelog Signed-off-by: panguixin <panguixin@bytedance.com> * add test Signed-off-by: panguixin <panguixin@bytedance.com> --------- Signed-off-by: panguixin <panguixin@bytedance.com> * Add Ashish Singh as maintainer (opensearch-project#14567) Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> * Allow @internalapi annotation on classes not meant to be constructed outside of the OpenSearch core (opensearch-project#14575) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Bump com.azure:azure-storage-common from 12.21.2 to 12.25.1 in /plugins/repository-azure (opensearch-project#14517) * Bump com.azure:azure-storage-common in /plugins/repository-azure Bumps [com.azure:azure-storage-common](https://github.com/Azure/azure-sdk-for-java) from 12.21.2 to 12.25.1. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-common_12.21.2...azure-storage-blob_12.25.1) --- updated-dependencies: - dependency-name: com.azure:azure-storage-common dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Add allowlist setting for search-pipeline-common processors (opensearch-project#14562) Add a new static setting that lets an operator choose specific search pipeline processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. A distinct setting exists for each of request, response, and search phase results processors. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <andrross@amazon.com> * Bump Apache Lucene to 9.11.1 (opensearch-project#14576) (opensearch-project#14581) (cherry picked from commit 0095fd1) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Add unittests for RemoteClusterStateAttributesManager (opensearch-project#14427) * Add unittests for RemoteClusterStateAttributesManager Signed-off-by: Shivansh Arora <hishiv@amazon.com> * Add Ashish Singh to codeowners (opensearch-project#14592) Signed-off-by: Ashish Singh <ssashish@amazon.com> * Add batching processor base type AbstractBatchingProcessor (opensearch-project#14554) Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Add @internalapi annotation to japicmp exclusions (opensearch-project#14597) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Fix issue 14519:Parsing a GetResult returns NPE if found field is mis… (opensearch-project#14552) * Fix issue 14519:Parsing a GetResult returns NPE if found field is missing. Signed-off-by: Vatsal <vatsal.v.anand@gmail.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> * Fix issue 14519:Parsing a GetResult returns NPE if found field is missing. Signed-off-by: Vatsal <vatsal.v.anand@gmail.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> * Fix issue 14519:Fix wildcart import. Signed-off-by: Vatsal <vatsal.v.anand@gmail.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> * Fix issue 14519:Fix wildcart import. Signed-off-by: Vatsal <vatsal.v.anand@gmail.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> * Fix issue 14519:Fix spotless issues. Signed-off-by: Vatsal <vatsal.v.anand@gmail.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> * Fix issue 14519:update changelog Signed-off-by: vatsal <vatsal.v.anand@gmail.com> --------- Signed-off-by: vatsal <vatsal.v.anand@gmail.com> Signed-off-by: Daniel Widdis <widdis@gmail.com> Co-authored-by: Daniel Widdis <widdis@gmail.com> * Star tree mapping changes (opensearch-project#14605) * Star tree mapping changes with feature flag --------- Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * Bump com.microsoft.azure:msal4j from 1.15.1 to 1.16.0 in /plugins/repository-azure (opensearch-project#14610) * Bump com.microsoft.azure:msal4j in /plugins/repository-azure Bumps [com.microsoft.azure:msal4j](https://github.com/AzureAD/microsoft-authentication-library-for-java) from 1.15.1 to 1.16.0. - [Release notes](https://github.com/AzureAD/microsoft-authentication-library-for-java/releases) - [Changelog](https://github.com/AzureAD/microsoft-authentication-library-for-java/blob/dev/changelog.txt) - [Commits](AzureAD/microsoft-authentication-library-for-java@v1.15.1...v1.16.0) --- updated-dependencies: - dependency-name: com.microsoft.azure:msal4j dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * [Bugfix] Fix ICacheKeySerializerTests flakiness (opensearch-project#14564) * Fix testInvalidInput flakiness Signed-off-by: Peter Alfonsi <petealft@amazon.com> * Addressed andrross's comment Signed-off-by: Peter Alfonsi <petealft@amazon.com> * rerun security check Signed-off-by: Peter Alfonsi <petealft@amazon.com> --------- Signed-off-by: Peter Alfonsi <petealft@amazon.com> Co-authored-by: Peter Alfonsi <petealft@amazon.com> * Correct typo in method name (opensearch-project#14621) Signed-off-by: vatsal <vatsal.v.anand@gmail.com> * Refactoring FilterPath.parse by using an iterative approach instead of recursion. (opensearch-project#14200) * Refactor FilterPath parse function (opensearch-project#12067) Signed-off-by: Robin Friedmann <robinfriedmann.rf@gmail.com> * Implement unit tests for FilterPathTests (opensearch-project#12067) Signed-off-by: Robin Friedmann <robinfriedmann.rf@gmail.com> * Write warn log if Filter is empty; Add comments (opensearch-project#12067) Signed-off-by: Robin Friedmann <robinfriedmann.rf@gmail.com> * Add changelog Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Remove unnecessary log statement Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Remove unused logger Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Spotless apply Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Remove incorrect changelog Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> --------- Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> Co-authored-by: Robin Friedmann <robinfriedmann.rf@gmail.com> * Removing String format in RemoteStoreMigrationAllocationDecider to optimise performance(opensearch-project#14612) Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com> * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata; Correct the check for deciding upload of HashesOfConsistentSettings (opensearch-project#14513) * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata * Correct the check for deciding upload of hashes of consistent settings Signed-off-by: Sooraj Sinha <soosinha@amazon.com> * Improve reroute performance by optimising List.removeAll in LocalShardsBalancer to filter remote search shard from relocation decision (opensearch-project#14613) Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com> * Fix assertion failure while deleting remote backed index (opensearch-project#14601) Signed-off-by: Sachin Kale <kalsac@amazon.com> * OnHeap Star Tree Implementation Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * addressed nits Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * addressed major nits Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * includes Count Aggregator Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * handling for missing doc values Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * addressing review comments Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * rebasing with main Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * support for empty sequential doc values iterator Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * nits Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * min and max star tree aggregators Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * star tree file formats * Star tree codec changes Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * Adding tests Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * Addressing comments Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * addressing review comments Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * Star tree merge changes Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * fix annotations * star-tree file formats reader and javadoc fixes * read for composite index values Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * doc values file format Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> --------- Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: kkewwei kkewwei@163.com Signed-off-by: kkewwei <kkewwei@163.com> Signed-off-by: Peter Alfonsi <petealft@amazon.com> Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Sagar Upadhyaya <upasagar@amazon.com> Signed-off-by: panguixin <panguixin@bytedance.com> Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Liyun Xiu <xiliyun@amazon.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> Signed-off-by: Daniel Widdis <widdis@gmail.com> Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com> Signed-off-by: Sooraj Sinha <soosinha@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> Signed-off-by: Bharathwaj G <bharath78910@gmail.com> Co-authored-by: bowenlan-amzn <bowenlan23@gmail.com> Co-authored-by: Prudhvi Godithi <pgodithi@amazon.com> Co-authored-by: Sagar <99425694+sgup432@users.noreply.github.com> Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com> Co-authored-by: kkewwei <kkewwei@163.com> Co-authored-by: Marc Handalian <handalm@amazon.com> Co-authored-by: Peter Alfonsi <peter.alfonsi@gmail.com> Co-authored-by: Peter Alfonsi <petealft@amazon.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <83309141+opensearch-ci-bot@users.noreply.github.com> Co-authored-by: Andrew Ross <andrross@amazon.com> Co-authored-by: panguixin <panguixin@bytedance.com> Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Shivansh Arora <hishiv@amazon.com> Co-authored-by: Ashish Singh <ssashish@amazon.com> Co-authored-by: Liyun Xiu <xiliyun@amazon.com> Co-authored-by: Vatsal <36672090+imvtsl@users.noreply.github.com> Co-authored-by: Daniel Widdis <widdis@gmail.com> Co-authored-by: Siddhant Deshmukh <deshsid@amazon.com> Co-authored-by: Robin Friedmann <robinfriedmann.rf@gmail.com> Co-authored-by: rishavz_sagar <rishavsagar4b1@gmail.com> Co-authored-by: Sooraj Sinha <81695996+soosinha@users.noreply.github.com> Co-authored-by: Sachin Kale <sachinpkale@gmail.com> Co-authored-by: Sarthak Aggarwal <sarthagg@amazon.com>

* fix file cache initialization Signed-off-by: panguixin <panguixin@bytedance.com> * changelog Signed-off-by: panguixin <panguixin@bytedance.com> * add test Signed-off-by: panguixin <panguixin@bytedance.com> --------- Signed-off-by: panguixin <panguixin@bytedance.com>

…project#14566) * fix file cache initialization * changelog * add test --------- (cherry picked from commit 2be25bb) Signed-off-by: panguixin <panguixin@bytedance.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: kkewwei <kkewwei@163.com>

* fix file cache initialization Signed-off-by: panguixin <panguixin@bytedance.com> * changelog Signed-off-by: panguixin <panguixin@bytedance.com> * add test Signed-off-by: panguixin <panguixin@bytedance.com> --------- Signed-off-by: panguixin <panguixin@bytedance.com>

bugmakerrrrrr requested review from anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, sachinpkale, saratvemulapalli, shwetathareja, sohami, tlfeng and VachaShah as code owners June 5, 2024 14:15

bugmakerrrrrr commented Jun 5, 2024

View reviewed changes

server/src/main/java/org/opensearch/node/Node.java Show resolved Hide resolved

bugmakerrrrrr commented Jun 5, 2024

View reviewed changes

server/src/main/java/org/opensearch/node/Node.java Outdated Show resolved Hide resolved

This was referenced Jun 13, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndicesRequestCacheIT #14288

Open

[AUTOCUT] Gradle Check Flaky Test Report for RemoteMigrationIndexMetadataUpdateIT #14295

Open

prudhvigodithi mentioned this pull request Jun 14, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndicesRequestCacheIT prudhvigodithi/OpenSearch#27

Closed

fix file cache initialization

91afdcb

Signed-off-by: panguixin <panguixin@bytedance.com>

bugmakerrrrrr added 2 commits June 26, 2024 00:29

changelog

b3be949

Signed-off-by: panguixin <panguixin@bytedance.com>

add test

bd73b60

Signed-off-by: panguixin <panguixin@bytedance.com>

bugmakerrrrrr force-pushed the fix_file_cache branch from afe886f to bd73b60 Compare June 25, 2024 17:03

opensearch-ci-bot mentioned this pull request Jun 25, 2024

[AUTOCUT] Gradle Check Flaky Test Report for AzureBlobStoreRepositoryTests #14291

Open

andrross approved these changes Jun 26, 2024

View reviewed changes

andrross added the backport 2.x Backport to 2.x branch label Jun 26, 2024

kotwanikunal approved these changes Jun 26, 2024

View reviewed changes

andrross merged commit 2be25bb into opensearch-project:main Jun 26, 2024
35 of 36 checks passed

opensearch-trigger-bot bot mentioned this pull request Jun 26, 2024

[Backport 2.x] Fix file cache initialization #14566

Merged

bugmakerrrrrr mentioned this pull request Jun 30, 2024

Setting a limit for file cache capacity #14602

Open

finnegancarroll mentioned this pull request Jul 10, 2024

[BUG] 2.11.1-2.18.0 searching searchable snapshots fails because search node run out of disk space after some time #11676

Open

finnegancarroll mentioned this pull request Jul 15, 2024

Make remote snapshot (local)block size configurable #14753

Closed

3 tasks

BrewTestBot mentioned this pull request Aug 8, 2024

opensearch 2.16.0 Homebrew/homebrew-core#180446

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix file cache initialization #14004

Fix file cache initialization #14004

bugmakerrrrrr commented Jun 5, 2024

bugmakerrrrrr commented Jun 5, 2024

github-actions bot commented Jun 5, 2024

github-actions bot commented Jun 5, 2024

codecov bot commented Jun 25, 2024 •

edited

Loading

bugmakerrrrrr commented Jun 25, 2024

github-actions bot commented Jun 25, 2024

Fix file cache initialization #14004

Fix file cache initialization #14004

Conversation

bugmakerrrrrr commented Jun 5, 2024

Description

Related Issues

Check List

bugmakerrrrrr commented Jun 5, 2024

github-actions bot commented Jun 5, 2024

github-actions bot commented Jun 5, 2024

codecov bot commented Jun 25, 2024 • edited Loading

Codecov Report

bugmakerrrrrr commented Jun 25, 2024

github-actions bot commented Jun 25, 2024

codecov bot commented Jun 25, 2024 •

edited

Loading