Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shard movement #183

Closed
wants to merge 284 commits into from
Closed

Shard movement #183

wants to merge 284 commits into from

Conversation

Poojita-Raj
Copy link
Owner

Description

[Describe what this change achieves]

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Resolves #182

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

dependabot bot and others added 30 commits June 19, 2023 10:23
…ildSrc (opensearch-project#8141)

* Bump com.networknt:json-schema-validator in /buildSrc

Bumps [com.networknt:json-schema-validator](https://github.com/networknt/json-schema-validator) from 1.0.83 to 1.0.84.
- [Release notes](https://github.com/networknt/json-schema-validator/releases)
- [Changelog](https://github.com/networknt/json-schema-validator/blob/master/CHANGELOG.md)
- [Commits](networknt/json-schema-validator@1.0.83...1.0.84)

---
updated-dependencies:
- dependency-name: com.networknt:json-schema-validator
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update changelog

Signed-off-by: dependabot[bot] <support@github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
opensearch-project#8035)

This commit refactors the Stream IO classes from the server to core
library to support cloud native and serverless extensions.

The following classes are refactored:

* StreamInput
* StreamOutput
* Writeable
* NamedWriteable
* NamedWriteableRegistry
* Index
* ShardId
* BytesReference
* ByteArray
* BigArray
* SecureString
* Text
* ParsingException
* RestStatus

The namespace is left unchanged but will be refactored in a follow up
commit to avoid split package in order to support jigsaw modularity.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
…y-azure-classic (opensearch-project#8140)

* Bump commons-io:commons-io in /plugins/discovery-azure-classic

Bumps commons-io:commons-io from 2.12.0 to 2.13.0.

---
updated-dependencies:
- dependency-name: commons-io:commons-io
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

* Added changelog entry

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Co-authored-by: Kartik Ganesh <gkart@amazon.com>
Multiple changes were incorrectly placed in the "Unreleased 3.0" section
on main and then backported. This brings the changelog on main in line
with 2.x for changes that have been backported.

Signed-off-by: Andrew Ross <andrross@amazon.com>
This fixes some minor differences with the 2.x changelog

Signed-off-by: Kartik Ganesh <gkart@amazon.com>
…earch-project#7919)

* Updates Node start to now pass localNode info so that it is available directly to a plugin instead of hacky implementation

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>

* Adds this PR to changelog

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>

* Makes onNodeStarted call plugin agnostic

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>

* Marking onNodeStarted() as deprecated

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>

* Adds a test to verify the behaviour of new signature of onNodeStarted

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>

* Updates test to also check for nodeId

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>

* Adds instructions to use the overloaded implementation of onNodeStarted and adds a test to check that current plugins are not affect with this implementation

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>

---------

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>
Signed-off-by: Darshit Chanpura <35282393+DarshitChanpura@users.noreply.github.com>
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
…oject#6969)

* add ExtensionActionUtill class

Signed-off-by: Kuanysh <kuanysh4646@gmail.com>

* add ExtensionActionUtilTest class

Signed-off-by: Kuanysh <kuanysh4646@gmail.com>

* refactoring

Signed-off-by: Kuanysh <kuanysh4646@gmail.com>

* add method convertParamsToBytes

Signed-off-by: Kuanysh <kuanysh4646@gmail.com>

* code refactoring

Signed-off-by: Kuanysh <kuanysh4646@gmail.com>

* add ExtensionActionUtill class

Signed-off-by: Kuanysh <kuanysh4646@gmail.com>

* add ExtensionActionUtilTest class

Signed-off-by: Kuanysh <kuanysh4646@gmail.com>

* refactoring

Signed-off-by: Kuanysh <kuanysh4646@gmail.com>

* add method convertParamsToBytes

Signed-off-by: Kuanysh <kuanysh4646@gmail.com>

* code refactoring

Signed-off-by: Kuanysh <kuanysh4646@gmail.com>

* refactoring

Signed-off-by: Kuanysh <kuanysh4646@gmail.com>

* added new name to tests

Signed-off-by: Kuanysh <kuanysh4646@gmail.com>

* code refactoring

Signed-off-by: Kuanysh <90975457+Kuanysh-kst@users.noreply.github.com>

* test code refactoring

Signed-off-by: Kuanysh <90975457+Kuanysh-kst@users.noreply.github.com>

* test class renaming

Signed-off-by: Kuanysh <90975457+Kuanysh-kst@users.noreply.github.com>

---------

Signed-off-by: Kuanysh <kuanysh4646@gmail.com>
Signed-off-by: Kuanysh <90975457+Kuanysh-kst@users.noreply.github.com>
…oject#7456)

* add equels for ClusterSettings

Signed-off-by: Kuanysh <kuanysh4646@gmail.com>

* added junit

Signed-off-by: Kuanysh <kuanysh4646@gmail.com>

* code refactoring

Signed-off-by: Kuanysh <kuanysh4646@gmail.com>

* added changes to handleAddSettingsUpdateConsumer

Signed-off-by: Kuanysh <90975457+Kuanysh-kst@users.noreply.github.com>

* code refactoring

Signed-off-by: Kuanysh Aimurzinov <kuanysh4646@gmail.com>

* code refactoring

Signed-off-by: Kuanysh Aimurzinov <kuanysh4646@gmail.com>

* changed main method

Signed-off-by: Kuanysh <90975457+Kuanysh-kst@users.noreply.github.com>

---------

Signed-off-by: Kuanysh <kuanysh4646@gmail.com>
Signed-off-by: Kuanysh <90975457+Kuanysh-kst@users.noreply.github.com>
Signed-off-by: Kuanysh Aimurzinov <kuanysh4646@gmail.com>
Signed-off-by: Andrew Ross <andrross@amazon.com>
… it (opensearch-project#8029)

* Implemented REST API for initializing extension

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

* Cleanup extensions.yml design

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

* Added tests for RestInitializeExtensionAction

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

* Pulled extensions REST request in extensions directory

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

* Removed forbidden APIs from rest action and modified tests

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

* Added entry in changelog

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

* Added test for parse

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

* Addressed PR comments

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

* Addressed PR comments

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

* Spotless Fixed

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

* Handled exceptions

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

* Handled test failure

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

---------

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
…rch-project#7998)

Add compatibility check gradle task ./gradlew checkCompatibility

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>
…rch-project#8063)

* fix thread issue

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

* fix thread issue

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

* Fix thresholds

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

* Swap to object based

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

* Spotless

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

* Swap to preserve nulls

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

* Spotless

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

* Resolve npe

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

* remove final declerations

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

* spotless

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

* add annotations

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

* push to rerun tests

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

* Fix idea

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

* Fix idea

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

---------

Signed-off-by: Stephen Crawford <steecraw@amazon.com>
… search source (opensearch-project#8206)

* Return null for aggregations factory even if empty aggs is present in search source

Signed-off-by: Jay Deng <jayd0104@gmail.com>

* Update rest-api-spec/src/main/resources/rest-api-spec/test/search.aggregation/60_empty.yml

Co-authored-by: Andriy Redko <drreta@gmail.com>
Signed-off-by: Jay Deng <jayd0104@gmail.com>

---------

Signed-off-by: Jay Deng <jayd0104@gmail.com>
Co-authored-by: Andriy Redko <drreta@gmail.com>
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
…rategy change when configuring remote storage with cluster setting (opensearch-project#8186)

* Removing overriding of replication type setting in updateRemoteStoreSettings()

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* fixing failing test testRemoteStoreNoUserOverrideConflictingReplicationTypeIndexSettings() by modifying exception message.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Fix failing tests.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* remove unnecessary system index integ test.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* revert previous change of removing system index integ test.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Address comments on PR.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Address comments.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Address comments on PR.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

---------

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
Signed-off-by: Rishikesh Pasham <62345295+Rishikesh1159@users.noreply.github.com>
This change was backported to 2.x in opensearch-project#7945.

Signed-off-by: Andrew Ross <andrross@amazon.com>
…8228)

* Fixes duplicate extension unique id

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

* Handled different extension unique id

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

---------

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
…mote store is used with system indices (opensearch-project#8235)

* Removing throwing of exception when remote store is used with system indices.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Add check of feature flag enabled.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Add unit test to improve code coverage.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Add warning log.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

---------

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
…ocation (opensearch-project#8244)

Signed-off-by: Marc Handalian <handalm@amazon.com>
…points and skip tests for 1.x bwc versions (opensearch-project#8203)

* [Segment Replication] Verify segment replication stats in bwc test

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Log cleanup

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless check

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add version check to skip test for 1.x bwc branches

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add version check to skip test for 1.x bwc branches for mixed clusters

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add version string in build to identify bwc version

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Use correct bwc version string

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments from opensearch-project#7626

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
…t/fixtures/hdfs-fixture (opensearch-project#8257)

* Bump org.apache.hadoop:hadoop-minicluster in /test/fixtures/hdfs-fixture

Bumps org.apache.hadoop:hadoop-minicluster from 3.3.5 to 3.3.6.

---
updated-dependencies:
- dependency-name: org.apache.hadoop:hadoop-minicluster
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update changelog

Signed-off-by: dependabot[bot] <support@github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…ildSrc (opensearch-project#8255)

* Bump com.networknt:json-schema-validator in /buildSrc

Bumps [com.networknt:json-schema-validator](https://github.com/networknt/json-schema-validator) from 1.0.84 to 1.0.85.
- [Release notes](https://github.com/networknt/json-schema-validator/releases)
- [Changelog](https://github.com/networknt/json-schema-validator/blob/master/CHANGELOG.md)
- [Commits](networknt/json-schema-validator@1.0.84...1.0.85)

---
updated-dependencies:
- dependency-name: com.networknt:json-schema-validator
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update changelog

Signed-off-by: dependabot[bot] <support@github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…n /plugins/repository-azure (opensearch-project#8256)

* Bump io.projectreactor.netty:reactor-netty-http

Bumps [io.projectreactor.netty:reactor-netty-http](https://github.com/reactor/reactor-netty) from 1.1.7 to 1.1.8.
- [Release notes](https://github.com/reactor/reactor-netty/releases)
- [Commits](reactor/reactor-netty@v1.1.7...v1.1.8)

---
updated-dependencies:
- dependency-name: io.projectreactor.netty:reactor-netty-http
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

* Update changelog

Signed-off-by: dependabot[bot] <support@github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Bumps [org.jruby.joni:joni](https://github.com/jruby/joni) from 2.1.48 to 2.2.1.
- [Commits](jruby/joni@joni-2.1.48...joni-2.2.1)

---
updated-dependencies:
- dependency-name: org.jruby.joni:joni
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: bansvaru <bansvaru@amazon.com>
* [Upgrade] Lucene 9.7.0 release

Upgrades to the official 9.7.0 release of lucene

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* update changelog

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

---------

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
gaiksaya and others added 7 commits August 1, 2023 15:27
Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>
Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>
…opensearch-project#9053)

This changes the version check for MediaType in PercolateQueryBuilder
for BWC.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
…ch-project#8767)

* [Segment Replication] Refactor remote replication source

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Unit test updates

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Segregate shard level tests for node to node and remote store segment replication

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix more unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Improve RemoteStoreReplicationSourceTests, remove unnecessary mocks and use actual failures for failure/exception use cases

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless check fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Ignore files already in store while computing segment file diff with primary

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Move read/writes from IndexInput/Output to RemoteSegmentMetadata

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review commnt

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update recovery flow to perform commits during recovery

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove un-necessary char

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update comment nit-pick

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove deletion logic causing read issues due to deleted segments_N

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
…nsearch-project#8737)

* Disallowing compression level for lz4 and best_compression codec
* setting up codec settings interface for validation

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Added the featureFlagSettings method to OpenSearchSingleNodeTestCase

Signed-off-by: Stefano Maglione <maglionestefanoinfo@gmail.com>
@github-actions
Copy link

github-actions bot commented Aug 2, 2023

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL:
  • CommitID: f2eb1fa
    Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
    Is the failure a flaky test unrelated to your change?

nknize and others added 10 commits August 2, 2023 15:01
…rch-project#9073)

This commit refactors the following network and transport libraries to
the opensearch common and core libraries respectively:

* o.o.common.network.Cidrs -> :libs:opensearch-common
* o.o.common.network.InetAddresses -> :libs:opensearch-common
* o.o.common.network.NetworkAddress -> :libs:opensearch-common
* o.o.common.transport.NetworkExceptionHelper -> :libs:opensearch-common
* o.o.common.transport.PortsRange -> :libs:opensearch-common

* o.o.common.transport.TransportAddress -> :libs:opensearch-core
* o.o.common.transport.BoundTransportAddress -> :libs:opensearch-core
* o.o.transport.TransportMessage -> :libs:opensearch-core
* o.o.transport.TransportResponse -> :libs:opensearch-core

The purpose is to reduce the change surface area of the core APIs to
minimize impact to downstream consumers while moving toward establishing
a formal API for cloud native or serverless implementations.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
…st index deletion. (opensearch-project#8472)

Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com>
…search-project#9057)

---------

Signed-off-by: Ashish Singh <ssashish@amazon.com>
…ckpoint validation (opensearch-project#8889)

* Fix test testDropPrimaryDuringReplication and clean up ReplicationCheckpoint validation.

This test is now occasionally failing with replicas having 0 documents. This occurs in a couple of ways:
1. After dropping the old primary the new primary is not publishing a checkpoint to replicas unless it indexes docs from translog after flipping to primary mode.
If there is nothing to index, it will not publish a checkpoint, but the other replica could have never sync'd with the original primary and be left out of date.
- This PR fixes this by force publishing a checkpoint after the new primary flips to primary mode.
2. The replica receives a checkpoint post failover and cancels its sync with the former primary that is still active, recognizing a primary term bump.
However this cancellation is async and immediately starting a new replication event could fail as its still replicating.
- This PR fixes this by attempting to process the latest received checkpoint on failure, if the shard is not failed and still behind.

This PR also introduces a few changes to ensure the accuracy of the ReplicationCheckpoint tracked on primary & replicas.
- Ensure the checkpoint stored in SegmentReplicationTarget is the checkpoint passed from the primary and not locally computed.  This ensures checks for primary term are accurate and not using a locally compued operationPrimaryTerm.
- Introduces a refresh listener for both primary & replica to update the ReplicationCheckpoint and store it in replicationTracker post refresh rather than redundantly computing when accessed.
- Removes unnecessary onCheckpointPublished method used to start replication timers manually.  This will happen automatically on primaries once its local cp is updated.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Handle NoSuchFileException when attempting to delete decref'd files.

To avoid divergent logic with remote store, we always incref/decref the segmentinfos.files(true) which includes the segments_n file.
Decref to 0 will attempt to delete the file from the store and its possible this _n file does not yet exist. This change will ignore if we get a noSuchFile while attempting to delete.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add more unit tests.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Clean up IndexShardTests.testCheckpointReffreshListenerWithNull

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Remove unnecessary catch for NoSuchFileException.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add another test for non segrep.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* PR Feedback.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* re-compute replication checkpoint on primary promotion.

Signed-off-by: Marc Handalian <handalm@amazon.com>

---------

Signed-off-by: Marc Handalian <handalm@amazon.com>
…e request during SearchContext creation (opensearch-project#9059)

* Change to determine if concurrent segment search should be used by the request during SearchContext creation. It caches the e evaluated output for all future invocation. It also provide executor to IndexSearcher based on this evaluation

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

* Add test case to enable/disable concurrent search cluster setting and verify context object state

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

---------

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>
…ect#9098)

The [commit to comment out `forceFailure()`][1] was done about 8 years
ago so I don't think we'll be bringing it back. Also this makes the code
base a slightly friendlier place :)

[1]: opensearch-project@e5a699f

Signed-off-by: Andrew Ross <andrross@amazon.com>
Signed-off-by: Suraj Singh <surajrider@gmail.com>
…ng search (opensearch-project#9047)

Signed-off-by: Neetika Singhal <neetiks@amazon.com>
…opensearch-project#9120)

This commit refactors the CollectionUtils, and Iterators utility classes
to the core and common libraries, respectively. Several other base
Strings utility methods are also refactored to the core Strings utility
library. This is done to incrementally move the remaining Strings
methods in server to the core String utility.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
@github-actions
Copy link

github-actions bot commented Aug 4, 2023

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL:
  • CommitID: 1d8821b
    Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
    Is the failure a flaky test unrelated to your change?

Signed-off-by: Poojita Raj <poojiraj@amazon.com>
Signed-off-by: Poojita Raj <poojiraj@amazon.com>
Signed-off-by: Poojita Raj <poojiraj@amazon.com>
Signed-off-by: Poojita Raj <poojiraj@amazon.com>
Signed-off-by: Poojita Raj <poojiraj@amazon.com>
Signed-off-by: Poojita Raj <poojiraj@amazon.com>
@github-actions
Copy link

github-actions bot commented Aug 4, 2023

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL:
  • CommitID: 342b6ed
    Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
    Is the failure a flaky test unrelated to your change?

@github-actions
Copy link

This PR is stalled because it has been open for 30 days with no activity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG]hi