Async write routing table flow #8
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…-project#12989) * Cache the shard routings with no weight for faster access The list of shards to run a query is determined for every request and the weight of the nodes guides the shard selection. Currently, IndexRoutingTable caches the shard routings with weight for faster access. But, during cases where the fail open option is enabled, shards with no weight is also returned lower in the order along with shards with weights. They will be used as fall back if the shards with weights can't be used due to some error. The shard routing with no weight is not cached, hence it does a full loop for every request, this impacts the search latency when the number of shards to query or the number of nodes in the cluster is high. The latency impact is very high when both the number of shards and the number of nodes are high. This change introduces a caching mechanism for shard routing with no weights similar to the existing cache for shard routing with weights. Signed-off-by: Prabhakar Sithanandam <prabhakar.s87@gmail.com> Co-authored-by: Prabhakar Sithanandam <prabs@amazon.com>
…2883) * Fix bulk API ignores ingest pipeline for upsert Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Modify change log Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Use existing pipeline instead Signed-off-by: Gao Binlong <gbinlong@amazon.com> --------- Signed-off-by: Gao Binlong <gbinlong@amazon.com>
) * Update links to documentation in rest-api-spec Signed-off-by: Craig Perkins <craig5008@gmail.com> * Remove changes in Debug_OpenSearch.xml Signed-off-by: Craig Perkins <craig5008@gmail.com> * Remove changes in Debug_OpenSearch.xml Signed-off-by: Craig Perkins <craig5008@gmail.com> * Add to CHANGELOG Signed-off-by: Craig Perkins <craig5008@gmail.com> --------- Signed-off-by: Craig Perkins <craig5008@gmail.com>
…nsearch-project#13056) Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Bump up commons-compress to 1.26.0 to fix CVE Signed-off-by: Aman Khare <amkhar@amazon.com> * Change log entry Signed-off-by: Aman Khare <amkhar@amazon.com> * Update ignoreMissingClasses Signed-off-by: Aman Khare <amkhar@amazon.com> * Update commons-codec and commons-lang3 dependencies also Signed-off-by: Aman Khare <amkhar@amazon.com> * Upgrade commons-codec to 1.16.1 Signed-off-by: Aman Khare <amkhar@amazon.com> * Add commons-io dependency in plugin-cli build.gradle Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * Revert "Update ignoreMissingClasses" This reverts commit d92fbda. Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * Adding SHA for commons-io-2.15.1.jar Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * adding license, notice files for commons-io Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * Add missing classes for thirdPartyAudit Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * Refactor Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * Test commit - to be reverted Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * Bump commons-compress to 1.26.1, tika to 2.9.1 Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * Remove Charsets class from exclusion list - not missing Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * Update tika to 2.9.2 Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * commons-io 2.16.0 Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * Refactor commons-io dependency mentions to avoid manual version setting/update Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> --------- Signed-off-by: Aman Khare <amkhar@amazon.com> Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> Co-authored-by: Aman Khare <amkhar@amazon.com>
* Add bwc version 2.13.1 Signed-off-by: GitHub <noreply@github.com> * Fix version identifier Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> --------- Signed-off-by: GitHub <noreply@github.com> Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Co-authored-by: opensearch-ci-bot <opensearch-ci-bot@users.noreply.github.com> Co-authored-by: Kunal Kotwani <kkotwani@amazon.com>
…ieldScripts (opensearch-project#12837) --------- Signed-off-by: Mohammad Qureshi <qreshi@amazon.com> Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> Co-authored-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
…h-project#13044) Signed-off-by: Sachin Kale <kalsac@amazon.com>
…h-project#12686) Signed-off-by: Shubh Sahu <shubhvs@amazon.com>
…12981) * Finish contributing updates; thanks @acarbonetto Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Fix indenting Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Readd build failure notes Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update CONTRIBUTING with review tips Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> * Update CONTRIBUTING with review tips Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> * Update contributing comment for review comment Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> * Fix contruibuting doc Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update CONTRIBUTING.md Co-authored-by: Owais Kazi <owaiskazi19@gmail.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update CONTRIBUTING.md Co-authored-by: Owais Kazi <owaiskazi19@gmail.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update CONTRIBUTING.md Co-authored-by: Owais Kazi <owaiskazi19@gmail.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update CONTRIBUTING.md Co-authored-by: Owais Kazi <owaiskazi19@gmail.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update CONTRIBUTING.md Co-authored-by: Owais Kazi <owaiskazi19@gmail.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update CONTRIBUTING.md Co-authored-by: Owais Kazi <owaiskazi19@gmail.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update CONTRIBUTING.md Co-authored-by: Owais Kazi <owaiskazi19@gmail.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update CONTRIBUTING.md Co-authored-by: Owais Kazi <owaiskazi19@gmail.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> --------- Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Co-authored-by: Andrew Carbonetto <andrewc@bitquilltech.com> Co-authored-by: Owais Kazi <owaiskazi19@gmail.com>
…opensearch-project#12988) Signed-off-by: Ashish Singh <ssashish@amazon.com>
…rch-project#13067) * Remove reference to INDEX_MAPPER_DYNAMIC_SETTING in AutoCreateIndex Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add to CHANGELOG Signed-off-by: Craig Perkins <cwperx@amazon.com> --------- Signed-off-by: Craig Perkins <cwperx@amazon.com>
…th java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer (opensearch-project#13100) Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
…pensearch-project#13099) WeightedRoutingKey is used as key in the HashMap to retreive the shard routings, for every usage the hash key is computed. WeightedRoutingKey internally uses WeightedRouting. As the number of the shards getting queried increases, the computation cost goes high. Computing the hash key during object creation as the object is immutable and using it through out the object thus saving on the additional computation. Signed-off-by: Prabhakar Sithanandam <prabhakar.s87@gmail.com>
…tadata (opensearch-project#13023) * Introduce interface changes to read/write blob with object metadata --------- Signed-off-by: Sandeep Kumawat <skumwt@amazon.com> Co-authored-by: Sandeep Kumawat <skumwt@amazon.com>
…project#12967) * Adding support for more than one protocol for transport Signed-off-by: Vacha Shah <vachshah@amazon.com> * Adding CHANGELOG entry Signed-off-by: Vacha Shah <vachshah@amazon.com> * Addressing comments Signed-off-by: Vacha Shah <vachshah@amazon.com> * Addressing comments Signed-off-by: Vacha Shah <vachshah@amazon.com> * Removing determineTransportProtocol Signed-off-by: Vacha Shah <vachshah@amazon.com> * Determine transport protocol only on first byte reference Signed-off-by: Vacha Shah <vachshah@amazon.com> * Making InboundBytesHandler closeable Signed-off-by: Vacha Shah <vachshah@amazon.com> * Fixing close() for InboundPipeline Signed-off-by: Vacha Shah <vachshah@amazon.com> * Adding DeprecatedAPI annotation to japicmp task Signed-off-by: Vacha Shah <vachshah@amazon.com> * Fixing for detect breaking changes workflow Signed-off-by: Vacha Shah <vachshah@amazon.com> * Fixing recursion Signed-off-by: Vacha Shah <vachshah@amazon.com> --------- Signed-off-by: Vacha Shah <vachshah@amazon.com>
opensearch-project#12960) Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
…ng. (opensearch-project#12488) * Add support for randomizing remote store enabled testing. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * fix spotless check Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Updating logic to randomly use remote store once per cluster instead of once per node. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Add Remote Store Randomization within Replication Type Randomization. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> --------- Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
) * Bump com.gradle.enterprise from 3.16.2 to 3.17 Bumps com.gradle.enterprise from 3.16.2 to 3.17. --- updated-dependencies: - dependency-name: com.gradle.enterprise dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
opensearch-project#13127) Signed-off-by: Craig Perkins <craig5008@gmail.com>
…3040) Contributors and maintainers frequently put changelog entries in the wrong section. It's an easy mistake to make! Given that making changes intended for the next minor release is the norm, this change optimizes that process by making `CHANGELOG.md` contain _only_ entries for the next minor. Truly breaking changes intended for the next major are kept in a separate file. Credit to @msfroh for the idea. Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Co-authored-by: Kunal Kotwani <kkotwani@amazon.com>
* implement mark() in class FilterStreamInput Signed-off-by: kkewwei <kkewwei@163.com> * implement markSupported() in class FilterStreamInput Signed-off-by: kkewwei <kkewwei@163.com> * add CHANGELOG Signed-off-by: kkewwei <kkewwei@163.com> --------- Signed-off-by: kkewwei <kkewwei@163.com>
Signed-off-by: frameflare <yangzenghua@outlook.com>
…y-hdfs (opensearch-project#13117) * Bump net.minidev:json-smart in /plugins/repository-hdfs Bumps [net.minidev:json-smart](https://github.com/netplex/json-smart-v2) from 2.5.0 to 2.5.1. - [Release notes](https://github.com/netplex/json-smart-v2/releases) - [Commits](netplex/json-smart-v2@2.5.0...2.5.1) --- updated-dependencies: - dependency-name: net.minidev:json-smart dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
opensearch-project#11868) Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com>
…equest (opensearch-project#12850) * Support derived fields definition in search request * adds support for fetch phase on derived fields * adds support for highlighting on derived fields --------- Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
…tartedShardsBatch (opensearch-project#12504) Signed-off-by: Aman Khare <amkhar@amazon.com>
…ng flaky test (opensearch-project#13164) Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
…oject#13508) Remove check of non zero assertion of time as it is non deterministic Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
* add use compound file setting Signed-off-by: Samuel Herman <sherman8915@gmail.com> * review feedback Signed-off-by: Samuel Herman <sherman8915@gmail.com> --------- Signed-off-by: Samuel Herman <sherman8915@gmail.com>
…ClusterManagerFailover (opensearch-project#13505) Signed-off-by: Rahul Karajgikar <karajgik@amazon.com> Co-authored-by: Rahul Karajgikar <karajgik@amazon.com>
…fault to 'true' (opensearch-project#13532) Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
… in /modules/lang-mustache (opensearch-project#13559) * Bump com.github.spullara.mustache.java:compiler Bumps [com.github.spullara.mustache.java:compiler](https://github.com/spullara/mustache.java) from 0.9.11 to 0.9.13. - [Commits](spullara/mustache.java@mustache.java-0.9.11...mustache.java-0.9.13) --- updated-dependencies: - dependency-name: com.github.spullara.mustache.java:compiler dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…#13469) * Update version for batch shard allocation to 2.14 Signed-off-by: Shivansh Arora <hishiv@amazon.com>
…ixtures/hdfs-fixture (opensearch-project#13557) * Bump org.apache.commons:commons-text in /test/fixtures/hdfs-fixture Bumps org.apache.commons:commons-text from 1.11.0 to 1.12.0. --- updated-dependencies: - dependency-name: org.apache.commons:commons-text dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
This solves the bug where RequestStats memory_size metric was going negative in certain scenarios as reported in the issue. It turns out that the issue occurs when an indexShard is deleted and then reallocated on the same node. So whenever stale entries from older shard are deleted, those are accounted for the new shard which has the same shardId. --------- Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>
…pensearch-project#13556) * Bump org.hdrhistogram:HdrHistogram from 2.1.12 to 2.2.1 in /server Bumps [org.hdrhistogram:HdrHistogram](https://github.com/HdrHistogram/HdrHistogram) from 2.1.12 to 2.2.1. - [Release notes](https://github.com/HdrHistogram/HdrHistogram/releases) - [Commits](HdrHistogram/HdrHistogram@HdrHistogram-2.1.12...HdrHistogram-2.2.1) --- updated-dependencies: - dependency-name: org.hdrhistogram:HdrHistogram dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…an be negative when searching (opensearch-project#13408) Signed-off-by: Gao Binlong <gbinlong@amazon.com>
…eStoreRestoreIT.testRTSRestoreWithNoDataPostRefreshPrimaryReplicaDown (opensearch-project#13489) * add TestIssueLogging on flaky test - org.opensearch.remotestore.RemoteStoreRestoreIT.testRTSRestoreWithNoDataPostRefreshPrimaryReplicaDown Signed-off-by: Varun Bansal <bansvaru@amazon.com>
Signed-off-by: Andrew Ross <andrross@amazon.com>
Signed-off-by: Sagar Upadhyaya <upasagar@amazon.com> Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
…tiple versions (opensearch-project#13624) * Fix get field mapping API returns 404 error in mixed cluster with multiple versions Signed-off-by: Gao Binlong <gbinlong@amazon.com> * modify changelog Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Add comment and remove unit test Signed-off-by: Gao Binlong <gbinlong@amazon.com> --------- Signed-off-by: Gao Binlong <gbinlong@amazon.com>
Signed-off-by: Kunal Khatua <kkhatua@amazon.com> Co-authored-by: Peter Nied <peternied@hotmail.com> Co-authored-by: Peter Nied <petern@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>
Signed-off-by: Himshikha Gupta <himshikh@amazon.com>
…' into async-write
…' into async-write
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
[Describe what this change achieves]
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.