Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade lucene version #5570

Merged
merged 6 commits into from
Dec 14, 2022

Conversation

dreamer-89
Copy link
Member

@dreamer-89 dreamer-89 commented Dec 14, 2022

Description

Upgrades the lucene snapshot to 9.5.0-snapshot-d5cef1c. Post 2.4.1 release, the next patch version 2.4.2 needed 9.4.2 lucene version which is not present in existing lucene snapshot.

Related: #5560

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

opensearch-ci-bot and others added 3 commits December 13, 2022 17:47
Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>
Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>
Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: Suraj Singh <surajrider@gmail.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@reta
Copy link
Collaborator

reta commented Dec 14, 2022

@dreamer-89 thanks for the update, could you please update Lucene versions as well [1] for 2.4.1 and 2.5.0 (from custom ones), thank you

[1] https://github.com/opensearch-project/OpenSearch/pull/5560/files#diff-5fe8bce6f41cbba30ab4dad09445194a255d5dc8a8fecb94e0152351b84b1588

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@dreamer-89
Copy link
Member Author

dreamer-89 commented Dec 14, 2022

@dreamer-89 thanks for the update, could you please update Lucene versions as well [1] for 2.4.1 and 2.5.0 (from custom ones), thank you

[1] https://github.com/opensearch-project/OpenSearch/pull/5560/files#diff-5fe8bce6f41cbba30ab4dad09445194a255d5dc8a8fecb94e0152351b84b1588

Thank you @reta for the quick update. Is this needed for gradle check to pass ? I see lot of java compilation errors in precommit.

/home/runner/work/OpenSearch/OpenSearch/server/src/main/java/org/opensearch/index/codec/CodecService.java:66: error: cannot find symbol
            codecs.put(BEST_COMPRESSION_CODEC, new Lucene94Codec(Mode.BEST_COMPRESSION));
                                                   ^
  symbol:   class Lucene94Codec
  location: class CodecService
> Task :server:compileJava
/home/runner/work/OpenSearch/OpenSearch/server/src/main/java/org/opensearch/index/codec/CodecService.java:66: error: cannot find symbol
            codecs.put(BEST_COMPRESSION_CODEC, new Lucene94Codec(Mode.BEST_COMPRESSION));
                                                                 ^
  symbol:   variable Mode
  location: class CodecService
/home/runner/work/OpenSearch/OpenSearch/server/src/main/java/org/opensearch/index/codec/CodecService.java:68: error: cannot find symbol
            codecs.put(DEFAULT_CODEC, new PerFieldMappingPostingFormatCodec(Mode.BEST_SPEED, mapperService, logger));
                                                                            ^
  symbol:   variable Mode
  location: class CodecService
/home/runner/work/OpenSearch/OpenSearch/server/src/main/java/org/opensearch/index/codec/CodecService.java:68: error: incompatible types: PerFieldMappingPostingFormatCodec cannot be converted to Codec
            codecs.put(DEFAULT_CODEC, new PerFieldMappingPostingFormatCodec(Mode.BEST_SPEED, mapperService, logger));
                                      ^
/home/runner/work/OpenSearch/OpenSearch/server/src/main/java/org/opensearch/index/codec/CodecService.java:69: error: cannot find symbol
            codecs.put(BEST_COMPRESSION_CODEC, new PerFieldMappingPostingFormatCodec(Mode.BEST_COMPRESSION, mapperService, logger));
                                                                                     ^
  symbol:   variable Mode
  location: class CodecService
/home/runner/work/OpenSearch/OpenSearch/server/src/main/java/org/opensearch/index/codec/CodecService.java:69: error: incompatible types: PerFieldMappingPostingFormatCodec cannot be converted to Codec
            codecs.put(BEST_COMPRESSION_CODEC, new PerFieldMappingPostingFormatCodec(Mode.BEST_COMPRESSION, mapperService, logger));

Signed-off-by: Suraj Singh <surajrider@gmail.com>
@reta
Copy link
Collaborator

reta commented Dec 14, 2022

Thank you @reta for the quick update. Is this needed for gradle check to pass ? I see lot of java compilation errors in precommit.

No, it is not needed to check to pass but it was one of the reasons to update. Also, the compilation issues are "normal" due to changes in Apache Lucene.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@mch2
Copy link
Member

mch2 commented Dec 14, 2022

@dreamer-89 From what I can tell most of these changes are around a new codec used with 9.5, with 9.4 moved to bwc codecs.

Signed-off-by: Suraj Singh <surajrider@gmail.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@dreamer-89 dreamer-89 merged commit 0475d1c into opensearch-project:main Dec 14, 2022
ryanbogan pushed a commit that referenced this pull request Dec 14, 2022
* Added bwc version 2.4.2

Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>

* Added 2.4.2.

Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>

* Update Lucene snapshot to 9.5.0-snapshot-d5cef1c

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update changelog entry

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add 2.4.2 bwc version

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Internal changes post lucene upgrade

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>
Signed-off-by: Suraj Singh <surajrider@gmail.com>
Co-authored-by: opensearch-ci-bot <opensearch-ci-bot@users.noreply.github.com>
Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>
saratvemulapalli pushed a commit that referenced this pull request Dec 15, 2022
…ature/identity (#5581)

* Fix flaky ShardIndexingPressureConcurrentExecutionTests (#5439)

Add conditional check on assertNull to fix flaky tests.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Fix bwc for cluster manager throttling settings (#5305)

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

* Update ingest-attachment plugin dependencies: Apache Tika 3.6.0, Apache Mime4j 0.8.8, Apache Poi 5.2.3, Apache PdfBox 2.0.27 (#5448)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Enhance CheckpointState to support no-op replication (#5282)

* CheckpointState enhanced to support no-op replication

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Co-authored-by: Bukhtawar Khan<bukhtawa@amazon.com>

* [BUG] org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT/test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials} flaky: randomizing basePath (#5482)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* [Bug] fix case sensitivity for wildcard queries (#5462)

Fixes the wildcard query to not normalize the pattern when case_insensitive is
set by the user. This is achieved by creating a new normalizedWildcardQuery
method so that query_string queries (which do not support case sensitivity) can
still normalize the pattern when the default analyzer is used; maintaining
existing behavior.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* Support OpenSSL Provider with default Netty allocator (#5460)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Revert "build no-jdk distributions as part of release build (#4902)" (#5465)

This reverts commit 8c9ca4e.

It seems that this wasn't entirely the correct way and is currently
blocking us from removing the `build.sh` from the `opensearch-build`
repository (i.e. this `build.sh` here is not yet being used).
See the discussion in opensearch-project/opensearch-build#2835 for
further details.

Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>

Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>

* Add max_shard_size parameter for Shrink API (fix supported version after backport) (#5503)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Sync CODEOWNERS with MAINTAINERS. (#5501)

Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>

Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>

* Added jackson dependency to server (#5366)

* Added jackson dependency to server

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Updated CHANGELOG

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Update build.gradle files

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Add RuntimePermission to fix errors

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Fix flaky test BulkIntegrationIT.testDeleteIndexWhileIndexing (#5491)

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* Add release notes for 2.4.1 (#5488)

Signed-off-by: Xue Zhou <xuezhou@amazon.com>

Signed-off-by: Xue Zhou <xuezhou@amazon.com>

* Properly skip OnDemandBlockSnapshotIndexInputTests.testVariousBlockSize on Windows. (#5511)

PR #5397 skipped this test in @before block but still
frequently throws a TestCouldNotBeSkippedException.  This is caused by the after block still executing and throwing  an exception
while cleaning the directory created at the path in @before.  Moving the assumption to the individual test prevents this exception by ensuring the path exists.

Signed-off-by: Marc Handalian <handalm@amazon.com>

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Merge first batch of feature/extensions into main (#5347)

* Merge first batch of feature/extensions into main

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Fixed CHANGELOG

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Fixed newline errors

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Renaming and CHANGELOG fixes

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Refactor extension loading into private method

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Removed skipValidation and added connectToExtensionNode method

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Remove unnecessary feature flag calls

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Renaming and exception handling

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Change latches to CompletableFuture

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Removed unnecessary validateSettingKey call

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Fix azure-core dependency

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Update SHAs

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Remove unintended dependency changes

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Removed dynamic settings regitration, removed info() method, and added NoopExtensionsManager

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Add javadoc

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Fixed spotless failure

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Removed NoopExtensionsManager

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Added functioning NoopExtensionsManager

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Added missing javadoc

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Remove forbiddenAPI

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Fix spotless

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Change logger.info to logger.error in handleException

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Fix ExtensionsManagerTests

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Removing unrelated change

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Update SHAs

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Bump commons-compress from 1.21 to 1.22 (#5520)

Bumps commons-compress from 1.21 to 1.22.

---
updated-dependencies:
- dependency-name: org.apache.commons:commons-compress
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [Segment Replication] Trigger a round of replication for replica shards during peer recovery when segment replication is enabled (#5332)

* Fix new added replica shards falling behind primary.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Trigger a round of replication during peer recovery when segment replication is enabled.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Remove unnecessary start replication overloaded method.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Add test for failure case and refactor some code.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Apply spotless check.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Addressing comments on the PR.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Remove unnecessary condition check.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Apply spotless check.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Add step listeners to resolve forcing round of segment replication.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Adding support to register settings dynamically (#5495)

* Adding support to register settings dynamically

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Update CHANGELOG

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Removed unnecessary registerSetting methods

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Change setting registration order

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Add unregisterSettings method

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Remove unnecessary feature flag

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Updated 1.3.7 release notes date (#5536)

Signed-off-by: owaiskazi19 <owaiskazi19@gmail.com>

Signed-off-by: owaiskazi19 <owaiskazi19@gmail.com>

* Pre conditions check before updating weighted routing metadata (#4955)

* Pre conditions check to allow weight updates for non decommissioned attribute

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

* Atomically update cluster state with decommission status and corresponding action (#5093)

* Atomically update the cluster state with decommission status and its corresponding action in the same execute call

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

* Update Netty to 4.1.86.Final (#5529)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Update release date in 2.4.1 release notes (#5549)

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update 2.4.1 release notes (#5552)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Refactor fuzziness interface on query builders (#5433)

* Refactor Object to Fuzziness type for all query builders

Signed-off-by: noCharger <lingzhichu.clz@gmail.com>

* Revise on bwc

Signed-off-by: noCharger <lingzhichu.clz@gmail.com>

* Update change log

Signed-off-by: noCharger <lingzhichu.clz@gmail.com>

Signed-off-by: noCharger <lingzhichu.clz@gmail.com>
Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>

* Upgrade lucene version (#5570)

* Added bwc version 2.4.2

Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>

* Added 2.4.2.

Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>

* Update Lucene snapshot to 9.5.0-snapshot-d5cef1c

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update changelog entry

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add 2.4.2 bwc version

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Internal changes post lucene upgrade

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>
Signed-off-by: Suraj Singh <surajrider@gmail.com>
Co-authored-by: opensearch-ci-bot <opensearch-ci-bot@users.noreply.github.com>
Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>

* Add CI bundle pattern to distribution download (#5348)

* Add CI bundle pattern for ivy repo

Signed-off-by: Zelin Hao <zelinhao@amazon.com>

* Gradle update

Signed-off-by: Zelin Hao <zelinhao@amazon.com>

* Extract path

Signed-off-by: Zelin Hao <zelinhao@amazon.com>

* Change with customDistributionDownloadType

Signed-off-by: Zelin Hao <zelinhao@amazon.com>

* Add default for exception handle

Signed-off-by: Zelin Hao <zelinhao@amazon.com>

* Add documentations

Signed-off-by: Zelin Hao <zelinhao@amazon.com>

Signed-off-by: Zelin Hao <zelinhao@amazon.com>

* Bump protobuf-java from 3.21.9 to 3.21.11 in /plugins/repository-hdfs (#5519)

* Bump protobuf-java from 3.21.9 to 3.21.11 in /plugins/repository-hdfs

Bumps [protobuf-java](https://github.com/protocolbuffers/protobuf) from 3.21.9 to 3.21.11.
- [Release notes](https://github.com/protocolbuffers/protobuf/releases)
- [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/generate_changelog.py)
- [Commits](protocolbuffers/protobuf@v3.21.9...v3.21.11)

---
updated-dependencies:
- dependency-name: com.google.protobuf:protobuf-java
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

* Updated changelog

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Co-authored-by: Owais Kazi <owaiskazi19@gmail.com>
Co-authored-by: Suraj Singh <surajrider@gmail.com>

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>
Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>
Signed-off-by: Ryan Bogan <rbogan@amazon.com>
Signed-off-by: Poojita Raj <poojiraj@amazon.com>
Signed-off-by: Xue Zhou <xuezhou@amazon.com>
Signed-off-by: Marc Handalian <handalm@amazon.com>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: owaiskazi19 <owaiskazi19@gmail.com>
Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: noCharger <lingzhichu.clz@gmail.com>
Signed-off-by: Zelin Hao <zelinhao@amazon.com>
Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
Co-authored-by: Rishikesh Pasham <62345295+Rishikesh1159@users.noreply.github.com>
Co-authored-by: Dhwanil Patel <dhwanip@amazon.com>
Co-authored-by: Andriy Redko <andriy.redko@aiven.io>
Co-authored-by: Ashish <ssashish@amazon.com>
Co-authored-by: Nick Knize <nknize@apache.org>
Co-authored-by: Ralph Ursprung <39383228+rursprung@users.noreply.github.com>
Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>
Co-authored-by: Ryan Bogan <10944539+ryanbogan@users.noreply.github.com>
Co-authored-by: Poojita Raj <poojiraj@amazon.com>
Co-authored-by: Xue Zhou <85715413+xuezhou25@users.noreply.github.com>
Co-authored-by: Marc Handalian <handalm@amazon.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Owais Kazi <owaiskazi19@gmail.com>
Co-authored-by: Rishab Nahata <rnnahata@amazon.com>
Co-authored-by: Suraj Singh <surajrider@gmail.com>
Co-authored-by: Louis Chu <lingzhichu.clz@gmail.com>
Co-authored-by: opensearch-ci-bot <opensearch-ci-bot@users.noreply.github.com>
Co-authored-by: Zelin Hao <87548827+zelinh@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants