[Upgrade] Lucene 9.0.0 release #1109

nknize · 2021-08-18T17:12:20Z

This PR upgrades the core codebase from Lucene 8.10.1 to lucene-9.0.0 release. It includes all necessary refactoring of features and API changes when upgrading to the new major Lucene release.

relates #1067
depends on #2452

nknize · 2021-08-18T17:13:18Z

TODO:

remove all deprecated features from Legacy version 6.x (will open a meta issue to track)

opensearch-ci-bot · 2021-08-18T17:14:20Z

✅ Gradle Wrapper Validation success 52d7adf3cf035003e921e35de7deb72c03c6b9bc

opensearch-ci-bot · 2021-08-18T17:14:52Z

✅ DCO Check Passed 52d7adf3cf035003e921e35de7deb72c03c6b9bc

opensearch-ci-bot · 2021-08-18T17:20:50Z

❌ Gradle Precommit failure 52d7adf3cf035003e921e35de7deb72c03c6b9bc
Log 960

opensearch-ci-bot · 2021-09-29T21:21:19Z

Can one of the admins verify this patch?

opensearch-ci-bot · 2021-10-26T15:04:39Z

✅ Gradle Wrapper Validation success 52d7adf3cf035003e921e35de7deb72c03c6b9bc

opensearch-ci-bot · 2021-10-26T15:05:13Z

✅ DCO Check Passed 52d7adf3cf035003e921e35de7deb72c03c6b9bc

opensearch-ci-bot · 2021-10-26T15:32:25Z

❌ Gradle Precommit failure 52d7adf3cf035003e921e35de7deb72c03c6b9bc
Log 1411

opensearch-ci-bot · 2021-12-01T17:38:32Z

✅ Gradle Wrapper Validation success c40161ecbcc40b9ed213028438e42f5c91d38113

opensearch-ci-bot · 2021-12-01T17:44:22Z

❌ Gradle Check failure c40161ecbcc40b9ed213028438e42f5c91d38113
Log 1280

Reports 1280

opensearch-ci-bot · 2021-12-01T17:46:23Z

❌ Gradle Precommit failure c40161ecbcc40b9ed213028438e42f5c91d38113
Log 1669

opensearch-ci-bot · 2021-12-01T20:16:00Z

✅ Gradle Wrapper Validation success 9a897b4ff762b4e92ba38b9baa33854f0414fda2

opensearch-ci-bot · 2021-12-01T20:17:16Z

❌ Gradle Precommit failure 9a897b4ff762b4e92ba38b9baa33854f0414fda2
Log 1677

opensearch-ci-bot · 2021-12-01T20:18:21Z

❌ Gradle Check failure 9a897b4ff762b4e92ba38b9baa33854f0414fda2
Log 1288

Reports 1288

opensearch-ci-bot · 2021-12-02T20:54:25Z

✅ Gradle Wrapper Validation success f94754a4c79c8b01a12a30ea1b5142971b9d887b

opensearch-ci-bot · 2021-12-02T20:58:24Z

❌ Gradle Check failure f94754a4c79c8b01a12a30ea1b5142971b9d887b
Log 1304

Reports 1304

opensearch-ci-bot · 2021-12-02T21:02:16Z

❌ Gradle Precommit failure f94754a4c79c8b01a12a30ea1b5142971b9d887b
Log 1685

opensearch-ci-bot · 2021-12-02T21:08:03Z

✅ Gradle Wrapper Validation success 65dae2e281d749ccb38e1f8aaf5f970b242a37c5

opensearch-ci-bot · 2021-12-02T21:08:55Z

❌ Gradle Precommit failure 65dae2e281d749ccb38e1f8aaf5f970b242a37c5
Log 1686

opensearch-ci-bot · 2021-12-02T21:13:38Z

❌ Gradle Check failure 65dae2e281d749ccb38e1f8aaf5f970b242a37c5
Log 1305

Reports 1305

dblock

I did a read, but it's a bit over my head given I know little about Lucene. I think we should merge it as is (feel free to update the small stuff before/after) and iterate for anything that falls out.

server/src/main/java/org/apache/lucene/util/SPIClassIterator.java

server/src/main/java/org/apache/lucene/util/packed/XPacked64.java

dblock · 2022-03-15T15:33:41Z

server/src/main/java/org/opensearch/action/admin/indices/segments/IndicesSegmentResponse.java

@@ -154,13 +154,6 @@ protected void addCustomXContentFields(XContentBuilder builder, Params params) t
                        if (segment.getSegmentSort() != null) {
                            toXContent(builder, segment.getSegmentSort());
                        }
-                        if (segment.ramTree != null) {


ramEstimator was removed from Lucene 9.. so we remove it from the segment stats.

reta · 2022-03-15T18:51:01Z

@nknize sorry, DCO check failed for some reasons :( could you please amend the last commit? thank you

opensearch-ci-bot · 2022-03-15T19:12:38Z

❌ Gradle Check failure 636db0caafa49a4b9317831b33795cce149696ea
Log 3414

Reports 3414

Removes types from UID class along with cleaning up all obsolete MapperService dependendices. This includes removing UID from the Translog Delete operation which is no longer needed due to type dependency removal. Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

This commit upgrades the core codebase from Lucene 8.9 to lucene-9.0.0. It includes all necessary refactoring of features and API changes when upgrading to a new major Lucene release. Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

This reverts commit e7af399. Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

opensearch-ci-bot · 2022-03-15T20:08:05Z

✅ Gradle Check success 0add86fc1c87a2876266c6b63e5e7be2f7197b95
Log 3415

Reports 3415

opensearch-ci-bot · 2022-03-15T20:24:49Z

✅ Gradle Check success f080199
Log 3416

Reports 3416

opensearch-ci-bot · 2022-03-15T20:31:51Z

✅ Gradle Check success 451a409
Log 3417

Reports 3417

`OpenSearch` 2.0 uses newer version of `Lucene` (9.0) which doesn't provide segments stats info. Ref: opensearch-project/OpenSearch#2029 opensearch-project/OpenSearch#1109 See also history for `server/src/main/java/org/opensearch/index/engine/SegmentsStats.java` in `OpenSearch` repo. Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Update nuget packages (opensearch-project#88) * Cleanup - removing stale dependencies. * Fix 2 typos in scripts project * Update `scripts` project file to include all relevant objects. * Rename `master` node role to `cluster_manager` as it was done in OpenSearch. Ref: opensearch-project/OpenSearch#2480 * Remove validation for indices segments stats. `OpenSearch` 2.0 uses newer version of `Lucene` (9.0) which doesn't provide segments stats info. Ref: opensearch-project/OpenSearch#2029 opensearch-project/OpenSearch#1109 See also history for `server/src/main/java/org/opensearch/index/engine/SegmentsStats.java` in `OpenSearch` repo. * Remove tests for `_type` validation in mapping APIs as it was removed from `OpenSearch`. Ref: opensearch-project/OpenSearch#2238 opensearch-project/OpenSearch#2480 * Remove usage of deprecated `search.remote` settings. Ref: opensearch-project/OpenSearch#1870 * Update abstractions package - patch to support OpenSearch 2.0. Update integration workflow to run tests on OpenSearch 2.0. * Rename `master_timeout` to `cluster_manager_timeout` in all APIs where it is used. * Enrich comments to already renamed `CatMaster`/`CatClusterManager` API. * Rename in `/_cluster/stats`/`cluster.stats` and `/_cluster/state`/`cluster.state`. * Add deprecation info. * Rename in comments. * Rename in test data. * Renamings in tests including `MasterEligible`, but mark it is obsolete. * Rename branch reference in scripting. * Mark `indices.exists_type`/`TypeExists` APIs as deprecated. * Update compatibility matrix and include it into `sln` file. * Add deprecation notice to all reference of `include_type_name`/`IncludeTypeName`. * Update compatibility matrix. * Remove `OpenDistro` compatibility notice. * Update repo link. * Add small README for each project being released. * Address PR opensearch-project#51 feedback. Signed-off-by: Yury-Fridlyand <yuryf@bitquilltech.com>

* Update nuget packages (#88) * Cleanup - removing stale dependencies. * Fix 2 typos in scripts project * Update `scripts` project file to include all relevant objects. * Rename `master` node role to `cluster_manager` as it was done in OpenSearch. Ref: opensearch-project/OpenSearch#2480 * Remove validation for indices segments stats. `OpenSearch` 2.0 uses newer version of `Lucene` (9.0) which doesn't provide segments stats info. Ref: opensearch-project/OpenSearch#2029 opensearch-project/OpenSearch#1109 See also history for `server/src/main/java/org/opensearch/index/engine/SegmentsStats.java` in `OpenSearch` repo. * Remove tests for `_type` validation in mapping APIs as it was removed from `OpenSearch`. Ref: opensearch-project/OpenSearch#2238 opensearch-project/OpenSearch#2480 * Remove usage of deprecated `search.remote` settings. Ref: opensearch-project/OpenSearch#1870 * Update abstractions package - patch to support OpenSearch 2.0. Update integration workflow to run tests on OpenSearch 2.0. * Rename `master_timeout` to `cluster_manager_timeout` in all APIs where it is used. * Enrich comments to already renamed `CatMaster`/`CatClusterManager` API. * Rename in `/_cluster/stats`/`cluster.stats` and `/_cluster/state`/`cluster.state`. * Add deprecation info. * Rename in comments. * Rename in test data. * Renamings in tests including `MasterEligible`, but mark it is obsolete. * Rename branch reference in scripting. * Mark `indices.exists_type`/`TypeExists` APIs as deprecated. * Update compatibility matrix and include it into `sln` file. * Add deprecation notice to all reference of `include_type_name`/`IncludeTypeName`. * Update compatibility matrix. * Remove `OpenDistro` compatibility notice. * Update repo link. * Add small README for each project being released. * Address PR #51 feedback. Signed-off-by: Yury-Fridlyand <yuryf@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yuryf@bitquilltech.com>

nknize added v2.0.0 Version 2.0.0 WIP Work in progress >upgrade Label used when upgrading library dependencies (e.g., Lucene) labels Aug 18, 2021

nknize marked this pull request as draft August 18, 2021 17:12

nknize force-pushed the upgrade/lucene9 branch from 52d7adf to c40161e Compare December 1, 2021 17:36

nknize force-pushed the upgrade/lucene9 branch from f94754a to 65dae2e Compare December 2, 2021 21:06

rishabhmaurya mentioned this pull request Jan 5, 2022

Opensearch Modularization #1838

Open

7 tasks

nknize force-pushed the upgrade/lucene9 branch from 65dae2e to f55d91b Compare January 10, 2022 21:02

nknize changed the title ~~[Upgrade] Lucene 9.0.0-snapshot-803d131fd08~~ [Upgrade] Lucene 9.0.0 release Jan 10, 2022

nknize marked this pull request as ready for review January 10, 2022 21:05

nknize removed the WIP Work in progress label Mar 14, 2022

dblock approved these changes Mar 15, 2022

View reviewed changes

nknize force-pushed the upgrade/lucene9 branch from 636db0c to 0add86f Compare March 15, 2022 19:20

nknize and others added 9 commits March 15, 2022 14:21

fix failing test

56d9324

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

[Upgrade] Lucene 9.0.0

66ea3d7

This commit upgrades the core codebase from Lucene 8.9 to lucene-9.0.0. It includes all necessary refactoring of features and API changes when upgrading to a new major Lucene release. Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

Fixing JDK-11 and HighlighterSearchIT::testPostingsHighlighter

f8e21b5

Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

More highlighter-related test fixes (#2)

360c7d2

Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

cleanup and test fixes

9488339

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

remove AwaitsFix from CustomUnifiedHighlighterTests

1388897

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

PluginsServiceTests is passing (#3)

e7af399

Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

merge removals

d35292a

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

nknize force-pushed the upgrade/lucene9 branch from 0add86f to aeb9c21 Compare March 15, 2022 19:32

Revert "PluginsServiceTests is passing (#3)"

f080199

This reverts commit e7af399. Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

nknize force-pushed the upgrade/lucene9 branch from aeb9c21 to f080199 Compare March 15, 2022 19:33

add spdx header to lucene copied files

451a409

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

nknize merged commit 006c832 into opensearch-project:main Mar 15, 2022

penghuo mentioned this pull request Mar 15, 2022

OpenSearch compatible test with Lucene 9 opensearch-project/sql#498

Closed

nknize mentioned this pull request Mar 21, 2022

[Meta] Upgrade Lucene #1065

Closed

2 tasks

VachaShah mentioned this pull request Apr 13, 2022

Jenkins Randomized Gradle Check Failed on 9383aff6e573011e4de4a7a5b5279722bb5d6082 #2216

Open

dbwiddis mentioned this pull request Jul 26, 2022

Update Japanese and Korean language analyzers #3950

Closed

dbwiddis mentioned this pull request Aug 24, 2024

[BUG] Couple of cases of infinite recursion #15387

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Upgrade] Lucene 9.0.0 release #1109

[Upgrade] Lucene 9.0.0 release #1109

nknize commented Aug 18, 2021 •

edited

Loading

nknize commented Aug 18, 2021

opensearch-ci-bot commented Aug 18, 2021

opensearch-ci-bot commented Aug 18, 2021

opensearch-ci-bot commented Aug 18, 2021

opensearch-ci-bot commented Sep 29, 2021

opensearch-ci-bot commented Oct 26, 2021

opensearch-ci-bot commented Oct 26, 2021

opensearch-ci-bot commented Oct 26, 2021

opensearch-ci-bot commented Dec 1, 2021

opensearch-ci-bot commented Dec 1, 2021

opensearch-ci-bot commented Dec 1, 2021

opensearch-ci-bot commented Dec 1, 2021

opensearch-ci-bot commented Dec 1, 2021

opensearch-ci-bot commented Dec 1, 2021

opensearch-ci-bot commented Dec 2, 2021

opensearch-ci-bot commented Dec 2, 2021

opensearch-ci-bot commented Dec 2, 2021

opensearch-ci-bot commented Dec 2, 2021

opensearch-ci-bot commented Dec 2, 2021

opensearch-ci-bot commented Dec 2, 2021

dblock left a comment

dblock Mar 15, 2022

nknize Mar 15, 2022

reta commented Mar 15, 2022

opensearch-ci-bot commented Mar 15, 2022

opensearch-ci-bot commented Mar 15, 2022

opensearch-ci-bot commented Mar 15, 2022

opensearch-ci-bot commented Mar 15, 2022

[Upgrade] Lucene 9.0.0 release #1109

[Upgrade] Lucene 9.0.0 release #1109

Conversation

nknize commented Aug 18, 2021 • edited Loading

nknize commented Aug 18, 2021

opensearch-ci-bot commented Aug 18, 2021

opensearch-ci-bot commented Aug 18, 2021

opensearch-ci-bot commented Aug 18, 2021

opensearch-ci-bot commented Sep 29, 2021

opensearch-ci-bot commented Oct 26, 2021

opensearch-ci-bot commented Oct 26, 2021

opensearch-ci-bot commented Oct 26, 2021

opensearch-ci-bot commented Dec 1, 2021

opensearch-ci-bot commented Dec 1, 2021

opensearch-ci-bot commented Dec 1, 2021

opensearch-ci-bot commented Dec 1, 2021

opensearch-ci-bot commented Dec 1, 2021

opensearch-ci-bot commented Dec 1, 2021

opensearch-ci-bot commented Dec 2, 2021

opensearch-ci-bot commented Dec 2, 2021

opensearch-ci-bot commented Dec 2, 2021

opensearch-ci-bot commented Dec 2, 2021

opensearch-ci-bot commented Dec 2, 2021

opensearch-ci-bot commented Dec 2, 2021

dblock left a comment

Choose a reason for hiding this comment

dblock Mar 15, 2022

Choose a reason for hiding this comment

nknize Mar 15, 2022

Choose a reason for hiding this comment

reta commented Mar 15, 2022

opensearch-ci-bot commented Mar 15, 2022

opensearch-ci-bot commented Mar 15, 2022

opensearch-ci-bot commented Mar 15, 2022

opensearch-ci-bot commented Mar 15, 2022

nknize commented Aug 18, 2021 •

edited

Loading