From 7645653b467b6b837ad39f02d476753e07fe6018 Mon Sep 17 00:00:00 2001 From: Brian Flores Date: Thu, 10 Oct 2024 12:21:24 -0700 Subject: [PATCH 1/4] Adds documentation to help fix local Lucene code errors Signed-off-by: Brian Flores --- DEVELOPER_GUIDE.md | 37 ++++++++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index d59381a45..471ef457e 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -88,12 +88,22 @@ Please follow these formatting guidelines: OpenSearch neural-search uses a [Gradle](https://docs.gradle.org/6.6.1/userguide/userguide.html) wrapper for its build. Run `gradlew` on Unix systems. -Build OpenSearch neural-search using `gradlew build` +Build OpenSearch neural-search using `gradlew build`. This command will +also run Integration Tests and Unit Tests. ``` ./gradlew build ``` +## Run Unit Tests +If you want to strictly test that your unit tests are passing +you can run the following. + +``` +./gradlew test +``` + + ## Run OpenSearch neural-search ### Run Single-node Cluster Locally @@ -292,3 +302,28 @@ original PR with an appropriate label `backport ` is merge run successfully on the PR. For example, if a PR on main needs to be backported to `2.x` branch, add a label `backport 2.x` to the PR and make sure the backport workflow runs on the PR along with other checks. Once this PR is merged to main, the workflow will create a backport PR to the `2.x` branch. + +## Building on Lucene Version updates +There may be a Lucene version update that can affect your workflow causing errors like +`java.lang.NoClassDefFoundError: org/apache/lucene/codecs/lucene99/Lucene99Codec`. +This results in having issues with not being able to do `./gradlew run` or `./gradlew build`. + +You can check this [PR](https://github.com/opensearch-project/k-NN/pull/2195) as an example of this event happening + +**Follow the steps to remedy the gradle run issue.** +1. From your cloned neural search repo root directory `rm -rf build .gradle` +2. Clear the following directories from your gradle folder located in your root directory + 1. `cd ~/.gradle` + 2. `rm -rf caches workers wrapper daemon` + 3. `cd -` switch back the previous directory (i.e. the neural search repo root directory) +3. Finally run `./gradlew run` + +**Follow the steps to remedy the gradle build issue** + +**PREREQ:** Make sure you have OpenSearch repo cloned locally + +1. From your cloned neural search repo root directory `rm -rf build .gradle` +2. Delete the .gradle folder and .m2 folder. `rm -rf ~/.gradle ~/.m2` +3. Head over to your OpenSearch cloned repo root directory + 1. `./gradlew publisToMavenLocal` +4. Finally run `./gradlew build` from the neural search repo From 9ed2f63908a3a9d3d1b1dcedc93792c8c001f1d9 Mon Sep 17 00:00:00 2001 From: Brian Flores Date: Fri, 11 Oct 2024 11:27:20 -0700 Subject: [PATCH 2/4] Update dev guide to be more specific on KNN dependency issue Signed-off-by: Brian Flores --- DEVELOPER_GUIDE.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 471ef457e..01cabf7d9 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -305,10 +305,13 @@ merged to main, the workflow will create a backport PR to the `2.x` branch. ## Building on Lucene Version updates There may be a Lucene version update that can affect your workflow causing errors like -`java.lang.NoClassDefFoundError: org/apache/lucene/codecs/lucene99/Lucene99Codec`. +`java.lang.NoClassDefFoundError: org/apache/lucene/codecs/lucene99/Lucene99Codec` or +`Provider org.opensearch.knn.index.codec.KNN910Codec.KNN910Codec could not be instantiated`. In this case +we can observe there may be an issue with a dependency with [KNN](https://github.com/opensearch-project/k-NN). This results in having issues with not being able to do `./gradlew run` or `./gradlew build`. -You can check this [PR](https://github.com/opensearch-project/k-NN/pull/2195) as an example of this event happening +You can check this [KNN PR](https://github.com/opensearch-project/k-NN/pull/2195) as an example of this event happening or this [Neural Search PR](https://github.com/opensearch-project/neural-search/pull/913#issuecomment-2400189329) that shows a developer going +through the same build issue. **Follow the steps to remedy the gradle run issue.** 1. From your cloned neural search repo root directory `rm -rf build .gradle` From b6624baca596d58d03c901a16c1a8dc63934f42b Mon Sep 17 00:00:00 2001 From: Brian Flores Date: Fri, 11 Oct 2024 14:28:07 -0700 Subject: [PATCH 3/4] Added lucene doc help to table of contents Signed-off-by: Brian Flores --- DEVELOPER_GUIDE.md | 1 + 1 file changed, 1 insertion(+) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 01cabf7d9..0ac3f05a4 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -15,6 +15,7 @@ - [Adding new tests](#adding-new-tests) - [Supported configurations](#supported-configurations) - [Submitting Changes](#submitting-changes) + - [Building on Lucene Version updates](#building-on-lucene-version-updates) # Developer Guide From a4445a36003b6963112539b24cffd90b19d034b3 Mon Sep 17 00:00:00 2001 From: Brian Flores Date: Fri, 11 Oct 2024 15:10:56 -0700 Subject: [PATCH 4/4] Added Major Dependencies Info under Debuging Signed-off-by: Brian Flores --- DEVELOPER_GUIDE.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 0ac3f05a4..37d78f9b1 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -11,11 +11,12 @@ - [Run Single-node Cluster Locally](#run-single-node-cluster-locally) - [Run Multi-node Cluster Locally](#run-multi-node-cluster-locally) - [Debugging](#debugging) + - [Major Dependencies](#major-dependencies) - [Backwards Compatibility Testing](#backwards-compatibility-testing) - [Adding new tests](#adding-new-tests) - [Supported configurations](#supported-configurations) - [Submitting Changes](#submitting-changes) - - [Building on Lucene Version updates](#building-on-lucene-version-updates) + - [Building On Lucene Version Updates](#building-on-lucene-version-updates) # Developer Guide @@ -238,6 +239,12 @@ Additionally, it is possible to attach one debugger to the cluster JVM and anoth ./gradlew :integTest -Dtest.debug=1 -Dcluster.debug=1 ``` +#### Major Dependencies +Currently, the major dependencies that Neural Search depends on are [ML-Commons](https://github.com/opensearch-project/ml-commons) and [K-NN](https://github.com/opensearch-project/k-NN). +Make sure to check on them when you observe a failure that affects Neural Search. +See [Building on Lucene Version updates](#building-on-lucene-version-updates) as an example where K-NN caused a build failure. +Also, please note that it may take time for developers to create a fix for your current dependency issue. + ## Backwards Compatibility Testing The purpose of Backwards Compatibility Testing and different types of BWC tests are explained [here](https://github.com/opensearch-project/opensearch-plugins/blob/main/TESTING.md#backwards-compatibility-testing). The BWC tests (i.e. Restart-Upgrade, Mixed-Cluster and Rolling-Upgrade scenarios) should be added with any new feature being added to Neural Search. @@ -304,14 +311,14 @@ run successfully on the PR. For example, if a PR on main needs to be backported `backport 2.x` to the PR and make sure the backport workflow runs on the PR along with other checks. Once this PR is merged to main, the workflow will create a backport PR to the `2.x` branch. -## Building on Lucene Version updates +## Building On Lucene Version Updates There may be a Lucene version update that can affect your workflow causing errors like `java.lang.NoClassDefFoundError: org/apache/lucene/codecs/lucene99/Lucene99Codec` or `Provider org.opensearch.knn.index.codec.KNN910Codec.KNN910Codec could not be instantiated`. In this case -we can observe there may be an issue with a dependency with [KNN](https://github.com/opensearch-project/k-NN). +we can observe there may be an issue with a dependency with [K-NN](https://github.com/opensearch-project/k-NN). This results in having issues with not being able to do `./gradlew run` or `./gradlew build`. -You can check this [KNN PR](https://github.com/opensearch-project/k-NN/pull/2195) as an example of this event happening or this [Neural Search PR](https://github.com/opensearch-project/neural-search/pull/913#issuecomment-2400189329) that shows a developer going +You can check this [K-NN PR](https://github.com/opensearch-project/k-NN/pull/2195) as an example of this event happening or this [Neural Search PR](https://github.com/opensearch-project/neural-search/pull/913#issuecomment-2400189329) that shows a developer going through the same build issue. **Follow the steps to remedy the gradle run issue.**