[Refactor] Various Utilities, Assertion, and Concurrency Exception from server to libraries #6875

nknize · 2023-03-29T19:58:41Z

To further reduce the surface area of utility methods in :server: this commit refactors the following:

MapBuilder from server module to common library
Assertions from server module to core library
BytesRef methods from CollectionUtils in server module to new BytesRefUtils in core library

It also removes CollectionUtils dependency on hppc in prep for refactoring CollectionUtils methods to the proper library.

related #5910

github-actions · 2023-03-29T20:33:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.snapshots.SearchableSnapshotIT.testCacheFilesAreClosedAfterUse
      1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.split/30_copy_settings/Copy settings during split index}
      1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.shrink/30_copy_settings/Copy settings during shrink index}

URL: https://build.ci.opensearch.org/job/gradle-check/13043/
CommitID: 61f473a
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov-commenter · 2023-03-29T20:39:20Z

Codecov Report

Merging #6875 (5881606) into main (7b708c4) will increase coverage by 0.17%.
The diff coverage is 84.21%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #6875      +/-   ##
============================================
+ Coverage     70.52%   70.70%   +0.17%     
- Complexity    59112    59235     +123     
============================================
  Files          4814     4812       -2     
  Lines        283743   283717      -26     
  Branches      40915    40915              
============================================
+ Hits         200110   200594     +484     
+ Misses        67154    66615     -539     
- Partials      16479    16508      +29

Impacted Files	Coverage Δ
...java/org/opensearch/common/collect/MapBuilder.java	`66.66% <ø> (ø)`
.../src/main/java/org/opensearch/core/Assertions.java	`75.00% <ø> (ø)`
...currency/OpenSearchRejectedExecutionException.java	`100.00% <ø> (ø)`
...ch/index/reindex/remote/RemoteResponseParsers.java	`90.75% <ø> (ø)`
...src/main/java/org/opensearch/ExceptionsHelper.java	`79.14% <ø> (-3.07%)`	⬇️
...rg/opensearch/action/bulk/TransportBulkAction.java	`77.85% <ø> (ø)`
...org/opensearch/action/support/RetryableAction.java	`85.89% <ø> (ø)`
...tion/support/replication/ReplicationOperation.java	`77.63% <ø> (+3.10%)`	⬆️
...upport/replication/TransportReplicationAction.java	`77.45% <ø> (+1.07%)`	⬆️
...opensearch/cluster/InternalClusterInfoService.java	`75.43% <ø> (-0.87%)`	⬇️
... and 46 more

... and 495 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

github-actions · 2023-03-29T22:50:39Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.split/30_copy_settings/Copy settings during split index}

URL: https://build.ci.opensearch.org/job/gradle-check/13060/
CommitID: 5881606
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-03-30T04:15:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/13079/
CommitID: f674d09
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

nknize · 2023-03-30T04:39:54Z

oye.. bouncycastle FIPS WhiteSource failure bonanza continues... :/

github-actions · 2023-03-30T04:46:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/13080/
CommitID: 4e811c7

reta · 2023-03-30T14:22:40Z

libs/core/build.gradle

@@ -79,6 +79,9 @@ dependencies {

 api "com.fasterxml.jackson.core:jackson-core:${versions.jackson}"

+ // lucene
+ api "org.apache.lucene:lucene-core:${versions.lucene}"


@nknize I have doubts we need to depend on Lucene in any core / common modules, the BytesRefUtils is very specialized utility class, why do we need to promote it to core?

Per #5910. Consider a plugin that has it's own custom Analyzer using AnalysisPlugin. Downstream plugins implement a TokenFilterFactory which needs Lucene's TokenStream. We need to take a dependency on lucene for this in whatever library houses those contracts (currently decided to be the opensearch-core library).

Separately from that, I could remove BytesRefUtils altogether as that logic isn't actually used anywhere. I chose to keep it (for now) as a convenience function for external plugins.

We need to take a dependency on lucene for this in whatever library houses those contracts (currently decided to be the opensearch-core library).

This is totally fine, and I believe by importing AnalysisPlugin the Lucene will be brought in, but the core does not need it and should not bring it I think. Take libs for example - they hardly ever need to know about Lucene but very likely will depend on core / common.

Separately from that, I could remove BytesRefUtils altogether as that logic isn't actually used anywhere.

👍 ty

...but the core does not need it and should not bring it I think

I think we're not on the same page. #5910 plans to move general contracts like AnalysisPlugin (and all of the others) out of :server and into opensearch-core library so plugins only need take a dependency on the API not the entire :server module. To do that requires classes like *Plugin, be refactored out of :server and into libs:opensearch-core. If you refactor AnalysisPlugin to :opensearch-core you need :opensearch-core to take a dependency on lucene. So, the core library will need this lucene dependency, otherwise these contracts have to stay in :server

What's the different route?

First thing, remove dependency to opensearch-core where opensearch-common would be sufficient (possibly moving few classes between modules). With that - we could be 1000% sure that no new transitive dependencies would surprise us along the away.

In the meantime I updated this PR to remove opensearch-core dependency from ssl-config and nio libraries.

@nknize libs/opensearch-cli ?

I was hoping all of this would be done incrementally after this PR was merged.

There is an argument for moving the xcontent classes out of core into the common library now and letting core take the dependency on Lucene and other third-party libraries. Which would mean all of the plugins would revert the PRs they just merged. This is why we need to decouple downstream repos to enable us to move on core changes.

…om server to libraries To further reduce the surface area of utility methods in :server: this commit refactors the following: * MapBuilder from server module to common library * Assertions from server module to core library * BytesRef methods from CollectionUtils in server module to new BytesRefUtils in core library It also removes CollectionUtils dependency on hppc in prep for refactoring CollectionUtils methods to the proper library. Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

github-actions · 2023-03-30T21:18:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/13132/
CommitID: d89e8d4

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

github-actions · 2023-03-31T19:07:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/13174/
CommitID: 12d20bd
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

github-actions · 2023-03-31T21:21:19Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/13183/
CommitID: 138b528

opensearch-trigger-bot · 2023-04-03T14:14:36Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-6875-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0a2009250cd857d68cdfa87340f8cd1b906d8c6e
# Push it to GitHub
git push --set-upstream origin backport/backport-6875-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-6875-to-2.x.

…om server to libraries (opensearch-project#6875) To further reduce the surface area of utility methods in :server: this commit refactors the following: * MapBuilder from server module to common library * Assertions from server module to core library * BytesRef methods from CollectionUtils in server module to new BytesRefUtils in core library It also removes CollectionUtils dependency on hppc in prep for refactoring CollectionUtils methods to the proper library. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> Signed-off-by: Valentin Mitrofanov <mitrofmep@gmail.com>

ashking94 · 2023-05-05T10:05:19Z

@nknize can we backport this PR to 2.x? Seeing failure in backport #7364 of original PR #7227.

…om server to libraries (opensearch-project#6875) To further reduce the surface area of utility methods in :server: this commit refactors the following: * MapBuilder from server module to common library * Assertions from server module to core library * BytesRef methods from CollectionUtils in server module to new BytesRefUtils in core library It also removes CollectionUtils dependency on hppc in prep for refactoring CollectionUtils methods to the proper library. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> (cherry picked from commit 0a20092)

…om server to libraries (#6875) (#7612) To further reduce the surface area of utility methods in :server: this commit refactors the following: * MapBuilder from server module to common library * Assertions from server module to core library * BytesRef methods from CollectionUtils in server module to new BytesRefUtils in core library It also removes CollectionUtils dependency on hppc in prep for refactoring CollectionUtils methods to the proper library. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> (cherry picked from commit 0a20092) Signed-off-by: bansvaru <bansvaru@amazon.com> Co-authored-by: Nicholas Walter Knize <nknize@apache.org>

nknize requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng, VachaShah and xuezhou25 as code owners March 29, 2023 19:58

github-actions bot added the distinguished-contributor label Mar 29, 2023

nknize added v3.0.0 Issues and PRs related to version 3.0.0 >breaking Identifies a breaking change. skip-changelog and removed >breaking Identifies a breaking change. labels Mar 29, 2023

nknize force-pushed the refactorUtilMethods1 branch from 61f473a to 5881606 Compare March 29, 2023 22:20

nknize force-pushed the refactorUtilMethods1 branch from 5881606 to f674d09 Compare March 30, 2023 04:13

nknize force-pushed the refactorUtilMethods1 branch from f674d09 to 4e811c7 Compare March 30, 2023 04:14

reta reviewed Mar 30, 2023

View reviewed changes

nknize force-pushed the refactorUtilMethods1 branch from 4e811c7 to d89e8d4 Compare March 30, 2023 20:48

remove dependency on opensearch-core lib from opensearch-nio

8d6af82

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

nknize mentioned this pull request Mar 31, 2023

[Discuss] Core Foundation Code Refactoring #5910

Open

remove opensearch-core dependency from ssl-config and nio libraries

12d20bd

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

remove opensearch-cli dependency on opensearch-core

138b528

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

reta approved these changes Mar 31, 2023

View reviewed changes

nknize merged commit 0a20092 into opensearch-project:main Apr 3, 2023

nknize added the backport 2.x Backport to 2.x branch label Apr 3, 2023

This was referenced Apr 3, 2023

Feature/get missing alias json exception 301 opensearch-project/opensearch-java#399

Closed

Fix compilation issues (after changes in main) opensearch-project/opensearch-java#423

Merged

ashking94 mentioned this pull request May 5, 2023

Add remote refresh segment pressure service, settings and tracker #7227

Merged

6 tasks

linuxpi mentioned this pull request May 17, 2023

[Refactor] Various Utilities, Assertion, and Concurrency Exception fr… #7612

Merged

6 tasks

linuxpi mentioned this pull request May 19, 2023

[Refactor] Various Utilities, Assertion, and Concurrency Exception fr… #7640

Closed

6 tasks

monusingh-1 mentioned this pull request May 22, 2023

Updated plugin code based on OpenSearch-3.0 upstream changes opensearch-project/cross-cluster-replication#766

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Refactor] Various Utilities, Assertion, and Concurrency Exception from server to libraries #6875

[Refactor] Various Utilities, Assertion, and Concurrency Exception from server to libraries #6875

nknize commented Mar 29, 2023

github-actions bot commented Mar 29, 2023

codecov-commenter commented Mar 29, 2023 •

edited

Loading

github-actions bot commented Mar 29, 2023

github-actions bot commented Mar 30, 2023

nknize commented Mar 30, 2023

github-actions bot commented Mar 30, 2023

reta Mar 30, 2023

nknize Mar 30, 2023

nknize Mar 30, 2023

reta Mar 30, 2023 •

edited

Loading

nknize Mar 30, 2023

reta Mar 31, 2023 •

edited

Loading

nknize Mar 31, 2023

reta Mar 31, 2023

nknize Mar 31, 2023

nknize Mar 31, 2023

github-actions bot commented Mar 30, 2023

github-actions bot commented Mar 31, 2023

github-actions bot commented Mar 31, 2023

opensearch-trigger-bot bot commented Apr 3, 2023

ashking94 commented May 5, 2023

[Refactor] Various Utilities, Assertion, and Concurrency Exception from server to libraries #6875

[Refactor] Various Utilities, Assertion, and Concurrency Exception from server to libraries #6875

Conversation

nknize commented Mar 29, 2023

github-actions bot commented Mar 29, 2023

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Mar 29, 2023 • edited Loading

Codecov Report

github-actions bot commented Mar 29, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Mar 30, 2023

Gradle Check (Jenkins) Run Completed with:

nknize commented Mar 30, 2023

github-actions bot commented Mar 30, 2023

Gradle Check (Jenkins) Run Completed with:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reta Mar 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reta Mar 31, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Mar 30, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Mar 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Mar 31, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Apr 3, 2023

ashking94 commented May 5, 2023

codecov-commenter commented Mar 29, 2023 •

edited

Loading

reta Mar 30, 2023 •

edited

Loading

reta Mar 31, 2023 •

edited

Loading