Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Compressors from CompressorFactory to CompressorRegistry for extensibility #9262

Merged
merged 8 commits into from
Aug 15, 2023

Conversation

nknize
Copy link
Collaborator

@nknize nknize commented Aug 11, 2023

This PR refactors the CompressorFactory static singleton class and CompressorType enum to a formal CompressorRegistry and enables downstream implementations to register their own compression implementations through a new CompressorProvider interface for use in compressing Blob stores and MediaType data. This is different from Lucene's Codec compression extension points which expose different compression implementations for Lucene's Stored Fields.

**Note the following implementation details and plans **

  • new :libs:opensearch-compress library - the purpose of this library is much like the :libs:opensearch-x-content library, to provide supplemental compression implementations that are optional for opensearch API users. This design choice is for one reason: keep the core API library lean. Note this does not mean :server is breaking because the zstd classes are not in the core library. It also does not mean downstreams HAVE to take a dependency on :opensearch-compress (only if they want to use zstd as an option).
  • I'd like to refactor Deflate to the new compress library as well and only have the NoneCompressor in the core library. This doesn't mean server will break. :server module will still use Deflate as the default store compression as it will take a dependency on :opensearch-core and :opensearch-compress. Downstream implementations of the opensearch library (e.g., such as :server) can optionally take a dependency on the opensearch-compress library if they want to leverage Deflate and Zstd. The purpose, again, is to keep core library a minimal API implementation of the distributed lucene API.

relates #5910
relates #8110
relates #2447

@nknize nknize added enhancement Enhancement or improvement to existing feature or request pending backport Identifies an issue or PR that still needs to be backported skip-changelog v2.10.0 labels Aug 11, 2023
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@opensearch-trigger-bot
Copy link
Contributor

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 29m 55s

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@opensearch-trigger-bot
Copy link
Contributor

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 21m 23s

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
@nknize nknize force-pushed the compressorSPIExtensions branch from 4a22989 to ec326ae Compare August 14, 2023 19:20
@nknize nknize requested a review from reta August 14, 2023 19:20
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Copy link
Collaborator

@reta reta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM to me!

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@nknize
Copy link
Collaborator Author

nknize commented Aug 14, 2023

These test failures are unreal.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      2 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteLargeBlob

@andrross andrross merged commit c6b67e1 into opensearch-project:main Aug 15, 2023
@nknize nknize added the backport 2.x Backport to 2.x branch label Aug 15, 2023
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-9262-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c6b67e1b938b5e34806216af5efef32954554bf3
# Push it to GitHub
git push --set-upstream origin backport/backport-9262-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-9262-to-2.x.

nknize added a commit to nknize/OpenSearch that referenced this pull request Aug 15, 2023
… extensibility (opensearch-project#9262)

This commit refactors the CompressorFactory static singleton
class and CompressorType enum to a formal CompressorRegistry and enables
downstream implementations to register their own compression
implementations for use in compressing Blob stores and MediaType data.
This is different from Lucene's Codec compression extension points which
expose different compression implementations for Lucene's Stored Fields.

---------

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
(cherry picked from commit c6b67e1)
nknize added a commit to nknize/OpenSearch that referenced this pull request Aug 15, 2023
… extensibility (opensearch-project#9262)

This commit refactors the CompressorFactory static singleton
class and CompressorType enum to a formal CompressorRegistry and enables
downstream implementations to register their own compression
implementations for use in compressing Blob stores and MediaType data.
This is different from Lucene's Codec compression extension points which
expose different compression implementations for Lucene's Stored Fields.

---------

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
(cherry picked from commit c6b67e1)
nknize added a commit to nknize/OpenSearch that referenced this pull request Aug 17, 2023
… extensibility (opensearch-project#9262)

This commit refactors the CompressorFactory static singleton
class and CompressorType enum to a formal CompressorRegistry and enables
downstream implementations to register their own compression
implementations for use in compressing Blob stores and MediaType data.
This is different from Lucene's Codec compression extension points which
expose different compression implementations for Lucene's Stored Fields.

---------

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
(cherry picked from commit c6b67e1)
nknize added a commit to nknize/OpenSearch that referenced this pull request Aug 21, 2023
… extensibility (opensearch-project#9262)

This commit refactors the CompressorFactory static singleton
class and CompressorType enum to a formal CompressorRegistry and enables
downstream implementations to register their own compression
implementations for use in compressing Blob stores and MediaType data.
This is different from Lucene's Codec compression extension points which
expose different compression implementations for Lucene's Stored Fields.

---------

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
(cherry picked from commit c6b67e1)
nknize added a commit that referenced this pull request Aug 21, 2023
… extensibility (#9262) (#9367)

This commit refactors the CompressorFactory static singleton
class and CompressorType enum to a formal CompressorRegistry and enables
downstream implementations to register their own compression
implementations for use in compressing Blob stores and MediaType data.
This is different from Lucene's Codec compression extension points which
expose different compression implementations for Lucene's Stored Fields.

---------

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
(cherry picked from commit c6b67e1)
cwperks added a commit to opensearch-project/security that referenced this pull request Aug 21, 2023
### Description

Reacting to changes from the backport of
opensearch-project/OpenSearch#9262.

This reason the main branch of Security is not encountering these errors
is because the affected class was removed in
#2900 and the change
has not been backported yet.

@willyborankin are you planning to backport
#2900?

* Category (Enhancement, New feature, Bug fix, Test fix, Refactoring,
Maintenance, Documentation)

Maintenance

### Check List
- [ ] New functionality includes testing
- [ ] New functionality has been documented
- [ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and
signing off your commits, please check
[here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

Signed-off-by: Craig Perkins <cwperx@amazon.com>
kkmr pushed a commit to kkmr/OpenSearch that referenced this pull request Aug 28, 2023
… extensibility (opensearch-project#9262)

* Refactor Compressors from CompressorFactory to CompressorRegistry for extensibility

This commit refactors the CompressorFactory static singleton
class and CompressorType enum to a formal CompressorRegistry and enables
downstream implementations to register their own compression
implementations for use in compressing Blob stores and MediaType data.
This is different from Lucene's Codec compression extension points which
expose different compression implementations for Lucene's Stored Fields.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix missing javadoc and relocate the common compress to compress package

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix deprecated case

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix typo in compress build.gradle

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* PR cleanup

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* PR changes

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* update with initial annotations

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* remove NONE singleton in CompressorRegistry and remove static block init

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

---------

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Signed-off-by: Kiran Reddy <kkreddy@amazon.com>
kaushalmahi12 pushed a commit to kaushalmahi12/OpenSearch that referenced this pull request Sep 12, 2023
… extensibility (opensearch-project#9262)

* Refactor Compressors from CompressorFactory to CompressorRegistry for extensibility

This commit refactors the CompressorFactory static singleton
class and CompressorType enum to a formal CompressorRegistry and enables
downstream implementations to register their own compression
implementations for use in compressing Blob stores and MediaType data.
This is different from Lucene's Codec compression extension points which
expose different compression implementations for Lucene's Stored Fields.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix missing javadoc and relocate the common compress to compress package

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix deprecated case

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix typo in compress build.gradle

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* PR cleanup

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* PR changes

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* update with initial annotations

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* remove NONE singleton in CompressorRegistry and remove static block init

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

---------

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>
brusic pushed a commit to brusic/OpenSearch that referenced this pull request Sep 25, 2023
… extensibility (opensearch-project#9262)

* Refactor Compressors from CompressorFactory to CompressorRegistry for extensibility

This commit refactors the CompressorFactory static singleton
class and CompressorType enum to a formal CompressorRegistry and enables
downstream implementations to register their own compression
implementations for use in compressing Blob stores and MediaType data.
This is different from Lucene's Codec compression extension points which
expose different compression implementations for Lucene's Stored Fields.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix missing javadoc and relocate the common compress to compress package

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix deprecated case

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix typo in compress build.gradle

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* PR cleanup

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* PR changes

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* update with initial annotations

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* remove NONE singleton in CompressorRegistry and remove static block init

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

---------

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
… extensibility (opensearch-project#9262)

* Refactor Compressors from CompressorFactory to CompressorRegistry for extensibility

This commit refactors the CompressorFactory static singleton
class and CompressorType enum to a formal CompressorRegistry and enables
downstream implementations to register their own compression
implementations for use in compressing Blob stores and MediaType data.
This is different from Lucene's Codec compression extension points which
expose different compression implementations for Lucene's Stored Fields.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix missing javadoc and relocate the common compress to compress package

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix deprecated case

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix typo in compress build.gradle

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* PR cleanup

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* PR changes

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* update with initial annotations

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* remove NONE singleton in CompressorRegistry and remove static block init

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

---------

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch backport-failed enhancement Enhancement or improvement to existing feature or request pending backport Identifies an issue or PR that still needs to be backported skip-changelog v2.10.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants