Experimental Codec Support #13992

sarthakaggarwal97 · 2024-06-05T10:16:13Z

Description

We currently do not have a way to mark and validate the codecs as experimental in OpenSearch. With this, we would be able to safely introduce experimental codecs for consumption.

Related Issues

Resolves #13723

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
API changes companion pull request created.
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

github-actions · 2024-06-05T11:06:39Z

❌ Gradle check result for 14ee1c0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-06-05T11:18:02Z

✅ Gradle check result for 03f1246: SUCCESS

codecov · 2024-06-05T11:18:41Z

Codecov Report

Attention: Patch coverage is 72.72727% with 3 lines in your changes missing coverage. Please review.

Project coverage is 71.64%. Comparing base (b15cb0c) to head (03f1246).
Report is 854 commits behind head on main.

Files with missing lines	Patch %	Lines
...java/org/opensearch/index/engine/EngineConfig.java	72.72%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13992      +/-   ##
============================================
+ Coverage     71.42%   71.64%   +0.22%     
- Complexity    59978    61336    +1358     
============================================
  Files          4985     5065      +80     
  Lines        282275   288206    +5931     
  Branches      40946    41743     +797     
============================================
+ Hits         201603   206476    +4873     
- Misses        63999    64677     +678     
- Partials      16673    17053     +380

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

reta · 2024-06-05T19:29:10Z

@sarthakaggarwal97 I am not onboard with this change, in my opinion:

we do need generic mechanism to enable / disable codecs that does not rely on OpenSearch custom interfaces (mentioned in [Feature Request] Support for Experimental Codecs #13723)
we should not take all or nothing approach to cut off experimental codecs, the users should be given a choice to have fine-grained control over which ones they want or don't want

On a general note, I think we should not be tight to the notion of "experimental codec in code" but provide the mechanism to configure inclusions / exclusions. Surely this is just my opinion, @dblock @andrross @msfroh any thoughts from you guys? Thanks!

andrross · 2024-06-06T16:46:59Z

Agree with @reta's comment above. I still don't understand what is preventing us from implementing this capability within custom-codecs behind a setting defined by that plugin?

sarthakaggarwal97 · 2024-06-10T05:46:26Z

@andrross the issue is that the codecs are validated and made available over at two places: EngineConfig and at CustomCodecService.

The validation that happens in the EngineConfig, the codecs are available via NamedSPI which in turn gets the codecs / services from resources file. I don't think there is a way to load granular resources/services/codecs based on the cluster settings.

We can definitely control whether to make the codecs available or not in CustomCodecService but I dont think there is way to make codecs not available in EngineConfig once loaded via resources.

If we don't make a validation change (as in the PR), we would always allow the experimental codec (as the codec was loaded through the resources), as it will be present in the NamedSPI. If the same experimental codec is not available in CodecService, the shards will fail.

If there is another way out by limiting the changes in custom-codecs plugin, we should go for it.

Tagging @reta @backslasht @mgodwan @shwetathareja @dblock @msfroh @ankitkala to add more thoughts.

reta · 2024-06-10T14:26:14Z

The validation that happens in the EngineConfig, the codecs are available via NamedSPI which in turn gets the codecs / services from resources file. I don't think there is a way to load granular resources/services/codecs based on the cluster settings.

So we have 2 issues here:

Codecs SPI (that come from Apache Lucene)
CodecService that manages codecs on OS side

As you rightfully mentioned, Apache Lucene has no direct support of codecs filtering (the only way we could distinguish stable and non-stable codecs is by convention, Apache Lucene uses sandbox there). Let us enumerate the options here that target any codec that user may use (not relying on any additional OpenSearch specific interfaces):

add new node level setting to explicitly enable / disable codecs, that would work with EngineConfig (using dependent setting) and it would also cover any codec, either supplied by Apache Lucene or custom-codecs plugin
add new node level setting to explicitly enable / disable sandbox codecs (we could get that from the package), it will work with Apache Lucene codecs but not custom-codecs plugin since we didn't follow the convention, this is also not convenient since it is "all-or-nothing" approach, I think we should be able to give more control to the user which codecs to enable / disable

We could narrow the scope and try to contain the change with custom-codecs plugin only but the split between "codecs coming from anywhere" and "settings are validated in core" makes it difficult to implement, especially with the need to allow individual codecs to be enabled / disabled.

andrross · 2024-06-10T18:07:46Z

@reta What is the behavior of the system if we just implement runtime checks based on the value of some "experimental" configuration setting that dynamically chooses the codecs to add in the CodecService here? https://github.com/opensearch-project/custom-codecs/blob/main/src/main/java/org/opensearch/index/codec/customcodecs/CustomCodecService.java#L47

reta · 2024-06-10T18:15:36Z

@reta What is the behavior of the system if we just implement runtime checks based on the value of some "experimental" configuration setting that dynamically chooses the codecs to add in the CodecService here?

@andrross that would not "hide" them from Apache Lucene: https://github.com/opensearch-project/custom-codecs/blob/main/src/main/resources/META-INF/services/org.apache.lucene.codecs.Codec

andrross · 2024-06-10T18:28:33Z

@reta What is the behavior of the system if we just implement runtime checks based on the value of some "experimental" configuration setting that dynamically chooses the codecs to add in the CodecService here?

@andrross that would not "hide" them from Apache Lucene: https://github.com/opensearch-project/custom-codecs/blob/main/src/main/resources/META-INF/services/org.apache.lucene.codecs.Codec

Right, but would they be usable to an OpenSearch user if CodecService prevent access?

reta · 2024-06-10T18:39:16Z

Right, but would they be usable to an OpenSearch user if CodecService prevent access?

Yes and no: the validation check will pass successfully for all codec related settings (those use Codec SPI), but it will fail later on if the codec is filtered out from CodecService when engine is going to be created. So in some places the codecs will be usable, in others - not.

sarthakaggarwal97 · 2024-06-11T05:47:15Z

Yes and no: the validation check will pass successfully for all codec related settings (those use Codec SPI), but it will fail later on if the codec is filtered out from CodecService when engine is going to be created. So in some places the codecs will be usable, in others - not.

yeah, exactly. The shard will fail in this scenario.

@reta @andrross what should be the next course of action on this?

reta · 2024-06-11T12:22:58Z

@reta @andrross what should be the next course of action on this?

@sarthakaggarwal97 I think we have to agree on 3 things in principle:

do we want to have enable / disable controls on individual codecs level or not? (I think we should)
do we want to have enable / disable controls on class of codecs (sandbox / experimental / ...)? (I think we should not)
do we want to limit enable / disable controls to the custom-codecs plugin only? (I think we should)

The decisions here would shape the solution.

sarthakaggarwal97 · 2024-06-11T13:34:56Z

@reta I'm aligned with your suggestion. If we are able to restrict the experimental codec changes to custom-codecs, then we should go for it. Granular control over experimental codec would be ideal as well.

reta · 2024-06-12T14:12:19Z

@reta I'm aligned with your suggestion. If we are able to restrict the experimental codec changes to custom-codecs, then we should go for it. Granular control over experimental codec would be ideal as well.

Thanks @sarthakaggarwal97 , let us close ti and move the discussion to opensearch-project/custom-codecs#148

opensearch-trigger-bot · 2024-07-12T15:21:44Z

This PR is stalled because it has been open for 30 days with no activity.

sandeshkr419 · 2024-08-12T21:22:25Z

server/src/main/java/org/opensearch/index/engine/EngineConfig.java

@@ -140,15 +140,19 @@ public Supplier<RetentionLeases> retentionLeasesSupplier() {
                return s;
            default:
                if (Codec.availableCodecs().contains(s)) {
-                    return s;
+                    if (!isExperimentalCodec(Codec.forName(s))) {
+                        return s;


nit: Can we just rename s to something better?

opensearch-trigger-bot · 2024-09-15T15:21:36Z

This PR is stalled because it has been open for 30 days with no activity.

sarthakaggarwal97 requested review from anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, sachinpkale, saratvemulapalli, shwetathareja, sohami, tlfeng and VachaShah as code owners June 5, 2024 10:16

github-actions bot added enhancement Enhancement or improvement to existing feature or request Indexing Indexing, Bulk Indexing and anything related to indexing labels Jun 5, 2024

experimental codec support

03f1246

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

sarthakaggarwal97 force-pushed the experimental-custom-codecs branch from 14ee1c0 to 03f1246 Compare June 5, 2024 10:19

sarthakaggarwal97 added the backport 2.x Backport to 2.x branch label Jun 5, 2024

reta mentioned this pull request Jun 12, 2024

[FEATURE] Add support for experimental codecs opensearch-project/custom-codecs#148

Open

opensearch-ci-bot mentioned this pull request Jun 13, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndicesRequestCacheIT #14288

Open

prudhvigodithi mentioned this pull request Jun 14, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndicesRequestCacheIT prudhvigodithi/OpenSearch#27

Closed

opensearch-trigger-bot bot added stalled Issues that have stalled and removed stalled Issues that have stalled labels Jul 12, 2024

sarthakaggarwal97 mentioned this pull request Jul 26, 2024

[BUG] 2.15 using qat_deflate with default docker image crashes node because of missing library opensearch-project/custom-codecs#168

Closed

sandeshkr419 reviewed Aug 12, 2024

View reviewed changes

opensearch-trigger-bot bot added the stalled Issues that have stalled label Sep 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Experimental Codec Support #13992

Experimental Codec Support #13992

sarthakaggarwal97 commented Jun 5, 2024 •

edited

Loading

github-actions bot commented Jun 5, 2024

github-actions bot commented Jun 5, 2024

codecov bot commented Jun 5, 2024 •

edited

Loading

reta commented Jun 5, 2024 •

edited

Loading

andrross commented Jun 6, 2024

sarthakaggarwal97 commented Jun 10, 2024 •

edited

Loading

reta commented Jun 10, 2024

andrross commented Jun 10, 2024

reta commented Jun 10, 2024

andrross commented Jun 10, 2024

reta commented Jun 10, 2024

sarthakaggarwal97 commented Jun 11, 2024 •

edited

Loading

reta commented Jun 11, 2024 •

edited

Loading

sarthakaggarwal97 commented Jun 11, 2024

reta commented Jun 12, 2024

opensearch-trigger-bot bot commented Jul 12, 2024

sandeshkr419 Aug 12, 2024

opensearch-trigger-bot bot commented Sep 15, 2024

Experimental Codec Support #13992

Are you sure you want to change the base?

Experimental Codec Support #13992

Conversation

sarthakaggarwal97 commented Jun 5, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Jun 5, 2024

github-actions bot commented Jun 5, 2024

codecov bot commented Jun 5, 2024 • edited Loading

Codecov Report

reta commented Jun 5, 2024 • edited Loading

andrross commented Jun 6, 2024

sarthakaggarwal97 commented Jun 10, 2024 • edited Loading

reta commented Jun 10, 2024

andrross commented Jun 10, 2024

reta commented Jun 10, 2024

andrross commented Jun 10, 2024

reta commented Jun 10, 2024

sarthakaggarwal97 commented Jun 11, 2024 • edited Loading

reta commented Jun 11, 2024 • edited Loading

sarthakaggarwal97 commented Jun 11, 2024

reta commented Jun 12, 2024

opensearch-trigger-bot bot commented Jul 12, 2024

sandeshkr419 Aug 12, 2024

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Sep 15, 2024

sarthakaggarwal97 commented Jun 5, 2024 •

edited

Loading

codecov bot commented Jun 5, 2024 •

edited

Loading

reta commented Jun 5, 2024 •

edited

Loading

sarthakaggarwal97 commented Jun 10, 2024 •

edited

Loading

sarthakaggarwal97 commented Jun 11, 2024 •

edited

Loading

reta commented Jun 11, 2024 •

edited

Loading