Support more than 1024 IP/masks with indexed field #16391

mkhludnev · 2024-10-19T10:20:44Z

Now

terms_query over IP with mask / values parsed as BooleanQuery.
BooleanQuery is prone to max clauses limit
as a result 1025 masks fails with too many clauses limit

Proposal

for indexed field combine masks with MultiRangePointQuery
for doc_values only field still fall back to Boolean Query

Related Issues

Resolves #16200

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-10-19T10:30:40Z

❌ Gradle check result for c9e2bd1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-10-19T17:36:20Z

❌ Gradle check result for c9e2bd1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-10-19T19:45:30Z

❌ Gradle check result for b6c3410: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

mkhludnev · 2024-10-20T07:03:26Z

it's an alt of #16202

server/src/main/java/org/opensearch/index/mapper/IpFieldMapper.java

github-actions · 2024-11-08T15:32:08Z

❌ Gradle check result for 6a11b54: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-11-11T23:29:16Z

❌ Gradle check result for 26ff736: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

* Updating Ip fields to use doc_values to search Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Fix IP tests Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Fix skip to allow yaml test to pass on main Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Update tests to use existing test file Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Changing skip version to match bwc Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Using exact match instead of range Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Spotless Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Fix IP field tests Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Fix spotless + precommit failure Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Get point out of query and into value Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Fix term tests Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Add skip test logic to only doc_values test Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> --------- Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

github-actions · 2024-11-13T08:50:44Z

❌ Gradle check result for 01db875: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-11-19T21:20:31Z

❌ Gradle check result for 5849b96: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-11-20T09:53:54Z

❌ Gradle check result for d03b618: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-11-20T10:36:14Z

❌ Gradle check result for 7e1f8b4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-11-20T15:24:05Z

❌ Gradle check result for c429cf0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

mkhludnev · 2024-11-20T15:51:02Z

Let's discuss edge cases, which are not obvious:

doc_values only field with mask/ values

In this case we can just create BooleanQuery { ssDvRangeQuery, ...}. It may lately fail with too many clauses, and it's reasonable. Note: it will be a way better with ssDvMultiRange

index & doc_values field with mask/ values

a strawman approach: forget about dv and IndexOrDvQuery, and just create MultiRangePointQuery
~~attempt to create IndexOrDvQuery(MultiRangePointQuery, BooleanQuery { ssDvRangeQuery, ...}) when number of BQ is less than MaxClauses limit.~~

The problem is that this boundary limit can't be decided on query parsing because sibling filter clauses may exceed MaxClauses limit.

github-actions · 2024-11-20T17:13:40Z

❌ Gradle check result for cfa3904: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/test/java/org/opensearch/search/SearchIpFieldTermsTests.java

msfroh

Thanks, @mkhludnev! Looks good.

I agree that your new test should probably be marked as an IT, but otherwise I'm happy to merge.

server/src/test/java/org/opensearch/search/SearchIpFieldTermsTests.java

Signed-off-by: Mikhail Khludnev <mkhl@apache.org>

github-actions · 2024-11-26T14:20:10Z

✅ Gradle check result for 341e256: SUCCESS

server/src/internalClusterTest/java/org/opensearch/search/fields/SearchIpFieldTermsIT.java

Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Mikhail Khludnev <mkhludnev@users.noreply.github.com>

Signed-off-by: Mikhail Khludnev <mkhl@apache.org>

github-actions · 2024-11-26T21:02:53Z

❌ Gradle check result for 7aa15da: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/internalClusterTest/java/org/opensearch/search/fields/SearchIpFieldTermsIT.java

github-actions · 2024-11-26T22:11:25Z

❌ Gradle check result for 7aa15da: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Mikhail Khludnev <mkhl@apache.org>

github-actions · 2024-11-27T07:53:55Z

✅ Gradle check result for f6e8303: SUCCESS

reta · 2024-11-27T08:46:35Z

@msfroh LGTY? thanks!

MultiRangeQuery for searching IP masks more 1025 masks in indexed field. --------- Signed-off-by: Mikhail Khludnev <mkhl@apache.org> Signed-off-by: Mikhail Khludnev <mkhludnev@users.noreply.github.com> Co-authored-by: Andriy Redko <drreta@gmail.com> (cherry picked from commit ad982c2) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

MultiRangeQuery for searching IP masks more 1025 masks in indexed field. --------- (cherry picked from commit ad982c2) Signed-off-by: Mikhail Khludnev <mkhl@apache.org> Signed-off-by: Mikhail Khludnev <mkhludnev@users.noreply.github.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Andriy Redko <drreta@gmail.com>

mkhludnev · 2024-11-28T06:17:12Z

Thank you so much, @reta @msfroh!

reta · 2024-11-28T06:18:46Z

Thank you so much, @reta @msfroh!

Thanks to you @mkhludnev !

github-actions bot added enhancement Enhancement or improvement to existing feature or request Search:Query Capabilities labels Oct 19, 2024

opensearch-ci-bot mentioned this pull request Oct 20, 2024

[AUTOCUT] Gradle Check Flaky Test Report for VerifyVersionConstantsIT #14585

Open

mkhludnev commented Oct 20, 2024

View reviewed changes

server/src/main/java/org/opensearch/index/mapper/IpFieldMapper.java Outdated Show resolved Hide resolved

mkhludnev mentioned this pull request Oct 20, 2024

[Feature Request] relax max Clauses Count limitation of termS query over IP field #16200

Closed

msfroh mentioned this pull request Nov 12, 2024

Updating Ip fields to use doc_values to search #11508

Merged

8 tasks

mkhludnev mentioned this pull request Nov 16, 2024

use terms in set for concrete IPs keep disjunctions over ranges. fix #16200 #16202

Closed

3 tasks

mkhludnev force-pushed the 16200-multirange branch from 01db875 to 5849b96 Compare November 19, 2024 21:10

opensearch-ci-bot mentioned this pull request Nov 15, 2024

[AUTOCUT] Gradle Check Flaky Test Report for RestoreShallowSnapshotV2IT #16658

Open

mkhludnev changed the title ~~IP field via MultrangeQuery fix #16200~~ Support more than 1024 IP/masks with indexed field Nov 20, 2024

mkhludnev marked this pull request as ready for review November 20, 2024 20:53

mkhludnev requested review from anasalkouz, andrross, ashking94, Bukhtawar and CEHENKLE as code owners November 20, 2024 20:53

reta approved these changes Nov 22, 2024

View reviewed changes

mkhludnev commented Nov 25, 2024

View reviewed changes

server/src/test/java/org/opensearch/search/SearchIpFieldTermsTests.java Outdated Show resolved Hide resolved

msfroh approved these changes Nov 25, 2024

View reviewed changes

server/src/test/java/org/opensearch/search/SearchIpFieldTermsTests.java Outdated Show resolved Hide resolved

mkhludnev added 2 commits November 26, 2024 15:52

test comment

5ff6d11

Signed-off-by: Mikhail Khludnev <mkhl@apache.org>

move heavy test to IT

341e256

Signed-off-by: Mikhail Khludnev <mkhl@apache.org>

mkhludnev requested review from reta and msfroh November 26, 2024 15:13

reta reviewed Nov 26, 2024

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/search/fields/SearchIpFieldTermsIT.java Outdated Show resolved Hide resolved

reta reviewed Nov 26, 2024

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/search/fields/SearchIpFieldTermsIT.java Outdated Show resolved Hide resolved

reta approved these changes Nov 26, 2024

View reviewed changes

mkhludnev and others added 2 commits November 26, 2024 22:47

Apply suggestions from code review

c7382e3

Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Mikhail Khludnev <mkhludnev@users.noreply.github.com>

move heavy test to IT

7aa15da

Signed-off-by: Mikhail Khludnev <mkhl@apache.org>

reta approved these changes Nov 26, 2024

View reviewed changes

reta reviewed Nov 26, 2024

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/search/fields/SearchIpFieldTermsIT.java Outdated Show resolved Hide resolved

This was referenced Nov 27, 2024

[AUTOCUT] Gradle Check Flaky Test Report for MinimumClusterManagerNodesIT #14289

Open

[AUTOCUT] Gradle Check Flaky Test Report for RemoteStoreMultipartIT #15819

Open

move heavy test to IT

f6e8303

Signed-off-by: Mikhail Khludnev <mkhl@apache.org>

reta approved these changes Nov 27, 2024

View reviewed changes

msfroh approved these changes Nov 27, 2024

View reviewed changes

msfroh merged commit ad982c2 into opensearch-project:main Nov 27, 2024
39 checks passed

opensearch-trigger-bot bot mentioned this pull request Nov 27, 2024

[Backport 2.x] Support more than 1024 IP/masks with indexed field #16734

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support more than 1024 IP/masks with indexed field #16391

Support more than 1024 IP/masks with indexed field #16391

mkhludnev commented Oct 19, 2024 •

edited

Loading

github-actions bot commented Oct 19, 2024

github-actions bot commented Oct 19, 2024

github-actions bot commented Oct 19, 2024

mkhludnev commented Oct 20, 2024

github-actions bot commented Nov 8, 2024

github-actions bot commented Nov 11, 2024

github-actions bot commented Nov 13, 2024

github-actions bot commented Nov 19, 2024

github-actions bot commented Nov 20, 2024

github-actions bot commented Nov 20, 2024

github-actions bot commented Nov 20, 2024

mkhludnev commented Nov 20, 2024 •

edited

Loading

github-actions bot commented Nov 20, 2024

msfroh left a comment

github-actions bot commented Nov 26, 2024

github-actions bot commented Nov 26, 2024

github-actions bot commented Nov 26, 2024

github-actions bot commented Nov 27, 2024

reta commented Nov 27, 2024

mkhludnev commented Nov 28, 2024

reta commented Nov 28, 2024

Support more than 1024 IP/masks with indexed field #16391

Support more than 1024 IP/masks with indexed field #16391

Conversation

mkhludnev commented Oct 19, 2024 • edited Loading

Now

Proposal

Related Issues

Check List

github-actions bot commented Oct 19, 2024

github-actions bot commented Oct 19, 2024

github-actions bot commented Oct 19, 2024

mkhludnev commented Oct 20, 2024

github-actions bot commented Nov 8, 2024

github-actions bot commented Nov 11, 2024

github-actions bot commented Nov 13, 2024

github-actions bot commented Nov 19, 2024

github-actions bot commented Nov 20, 2024

github-actions bot commented Nov 20, 2024

github-actions bot commented Nov 20, 2024

mkhludnev commented Nov 20, 2024 • edited Loading

doc_values only field with mask/ values

index & doc_values field with mask/ values

github-actions bot commented Nov 20, 2024

msfroh left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 26, 2024

github-actions bot commented Nov 26, 2024

github-actions bot commented Nov 26, 2024

github-actions bot commented Nov 27, 2024

reta commented Nov 27, 2024

mkhludnev commented Nov 28, 2024

reta commented Nov 28, 2024

mkhludnev commented Oct 19, 2024 •

edited

Loading

mkhludnev commented Nov 20, 2024 •

edited

Loading