Set INDICES_MAX_CLAUSE_COUNT dynamically #13568

harshavamsi · 2024-05-06T22:36:22Z

Description

Changes the INDICES_MAX_CLAUSE_COUNT_SETTING to be dynamic instead of being static.

This PR is virtually identical to #1527, but there was earlier push back around making this an updateable dynamic setting due to the assumption that users would abuse this setting rather than optimize their underlying query. Lucene has since updated the way the clause counts work and this is now enforced at the indexSearcher level -- https://issues.apache.org/jira/browse/LUCENE-8811.

More info can be found in #12549 around the motivation for this change, but I ended up raising a PR to make a simple dynamic setting.

Related Issues

Resolves #1526 #12549

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created https://github.com/opensearch-project/documentation-website/pull/7391/files

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

github-actions · 2024-05-06T22:47:45Z

❌ Gradle check result for dd140e2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/search/SearchService.java

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

github-actions · 2024-05-07T00:28:55Z

❌ Gradle check result for 28d5525: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

CHANGELOG.md

reta · 2024-06-13T20:46:55Z

2.15.0 has sailed away, targeting this change to 2.16.0

github-actions · 2024-06-13T21:14:18Z

✅ Gradle check result for cd92a9b: SUCCESS

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

github-actions · 2024-06-13T21:21:09Z

❌ Gradle check result for b221209: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-06-13T22:10:53Z

❌ Gradle check result for e715516: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

harshavamsi · 2024-06-14T16:25:49Z

Opened https://github.com/opensearch-project/documentation-website/pull/7391/files

github-actions · 2024-06-17T14:43:02Z

❌ Gradle check result for e715516: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

harshavamsi · 2024-06-17T17:59:43Z

❌ Gradle check result for e715516: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

#14300
#12651
@msfroh can we merge?

github-actions · 2024-06-18T18:26:57Z

✅ Gradle check result for e715516: SUCCESS

--------- Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> (cherry picked from commit c5c6024) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

--------- (cherry picked from commit c5c6024) Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

--------- Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

…pensearch-project#14436) --------- (cherry picked from commit c5c6024) Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: kkewwei <kkewwei@163.com>

--------- Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

Set INDICES_MAX_CLAUSE_COUNT dynamically

dd140e2

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

jainankitk reviewed May 6, 2024

View reviewed changes

server/src/main/java/org/opensearch/search/SearchService.java Show resolved Hide resolved

harshavamsi force-pushed the dynamic_max_clause branch from 6ccaa23 to 28d5525 Compare May 7, 2024 00:24

Adds new test to verify clause count is updated

4ff1d13

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

harshavamsi force-pushed the dynamic_max_clause branch from 28d5525 to 4ff1d13 Compare May 7, 2024 00:27

Changelog

6321ca8

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

harshavamsi marked this pull request as ready for review May 7, 2024 00:58

harshavamsi requested review from anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, sachinpkale, saratvemulapalli, shwetathareja, sohami, tlfeng and VachaShah as code owners May 7, 2024 00:58

reta reviewed Jun 13, 2024

View reviewed changes

CHANGELOG.md Show resolved Hide resolved

github-actions bot added the v3.0.0 Issues and PRs related to version 3.0.0 label Jun 13, 2024

Fix changelog

e715516

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

harshavamsi force-pushed the dynamic_max_clause branch from b221209 to e715516 Compare June 13, 2024 21:20

harshavamsi mentioned this pull request Jun 13, 2024

[DOC]Change the docs around max_clause_count opensearch-project/documentation-website#7385

Closed

4 tasks

This was referenced Jun 13, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndicesRequestCacheIT #14288

Open

[AUTOCUT] Gradle Check Flaky Test Report for SearchWithRandomIOExceptionsIT #14306

Open

reta approved these changes Jun 14, 2024

View reviewed changes

opensearch-ci-bot mentioned this pull request Jun 14, 2024

[AUTOCUT] Gradle Check Flaky Test Report for RemoteSplitIndexIT #14296

Open

prudhvigodithi mentioned this pull request Jun 14, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndicesRequestCacheIT prudhvigodithi/OpenSearch#27

Closed

This was referenced Jun 17, 2024

[AUTOCUT] Gradle Check Flaky Test Report for TieredSpilloverCacheStatsIT #14300

Open

[AUTOCUT] Gradle Check Flaky Test Report for AzureBlobContainerRetriesTests #14318

Open

msfroh merged commit c5c6024 into opensearch-project:main Jun 18, 2024
32 checks passed

opensearch-trigger-bot bot mentioned this pull request Jun 18, 2024

[Backport 2.x] Set INDICES_MAX_CLAUSE_COUNT dynamically #14436

Merged

harshavamsi deleted the dynamic_max_clause branch June 18, 2024 18:38

harshavamsi added a commit to harshavamsi/OpenSearch that referenced this pull request Jul 12, 2024

Set INDICES_MAX_CLAUSE_COUNT dynamically (opensearch-project#13568)

d4d8666

--------- Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

BrewTestBot mentioned this pull request Aug 8, 2024

opensearch 2.16.0 Homebrew/homebrew-core#180446

Merged

wdongyu pushed a commit to wdongyu/OpenSearch that referenced this pull request Aug 22, 2024

Set INDICES_MAX_CLAUSE_COUNT dynamically (opensearch-project#13568)

7aaee0c

--------- Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

msfroh mentioned this pull request Nov 26, 2024

[Feature Request] Make rewrite-level max_clause_count check configurable at the index and/or query level #16723

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set INDICES_MAX_CLAUSE_COUNT dynamically #13568

Set INDICES_MAX_CLAUSE_COUNT dynamically #13568

harshavamsi commented May 6, 2024 •

edited

Loading

github-actions bot commented May 6, 2024

github-actions bot commented May 7, 2024

reta commented Jun 13, 2024

github-actions bot commented Jun 13, 2024

github-actions bot commented Jun 13, 2024

github-actions bot commented Jun 13, 2024

harshavamsi commented Jun 14, 2024

github-actions bot commented Jun 17, 2024

harshavamsi commented Jun 17, 2024

github-actions bot commented Jun 18, 2024

Set INDICES_MAX_CLAUSE_COUNT dynamically #13568

Set INDICES_MAX_CLAUSE_COUNT dynamically #13568

Conversation

harshavamsi commented May 6, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented May 6, 2024

github-actions bot commented May 7, 2024

reta commented Jun 13, 2024

github-actions bot commented Jun 13, 2024

github-actions bot commented Jun 13, 2024

github-actions bot commented Jun 13, 2024

harshavamsi commented Jun 14, 2024

github-actions bot commented Jun 17, 2024

harshavamsi commented Jun 17, 2024

github-actions bot commented Jun 18, 2024

harshavamsi commented May 6, 2024 •

edited

Loading