Skip to content

Conversation

@gaobinlong
Copy link
Contributor

@gaobinlong gaobinlong commented Aug 13, 2025

Description

In Lucene 9.0.0, this change was introduced that removes the ability of TopDocs.merge to set shard indices, and we adopted the change in 2.0.0, but CollapseTopFieldDocs.merge() still supports setting shard indices, that should also be removed.

As a result, when collapsing search results with concurrent segment search enabled, the shardIndex of the top docs are set twice in both queryPhase and FetchSearchPhase, that causes incorrect result are collected in queryPhase and later triggers an assertion error.

Actually, setting shard indices in queryPhase when calling CollapseTopFieldDocs.merge() is not needed because we are merging segment level results, they are all from one shard, we can only compare the lucene doc id when tie breaking for same sort value, small doc id wins.

This PR removes the setShardIndex parameter in CollapseTopFieldDocs.merge() method, following the change of the Lucene PR: apache/lucene-solr#757.

Related Issues

Resolves #19051 and #19111.

Check List

  • Functionality includes testing.
    - [ ] API changes companion pull request created, if applicable.
    - [ ] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Contributor

❌ Gradle check result for 4d4d71f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Binlong Gao <gbinlong@amazon.com>
Signed-off-by: Binlong Gao <gbinlong@amazon.com>
Signed-off-by: Binlong Gao <gbinlong@amazon.com>
Signed-off-by: Binlong Gao <gbinlong@amazon.com>
Signed-off-by: Binlong Gao <gbinlong@amazon.com>
@github-actions
Copy link
Contributor

❌ Gradle check result for fb3b44d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Binlong Gao <gbinlong@amazon.com>
@github-actions
Copy link
Contributor

❌ Gradle check result for 65f8d3b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Contributor

✅ Gradle check result for a803b86: SUCCESS

@codecov
Copy link

codecov bot commented Aug 20, 2025

Codecov Report

❌ Patch coverage is 93.33333% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 72.83%. Comparing base (c442cce) to head (a803b86).
⚠️ Report is 25 commits behind head on main.

Files with missing lines Patch % Lines
...e/lucene/search/grouping/CollapseTopFieldDocs.java 92.30% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #19053      +/-   ##
============================================
- Coverage     72.98%   72.83%   -0.16%     
+ Complexity    69475    69436      -39     
============================================
  Files          5647     5647              
  Lines        319210   319206       -4     
  Branches      46171    46170       -1     
============================================
- Hits         232977   232493     -484     
- Misses        67385    67875     +490     
+ Partials      18848    18838      -10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@gaobinlong gaobinlong changed the title Do not set shardIndex of top docs in CollapsingTopDocsCollectorContext Remove the setShardIndex parameter in CollapseTopFieldDocs.merge() Aug 20, 2025
@gaobinlong gaobinlong marked this pull request as ready for review August 20, 2025 13:06
@gaobinlong
Copy link
Contributor Author

Hi @reta , mind taking a look at this PR? Thanks!

@reta
Copy link
Contributor

reta commented Aug 28, 2025

Hi @reta , mind taking a look at this PR? Thanks!

Thanks @gaobinlong , LGTM, I just have one question / comment to understand what potential impact it could have, thank you.

@gaobinlong gaobinlong merged commit 2eb9b53 into opensearch-project:main Aug 29, 2025
36 checks passed
pranikum pushed a commit to pranikum/OpenSearch that referenced this pull request Sep 4, 2025
…pensearch-project#19053)

* Do not set shardIndex of top docs in CollapsingTopDocsCollectorContext

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Modify change log

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Format code

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Remove setShardIndex parameter in CollapseTopFieldDocs.merge()

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Modify change log

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* tiny change

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

---------

Signed-off-by: Binlong Gao <gbinlong@amazon.com>
kh3ra pushed a commit to kh3ra/OpenSearch that referenced this pull request Sep 5, 2025
…pensearch-project#19053)

* Do not set shardIndex of top docs in CollapsingTopDocsCollectorContext

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Modify change log

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Format code

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Remove setShardIndex parameter in CollapseTopFieldDocs.merge()

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Modify change log

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* tiny change

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

---------

Signed-off-by: Binlong Gao <gbinlong@amazon.com>
jainankitk pushed a commit to jainankitk/OpenSearch that referenced this pull request Sep 22, 2025
…pensearch-project#19053)

* Do not set shardIndex of top docs in CollapsingTopDocsCollectorContext

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Modify change log

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Format code

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Remove setShardIndex parameter in CollapseTopFieldDocs.merge()

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Modify change log

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* tiny change

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

---------

Signed-off-by: Binlong Gao <gbinlong@amazon.com>
jainankitk pushed a commit to jainankitk/OpenSearch that referenced this pull request Sep 22, 2025
…pensearch-project#19053)

* Do not set shardIndex of top docs in CollapsingTopDocsCollectorContext

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Modify change log

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Format code

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Remove setShardIndex parameter in CollapseTopFieldDocs.merge()

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Modify change log

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* tiny change

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

---------

Signed-off-by: Binlong Gao <gbinlong@amazon.com>
Signed-off-by: Ankit Jain <jainankitk@apache.org>
jainankitk pushed a commit to jainankitk/OpenSearch that referenced this pull request Sep 22, 2025
…pensearch-project#19053)

* Do not set shardIndex of top docs in CollapsingTopDocsCollectorContext

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Modify change log

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Format code

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Remove setShardIndex parameter in CollapseTopFieldDocs.merge()

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Modify change log

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* tiny change

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

---------

Signed-off-by: Binlong Gao <gbinlong@amazon.com>
Signed-off-by: Ankit Jain <jainankitk@apache.org>
asimmahmood1 pushed a commit to jainankitk/OpenSearch that referenced this pull request Sep 23, 2025
…pensearch-project#19053)

* Do not set shardIndex of top docs in CollapsingTopDocsCollectorContext

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Modify change log

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Format code

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Remove setShardIndex parameter in CollapseTopFieldDocs.merge()

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Modify change log

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* tiny change

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

---------

Signed-off-by: Binlong Gao <gbinlong@amazon.com>
vinaykpud pushed a commit to vinaykpud/OpenSearch that referenced this pull request Sep 26, 2025
…pensearch-project#19053)

* Do not set shardIndex of top docs in CollapsingTopDocsCollectorContext

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Modify change log

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Format code

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Remove setShardIndex parameter in CollapseTopFieldDocs.merge()

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Modify change log

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* tiny change

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

---------

Signed-off-by: Binlong Gao <gbinlong@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working Search Search query, autocomplete ...etc

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Collapsing search results with concurrent segment search enabled triggers assertion error

2 participants