Support partial filter push down #3850

qianheng-aws · 2025-07-04T10:31:10Z

Description

Support partial filter push down

Related Issues

Resolves #3470

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Heng Qian <qianheng@amazon.com>

qianheng-aws · 2025-07-11T04:17:42Z

ping @LantaoJin @penghuo

LantaoJin · 2025-07-11T09:33:15Z

integ-test/src/test/resources/expectedOutput/calcite/explain_partial_filter_push.json

+{
+  "calcite": {
+    "logical": "LogicalProject(age=[$8], address=[$2])\n  LogicalFilter(condition=[AND(>=($8, 1), =($2, '880 Holmes Lane'))])\n    CalciteLogicalIndexScan(table=[[OpenSearch, opensearch-sql_test_index_account]])\n",
+    "physical": "EnumerableCalc(expr#0..1=[{inputs}], expr#2=['880 Holmes Lane':VARCHAR], expr#3=[=($t0, $t2)], age=[$t1], address=[$t0], $condition=[$t3])\n  CalciteEnumerableIndexScan(table=[[OpenSearch, opensearch-sql_test_index_account]], PushDownContext=[[PROJECT->[address, age], FILTER->AND(>=($1, 1), =($0, '880 Holmes Lane'))], OpenSearchRequestBuilder(sourceBuilder={\"from\":0,\"timeout\":\"1m\",\"query\":{\"bool\":{\"must\":[{\"range\":{\"age\":{\"from\":1,\"to\":null,\"include_lower\":true,\"include_upper\":true,\"boost\":1.0}}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"_source\":{\"includes\":[\"address\",\"age\"],\"excludes\":[]},\"sort\":[{\"_doc\":{\"order\":\"asc\"}}]}, requestedTotalSize=2147483647, pageSize=null, startFrom=0)])\n"


shouldn't be FILTER->[>=($1, 1)] instead of FILTER->AND(>=($1, 1), =($0, '880 Holmes Lane')) in PushDownContext?

It uses the original condition currently as its digest now. If we needs to use the pushed condition as its digest, we need to store that RexNode as well like non-pushed condition. Both is ok for functionality, the latter one should be more appropriate for explanation.

Will make that change.

Signed-off-by: Heng Qian <qianheng@amazon.com>

…ushDown

Signed-off-by: Heng Qian <qianheng@amazon.com>

qianheng-aws · 2025-07-15T02:51:45Z

Find a good case for Calcite with partial push down:

Calcite with partial down could execute this PPL while v2 will throw exception:

source=opensearch-sql_test_index_beer | eval answerId= AcceptedAnswerId + 1 | where simple_query_string(['Tags'], 'taste') and answerId > 200

It's because Calcite could only push the relevance function into scan through eval command since it's unrelated to the eval command. It's made by FilterProjectTransposeRule and Partial Push Down feature. And the final plan is:

EnumerableProject(ParentId=[$0], CreationDate=[$1], Title=[$2], ViewCount=[$3], LastEditorUserId=[$4], ContentLicense=[$5], OwnerUserId=[$6], Score=[$7], FavoriteCount=[$8], LastActivityDate=[$9], AnswerCount=[$10], CommentCount=[$11], ClosedDate=[$12], Id=[$13], LastEditDate=[$14], PostTypeId=[$15], AcceptedAnswerId=[$16], Body=[$17], Tags=[$18], $f19=[+($16, 1)])
  EnumerableFilter(condition=[>(+($16, 1), 200)])
    CalciteEnumerableIndexScan(table=[[OpenSearch, opensearch-sql_test_index_beer]], PushDownContext=[[PROJECT->[ParentId, CreationDate, Title, ViewCount, LastEditorUserId, ContentLicense, OwnerUserId, Score, FavoriteCount, LastActivityDate, AnswerCount, CommentCount, ClosedDate, Id, LastEditDate, PostTypeId, AcceptedAnswerId, Body, Tags], FILTER->simple_query_string(MAP('fields', MAP('Tags':VARCHAR, 1.0E0:DOUBLE)), MAP('query', 'taste':VARCHAR))], OpenSearchRequestBuilder(sourceBuilder={"from":0,"timeout":"1m","query":{"bool":{"must":[{"simple_query_string":{"query":"taste","fields":["Tags^1.0"],"flags":-1,"default_operator":"or","analyze_wildcard":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"fuzzy_transpositions":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"_source":{"includes":["ParentId","CreationDate","Title","ViewCount","LastEditorUserId","ContentLicense","OwnerUserId","Score","FavoriteCount","LastActivityDate","AnswerCount","CommentCount","ClosedDate","Id","LastEditDate","PostTypeId","AcceptedAnswerId","Body","Tags"],"excludes":[]},"sort":[{"_doc":{"order":"asc"}}]}, requestedTotalSize=2147483647, pageSize=null, startFrom=0)])

Signed-off-by: Heng Qian <qianheng@amazon.com>

opensearch/src/main/java/org/opensearch/sql/opensearch/request/PredicateAnalyzer.java

penghuo · 2025-07-16T23:38:52Z

...search/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteLogicalIndexScan.java

+      if (queryExpression.isPartial()) {
+        // Only CompoundQueryExpression could be partial.
+        List<RexNode> conditions = queryExpression.getUnAnalyzableNodes();
+        RexNode newCondition = constructCondition(conditions, getCluster().getRexBuilder());
+        return filter.copy(filter.getTraitSet(), newScan, newCondition);
+      }


nit, pushDownFilter should return newScan, if it is partial, return pair of newScan and un-pushed filters, then let IndexScanRule create new Filter node?

…ushDown # Conflicts: # opensearch/src/main/java/org/opensearch/sql/opensearch/request/PredicateAnalyzer.java # opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/AbstractCalciteIndexScan.java

Signed-off-by: Heng Qian <qianheng@amazon.com>

opensearch-trigger-bot · 2025-07-18T15:32:12Z

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-3850-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0b4423e9f3b50670af922a8608116d7182fd728f
# Push it to GitHub
git push --set-upstream origin backport/backport-3850-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-3850-to-2.19-dev.

LantaoJin · 2025-07-19T00:50:52Z

@qianheng-aws please manually backport via above instructions.

* Support partial filter push down Signed-off-by: Heng Qian <qianheng@amazon.com> * Add doc for PushDownAction Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT Signed-off-by: Heng Qian <qianheng@amazon.com> * Refine code to only keep non-push-down condition in the new filter Signed-off-by: Heng Qian <qianheng@amazon.com> * Refine code Signed-off-by: Heng Qian <qianheng@amazon.com> * Only show the pushed conditions in the PushDownContext Signed-off-by: Heng Qian <qianheng@amazon.com> * Ignore test when push down disabled Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT after merging main Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT because mapping changed after merging main Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> (cherry picked from commit 0b4423e)

qianheng-aws · 2025-07-21T02:46:12Z

@qianheng-aws please manually backport via above instructions.

Backport PR is here: #3899

* Support partial filter push down * Add doc for PushDownAction * Fix IT * Refine code to only keep non-push-down condition in the new filter * Refine code * Only show the pushed conditions in the PushDownContext * Ignore test when push down disabled * Fix IT after merging main * Fix IT because mapping changed after merging main --------- (cherry picked from commit 0b4423e) Signed-off-by: Heng Qian <qianheng@amazon.com>

qianheng-aws added 2 commits July 4, 2025 18:26

Support partial filter push down

60cd893

Signed-off-by: Heng Qian <qianheng@amazon.com>

Add doc for PushDownAction

996a82b

Signed-off-by: Heng Qian <qianheng@amazon.com>

qianheng-aws requested review from GumpacG, LantaoJin, MaxKsyunz, Swiddis, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, noCharger, penghuo, ps48, seankao-az and ykmr1224 as code owners July 4, 2025 10:31

qianheng-aws added calcite calcite migration releated pushdown pushdown related issues enhancement New feature or request labels Jul 4, 2025

Fix IT

b23d7ce

Signed-off-by: Heng Qian <qianheng@amazon.com>

qianheng-aws enabled auto-merge (squash) July 5, 2025 04:53

qianheng-aws disabled auto-merge July 5, 2025 04:53

Refine code to only keep non-push-down condition in the new filter

f55ca70

Signed-off-by: Heng Qian <qianheng@amazon.com>

LantaoJin added the backport 2.19-dev label Jul 8, 2025

Refine code

a1159e1

Signed-off-by: Heng Qian <qianheng@amazon.com>

LantaoJin reviewed Jul 11, 2025

View reviewed changes

LantaoJin mentioned this pull request Jul 11, 2025

Filter script pushdown with RelJson serialization in Calcite #3859

Merged

7 tasks

qianheng-aws added 3 commits July 14, 2025 17:00

Only show the pushed conditions in the PushDownContext

1c7e1f5

Signed-off-by: Heng Qian <qianheng@amazon.com>

Merge remote-tracking branch 'refs/remotes/origin/main' into PartialP…

0ee1767

…ushDown

Ignore test when push down disabled

86ed34a

Signed-off-by: Heng Qian <qianheng@amazon.com>

Fix IT after merging main

ac0c48d

Signed-off-by: Heng Qian <qianheng@amazon.com>

LantaoJin previously approved these changes Jul 15, 2025

View reviewed changes

penghuo reviewed Jul 16, 2025

View reviewed changes

Merge remote-tracking branch 'refs/remotes/origin/main' into PartialP…

456891b

…ushDown # Conflicts: # opensearch/src/main/java/org/opensearch/sql/opensearch/request/PredicateAnalyzer.java # opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/AbstractCalciteIndexScan.java

qianheng-aws dismissed LantaoJin’s stale review via 456891b July 18, 2025 02:20

Fix IT because mapping changed after merging main

4649db8

Signed-off-by: Heng Qian <qianheng@amazon.com>

LantaoJin approved these changes Jul 18, 2025

View reviewed changes

penghuo approved these changes Jul 18, 2025

View reviewed changes

penghuo merged commit 0b4423e into opensearch-project:main Jul 18, 2025
23 checks passed

opensearch-trigger-bot bot added the backport-failed label Jul 18, 2025

LantaoJin added the backport-manually Filed a PR to backport manually. label Jul 22, 2025

opensearch-project deleted a comment from opensearch-trigger-bot bot Aug 13, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support partial filter push down #3850

Support partial filter push down #3850

Uh oh!

qianheng-aws commented Jul 4, 2025 •

edited

Loading

Uh oh!

qianheng-aws commented Jul 11, 2025

Uh oh!

LantaoJin Jul 11, 2025

Uh oh!

qianheng-aws Jul 14, 2025

Uh oh!

qianheng-aws commented Jul 15, 2025

Uh oh!

Uh oh!

penghuo Jul 16, 2025 •

edited

Loading

Uh oh!

qianheng-aws Jul 17, 2025

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Jul 18, 2025

Uh oh!

LantaoJin commented Jul 19, 2025

Uh oh!

qianheng-aws commented Jul 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Support partial filter push down #3850

Support partial filter push down #3850

Uh oh!

Conversation

qianheng-aws commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

qianheng-aws commented Jul 11, 2025

Uh oh!

LantaoJin Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

qianheng-aws Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

qianheng-aws commented Jul 15, 2025

Uh oh!

Uh oh!

penghuo Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qianheng-aws Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Jul 18, 2025

Uh oh!

LantaoJin commented Jul 19, 2025

Uh oh!

qianheng-aws commented Jul 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

qianheng-aws commented Jul 4, 2025 •

edited

Loading

penghuo Jul 16, 2025 •

edited

Loading