Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds plan context for Scroll physical plan building #713

Closed
wants to merge 10 commits into from
Closed

Adds plan context for Scroll physical plan building #713

wants to merge 10 commits into from

Conversation

seankao-az
Copy link
Collaborator

@seankao-az seankao-az commented Jul 28, 2022

Description

  1. Adds plan context that can be modified during request handling and/or logical plan analysis. It is then used to help build the physical plan. Right now it has only one field, namely, IndexScanMode, IndexScanType, which decides whether to invoke a regular query request or a scroll request to OpenSearch.
    Before:
    logical plan => physical plan
    After:
    (logical plan, plan context) => physical plan
    An example use case is to distinguish scrolling requests from regular query requests ([FEATURE] - Support pagination for PPL and SQL query #656) . Both of them can have the exact same query, and therefore, have the exact same logical plan. They differ only in the underlying index scan operator choice.
    We can set the index scan type to SCROLL during request handling.
    Another use case is to allow for some commands to request more data rows than query.size_limit and index.max_result_window (Extend size_limit setting in query engine to support unlimited index query. #703).

  2. Pull opensearch index scan results in batches, rather than loading all at once.

Issues Resolved

This is a building block for both #656 and #703

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
@seankao-az seankao-az changed the title Adds plan context for physical plan building Adds plan context for Scroll physical plan building Jul 28, 2022
@codecov-commenter
Copy link

Codecov Report

Merging #713 (3fade3c) into main (592c58b) will decrease coverage by 31.98%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##               main     #713       +/-   ##
=============================================
- Coverage     94.74%   62.76%   -31.99%     
=============================================
  Files           283       10      -273     
  Lines          7676      658     -7018     
  Branches        560      118      -442     
=============================================
- Hits           7273      413     -6860     
+ Misses          349      192      -157     
+ Partials         54       53        -1     
Flag Coverage Δ
query-workbench 62.76% <ø> (ø)
sql-engine ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...a/org/opensearch/sql/analysis/AnalysisContext.java
.../main/java/org/opensearch/sql/planner/Planner.java
...rc/main/java/org/opensearch/sql/storage/Table.java
...search/sql/opensearch/storage/OpenSearchIndex.java
...ch/sql/opensearch/storage/OpenSearchIndexScan.java
...ensearch/storage/system/OpenSearchSystemIndex.java
...c/main/java/org/opensearch/sql/ppl/PPLService.java
...c/main/java/org/opensearch/sql/sql/SQLService.java
...n/java/org/opensearch/sql/utils/OperatorUtils.java
...opensearch/executor/OpenSearchExecutionEngine.java
... and 263 more

Help us with your feedback. Take ten seconds to tell us how you rate us.

Signed-off-by: Sean Kao <seankao@amazon.com>
@seankao-az
Copy link
Collaborator Author

seankao-az commented Aug 1, 2022

The PlanContext isn't that useful compared to its added complexity to the overall design

@seankao-az seankao-az closed this Aug 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants