Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Streaming Plan Impl #1068

Merged

Conversation

penghuo
Copy link
Collaborator

@penghuo penghuo commented Nov 11, 2022

Description

Major Change

  1. Add StreamingPlan implementation which is the high level abstration of a streaming query. Parser should parse streaming query and submit to QueryManager.
  2. Add add(Split) interface to PhysicalOperator
  3. Add StreamingSource asStreamingSource() on Table. Table which could be used as StreamingSource could implement this interface.

Minor Change

  1. Add cancel interface to QueryManager.
  2. Add StreamingIT which run streaming query on source of local filesystem.

FAQ

  1. Could QueryService and ExecutionEngine been combined as single interface. seems they could be combined.

Todo in Phase 2

  1. [FEATURE] Implement cancel interface of QueryManager in OpenSearchQueryManager. #1071
  2. [FEATURE] Manage async query status. #1072

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Peng Huo <penghuo@gmail.com>
@codecov-commenter
Copy link

codecov-commenter commented Nov 11, 2022

Codecov Report

Merging #1068 (ed263a8) into 2.x (e280866) will decrease coverage by 2.50%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##                2.x    #1068      +/-   ##
============================================
- Coverage     98.28%   95.78%   -2.51%     
- Complexity     3450     3461      +11     
============================================
  Files           345      357      +12     
  Lines          8580     9295     +715     
  Branches        547      669     +122     
============================================
+ Hits           8433     8903     +470     
- Misses          142      334     +192     
- Partials          5       58      +53     
Flag Coverage Δ
query-workbench 62.76% <ø> (?)
sql-engine 98.29% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...g/opensearch/sql/executor/execution/QueryPlan.java 100.00% <ø> (ø)
.../org/opensearch/sql/executor/ExecutionContext.java 100.00% <100.00%> (ø)
...java/org/opensearch/sql/executor/QueryManager.java 100.00% <100.00%> (ø)
...java/org/opensearch/sql/executor/QueryService.java 100.00% <100.00%> (ø)
...rch/sql/executor/execution/StreamingQueryPlan.java 100.00% <100.00%> (ø)
...ecutor/streaming/MicroBatchStreamingExecution.java 100.00% <100.00%> (ø)
.../opensearch/sql/planner/physical/PhysicalPlan.java 100.00% <100.00%> (ø)
...rc/main/java/org/opensearch/sql/storage/Table.java 100.00% <100.00%> (ø)
...opensearch/executor/OpenSearchExecutionEngine.java 100.00% <100.00%> (ø)
...ch/public/components/QueryResults/QueryResults.tsx 61.60% <0.00%> (ø)
... and 9 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@penghuo penghuo changed the title Hp/feature/maximus/streamingplan Add Streaming Source Impl Nov 12, 2022
@penghuo penghuo changed the title Add Streaming Source Impl Add Streaming Plan Impl Nov 14, 2022
Signed-off-by: Peng Huo <penghuo@gmail.com>
Signed-off-by: Peng Huo <penghuo@gmail.com>
Signed-off-by: Peng Huo <penghuo@gmail.com>
Signed-off-by: Peng Huo <penghuo@gmail.com>
Signed-off-by: Peng Huo <penghuo@gmail.com>
Signed-off-by: Peng Huo <penghuo@gmail.com>
Signed-off-by: Peng Huo <penghuo@gmail.com>
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo penghuo changed the base branch from feature/maximus-m1 to 2.x November 16, 2022 20:52
@penghuo penghuo marked this pull request as ready for review November 16, 2022 20:52
@penghuo penghuo requested a review from a team as a code owner November 16, 2022 20:52
@penghuo penghuo added the enhancement New feature or request label Nov 16, 2022
@penghuo penghuo added this to the Maximus M1 - Phase 1 milestone Nov 16, 2022
@penghuo penghuo self-assigned this Nov 16, 2022
dai-chen
dai-chen previously approved these changes Nov 17, 2022
Copy link
Collaborator

@dai-chen dai-chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes!

Signed-off-by: Peng Huo <penghuo@gmail.com>
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo penghuo dismissed stale reviews from Yury-Fridlyand and dai-chen via d1d7a25 November 19, 2022 00:46
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo penghuo merged commit 42143b9 into opensearch-project:2.x Nov 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Development

Successfully merging this pull request may close these issues.

4 participants