Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Statement, QueryExecution and QueryManager #845

Conversation

penghuo
Copy link
Collaborator

@penghuo penghuo commented Sep 19, 2022

New abstraction

QueryManager is the high level interface of the core engine, Parser parse raw query as the Plan and sumitted to QueryManager.

  1. AstBuilder analyze raw query string and create Statement.
  2. QueryPlanFactory create Plan for different Statement.
  3. QueryManager execute Plan.

Screen Shot 2022-10-25 at 5 05 46 PM

Core engine define the interface of QueryManager. Each execution engine should provide the implementation of QueryManager which bind to execution enviroment.
QueryManager manage all the submitted plans and define the following interface

  • submit: submit queryexecution.
  • cancel: cancel query execution.
  • get: get query execution info of specific query.

image

Parser parse raw query as Statement and create AbstractPlan. Each AbstractPlan decide how to execute the query in QueryManager.

image (1)

QueryService is the low level interface of core engine, each Plan decide how to execute the query and use QueryService to analyze, plan, optimize and execute the query.

Screen Shot 2022-10-25 at 5 07 29 PM

Change of existing logic

  1. Remove the schedule logic in NIO thread. After the change,
    a. Parser will be executed in NIO thread.
    b. QueryManager decide query execution strategy. e.g. OpenSearchQueryManager schedule the QueryExecution running in sql-worker thread pool.

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@penghuo penghuo marked this pull request as ready for review September 19, 2022 20:08
@penghuo penghuo requested a review from a team as a code owner September 19, 2022 20:08
Metrics.getInstance()
.getNumericalMetric(MetricName.PPL_FAILED_REQ_COUNT_CUS)
.increment();
return channel ->
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove the schedule logic. In OpenSearch, QueryManager will schedule the query in sql-worker thread.

@codecov-commenter
Copy link

codecov-commenter commented Oct 3, 2022

Codecov Report

Merging #845 (26fc997) into feature/maximus-m1 (63f3449) will decrease coverage by 2.65%.
The diff coverage is 98.97%.

@@                   Coverage Diff                    @@
##             feature/maximus-m1     #845      +/-   ##
========================================================
- Coverage                 97.58%   94.92%   -2.66%     
- Complexity                 3177     3203      +26     
========================================================
  Files                       307      325      +18     
  Lines                      7898     8612     +714     
  Branches                    513      634     +121     
========================================================
+ Hits                       7707     8175     +468     
- Misses                      190      382     +192     
- Partials                      1       55      +54     
Flag Coverage Δ
query-workbench 62.76% <ø> (?)
sql-engine 97.58% <98.97%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...java/org/opensearch/sql/ppl/parser/AstBuilder.java 100.00% <ø> (ø)
...rg/opensearch/sql/sql/config/SQLServiceConfig.java 100.00% <ø> (ø)
...opensearch/sql/ppl/parser/AstStatementBuilder.java 83.33% <83.33%> (ø)
...g/opensearch/sql/executor/DefaultQueryManager.java 100.00% <100.00%> (ø)
...main/java/org/opensearch/sql/executor/QueryId.java 100.00% <100.00%> (ø)
...java/org/opensearch/sql/executor/QueryService.java 100.00% <100.00%> (ø)
...opensearch/sql/executor/execution/ExplainPlan.java 100.00% <100.00%> (ø)
...g/opensearch/sql/executor/execution/QueryPlan.java 100.00% <100.00%> (ø)
...earch/sql/executor/execution/QueryPlanFactory.java 100.00% <100.00%> (ø)
...ql/opensearch/executor/OpenSearchQueryManager.java 100.00% <100.00%> (ø)
... and 16 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@penghuo penghuo marked this pull request as draft October 12, 2022 06:04
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo penghuo force-pushed the feature/maximus/statement branch from 16eb89e to 2080313 Compare October 26, 2022 00:06
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo penghuo changed the title Add Statement, QueryExecution and QueryManager [Feature] Add Statement, QueryExecution and QueryManager Oct 26, 2022
@penghuo penghuo self-assigned this Oct 26, 2022
@penghuo penghuo added the enhancement New feature or request label Oct 26, 2022
@penghuo penghuo marked this pull request as ready for review October 26, 2022 01:02
@penghuo penghuo changed the title [Feature] Add Statement, QueryExecution and QueryManager Add Statement, QueryExecution and QueryManager Oct 26, 2022
@penghuo penghuo added this to the Maximus M1 - Phase 1 milestone Oct 26, 2022
Signed-off-by: Peng Huo <penghuo@gmail.com>
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo penghuo merged commit 30fcd79 into opensearch-project:feature/maximus-m1 Oct 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Development

Successfully merging this pull request may close these issues.

3 participants