Merge paginated plan optimizer into the regular optimizer. #1516

MaxKsyunz · 2023-04-12T18:08:04Z

Address this comment in #1497

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

This also made the paginationPlanner in QueryService unnecessary. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

core/src/test/java/org/opensearch/sql/planner/optimizer/LogicalPlanOptimizerTest.java

core/src/main/java/org/opensearch/sql/planner/optimizer/rule/CreatePagingTableScanBuilder.java

core/src/main/java/org/opensearch/sql/executor/QueryService.java

PaginateOperator is not optimized away when scan builder is created. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

…lPlanOptimizerTest.java Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com>

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

…zer' into feature/pagination/P1-one-optimizer

CreatePagingTableScanBuilder now does the same thing. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

…ature/pagination/P1-one-optimizer # Conflicts: # core/src/test/java/org/opensearch/sql/executor/execution/ContinuePaginatedPlanTest.java # core/src/test/java/org/opensearch/sql/executor/execution/PaginatedPlanTest.java # integ-test/src/test/java/org/opensearch/sql/ppl/StandaloneIT.java # plugin/src/main/java/org/opensearch/sql/plugin/config/OpenSearchPluginModule.java

acarbonetto · 2023-04-13T20:51:30Z

core/src/main/java/org/opensearch/sql/planner/optimizer/rule/CreatePagingTableScanBuilder.java

+      var plan = plans.pop();
+      if (plan.getChild().stream().anyMatch(LogicalRelation.class::isInstance)) {
+        if (plan.getChild().size() > 1) {
+          throw new UnsupportedOperationException();


Add a message to explain what's wrong here

acarbonetto · 2023-04-13T20:55:41Z

core/src/main/java/org/opensearch/sql/planner/optimizer/rule/CreatePagingTableScanBuilder.java

+   *     {@link this.logicalPaginate}, false otherwise.
+   */
+  private boolean findLogicalRelation(LogicalPaginate logicalPaginate) {
+    Deque<LogicalPlan> plans = new ArrayDeque<>();


nit: you could create ArrayDeque with logicalPagination

new ArrayDeque<>(logicalPaginate);

Sadly, I cannot -- that constructor expects a collection.

Compared to adding a conversion to collection, just calling push is simpler.

acarbonetto · 2023-04-13T21:01:02Z

core/src/main/java/org/opensearch/sql/planner/optimizer/rule/CreatePagingTableScanBuilder.java

+        relationParent = plan;
+        return true;
+      }
+      plan.getChild().forEach(plans::push);


There's an addAll() but no pushAll() method... hmm
Maybe use addAll() and removeFirst() instead?

I'll switch for the sake of removing forEach(plans:push).

acarbonetto · 2023-04-13T21:01:41Z

core/src/test/java/org/opensearch/sql/planner/optimizer/LogicalPlanOptimizerTest.java

+      verify(table).createScanBuilder();
+      verify(table, never()).createPagedScanBuilder(anyInt());
+      // Assert that createPagedTableScan was not called
+      // Assert that createTableScan was called


codecov-commenter · 2023-04-13T21:08:34Z

Codecov Report

Merging #1516 (9df41c7) into feature/pagination/P1 (bc8c73f) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@                     Coverage Diff                     @@
##             feature/pagination/P1    #1516      +/-   ##
===========================================================
- Coverage                    98.51%   98.51%   -0.01%     
+ Complexity                    4018     4013       -5     
===========================================================
  Files                          362      361       -1     
  Lines                         9911     9905       -6     
  Branches                       644      643       -1     
===========================================================
- Hits                          9764     9758       -6     
  Misses                         142      142              
  Partials                         5        5

Flag	Coverage Δ
sql-engine	`98.51% <100.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...pensearch/sql/planner/logical/LogicalRelation.java	`100.00% <ø> (ø)`
...ch/sql/planner/optimizer/LogicalPlanOptimizer.java	`100.00% <ø> (ø)`
...java/org/opensearch/sql/executor/QueryService.java	`100.00% <100.00%> (ø)`
...opensearch/sql/planner/logical/LogicalPlanDSL.java	`100.00% <100.00%> (ø)`
...r/optimizer/rule/CreatePagingTableScanBuilder.java	`100.00% <100.00%> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

…zer. (opensearch-project#1516) Merge paginated plan optimizer into the regular optimizer. --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com>

* Support pagination in V2 engine, phase 1 (#226) * Fixing integration tests broken during POC Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Comment to clarify an exception. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Add support for paginated scroll request, first page. Implement PaginatedPlanCache.convertToPlan for second page to work. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Progress on paginated scroll request, subsequent page. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Move `ExpressionSerializer` from `opensearch` to `core`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Rename `Cursor` `asString` to `toString`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Disable scroll cleaning. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Add full cursor serialization and deserialization. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Misc fixes. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Further work on pagination. * Added push down page size from `LogicalPaginate` to `LogicalRelation`. * Improved cursor encoding and decoding. * Added cursor compression. * Fixed issuing `SearchScrollRequest`. * Fixed returning last empty page. * Minor code grooming/commenting. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Pagination fix for empty indices. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix error reporting on wrong cursor. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor comments and error reporting improvement. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Add an end-to-end integration test. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Add `explain` request handlers. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Add IT for explain. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Address issues flagged by checkstyle build step (#229) Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Pagination, phase 1: Add unit tests for `:core` module with coverage. (#230) * Add unit tests for `:core` module with coverage. Uncovered: `toCursor`, because it is will be changed soon. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Pagination, phase 1: Add unit tests for SQL module with coverage. (#239) * Add unit tests for SQL module with coverage. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Update sql/src/main/java/org/opensearch/sql/sql/domain/SQLQueryRequest.java Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> --------- Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> * Pagination, phase 1: Add unit tests for `:opensearch` module with coverage. (#233) * Add UT for `:opensearch` module with full coverage, except `toCursor`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix checkstyle. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> --------- Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix the merges. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix explain. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix scroll cleaning. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Store `TotalHits` and use it to report `total` in response. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Add missing UT for `:protocol` module. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix PPL UTs damaged in f4ea4ad. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor checkstyle fixes. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fallback to v1 engine for pagination (#245) * Pagination fallback integration tests. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Add UT with coverage for `toCursor` serialization. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix broken tests in `legacy`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix getting `total` from non-paged requests and from queries without `FROM` clause. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix scroll cleaning. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix cursor request processing. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Update ITs. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix (again) TotalHits feature. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix typo in prometheus config. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Recover commented logging. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Move `test_pagination_blackbox` to a separate class and add logging. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Address some PR feedbacks: rename some classes and revert unnecessary whitespace changed. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor commenting. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Address PR comments. * Add javadocs * Renames * Cleaning up some comments * Remove unused code * Speed up IT Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor missing changes. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Integration tests for fetch_size, max_result_window, and query.size_limit (#248) Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Remove `PaginatedQueryService`, extend `QueryService` to hold two planners and use them. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Move push down functions from request builders to a new interface. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Some file moves. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor clean-up according to PR review. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: MaxKsyunz <maxk@bitquilltech.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Co-authored-by: Max Ksyunz <max.ksyunz@improving.com> * Make scroll timeout configurable. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix IT to set cursor keep alive parameter. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Remove `QueryId.None`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Rename according to PR feedback. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Remove default implementations of `PushDownRequestBuilder`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Merge paginated plan optimizer into the regular optimizer. (#1516) Merge paginated plan optimizer into the regular optimizer. --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Complete rework on serialization and deserialization. (#1498) Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Resolve merge conflicts and fix tests. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor cleanup. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor cleanup - missing changes for the previous commit. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Remove paginate operator (#1528) * Remove PaginateOperator class since it is no longer used. --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Remove `PaginatedPlan` - move logic to `QueryPlan`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Remove default implementations from `SerializablePlan`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Add a doc. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Update design graphs. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * More fixes for merge from upstream/main. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: MaxKsyunz <maxk@bitquilltech.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Co-authored-by: Max Ksyunz <max.ksyunz@improving.com>

* Support pagination in V2 engine, phase 1 (#226) * Fixing integration tests broken during POC Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Comment to clarify an exception. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Add support for paginated scroll request, first page. Implement PaginatedPlanCache.convertToPlan for second page to work. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Progress on paginated scroll request, subsequent page. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Move `ExpressionSerializer` from `opensearch` to `core`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Rename `Cursor` `asString` to `toString`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Disable scroll cleaning. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Add full cursor serialization and deserialization. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Misc fixes. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Further work on pagination. * Added push down page size from `LogicalPaginate` to `LogicalRelation`. * Improved cursor encoding and decoding. * Added cursor compression. * Fixed issuing `SearchScrollRequest`. * Fixed returning last empty page. * Minor code grooming/commenting. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Pagination fix for empty indices. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix error reporting on wrong cursor. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor comments and error reporting improvement. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Add an end-to-end integration test. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Add `explain` request handlers. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Add IT for explain. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Address issues flagged by checkstyle build step (#229) Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Pagination, phase 1: Add unit tests for `:core` module with coverage. (#230) * Add unit tests for `:core` module with coverage. Uncovered: `toCursor`, because it is will be changed soon. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Pagination, phase 1: Add unit tests for SQL module with coverage. (#239) * Add unit tests for SQL module with coverage. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Update sql/src/main/java/org/opensearch/sql/sql/domain/SQLQueryRequest.java Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> --------- Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> * Pagination, phase 1: Add unit tests for `:opensearch` module with coverage. (#233) * Add UT for `:opensearch` module with full coverage, except `toCursor`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix checkstyle. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> --------- Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix the merges. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix explain. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix scroll cleaning. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Store `TotalHits` and use it to report `total` in response. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Add missing UT for `:protocol` module. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix PPL UTs damaged in f4ea4ad. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor checkstyle fixes. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fallback to v1 engine for pagination (#245) * Pagination fallback integration tests. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Add UT with coverage for `toCursor` serialization. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix broken tests in `legacy`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix getting `total` from non-paged requests and from queries without `FROM` clause. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix scroll cleaning. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix cursor request processing. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Update ITs. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix (again) TotalHits feature. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix typo in prometheus config. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Recover commented logging. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Move `test_pagination_blackbox` to a separate class and add logging. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Address some PR feedbacks: rename some classes and revert unnecessary whitespace changed. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor commenting. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Address PR comments. * Add javadocs * Renames * Cleaning up some comments * Remove unused code * Speed up IT Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor missing changes. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Integration tests for fetch_size, max_result_window, and query.size_limit (#248) Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Remove `PaginatedQueryService`, extend `QueryService` to hold two planners and use them. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Move push down functions from request builders to a new interface. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Some file moves. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor clean-up according to PR review. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: MaxKsyunz <maxk@bitquilltech.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Co-authored-by: Max Ksyunz <max.ksyunz@improving.com> * Make scroll timeout configurable. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix IT to set cursor keep alive parameter. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Remove `QueryId.None`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Rename according to PR feedback. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Remove default implementations of `PushDownRequestBuilder`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Merge paginated plan optimizer into the regular optimizer. (opensearch-project#1516) Merge paginated plan optimizer into the regular optimizer. --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Complete rework on serialization and deserialization. (opensearch-project#1498) Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Resolve merge conflicts and fix tests. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor cleanup. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor cleanup - missing changes for the previous commit. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Remove paginate operator (opensearch-project#1528) * Remove PaginateOperator class since it is no longer used. --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Remove `PaginatedPlan` - move logic to `QueryPlan`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Remove default implementations from `SerializablePlan`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Add a doc. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Update design graphs. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * More fixes for merge from upstream/main. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: MaxKsyunz <maxk@bitquilltech.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Co-authored-by: Max Ksyunz <max.ksyunz@improving.com>

Merged paginated plan optimizer into the regular optimizer.

eb23fda

This also made the paginationPlanner in QueryService unnecessary. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

MaxKsyunz requested review from anirudha, penghuo, ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, rupal-bq, mengweieric, vmmusings and Yury-Fridlyand as code owners April 12, 2023 18:08

MaxKsyunz mentioned this pull request Apr 12, 2023

Support pagination in V2 engine, phase 1 #1497

Merged

6 tasks

Yury-Fridlyand reviewed Apr 12, 2023

View reviewed changes

MaxKsyunz and others added 6 commits April 12, 2023 15:05

Update incorrect unit test.

18aa5d8

PaginateOperator is not optimized away when scan builder is created. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

Add unit test to get full coverage.

dd8486a

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

Update core/src/test/java/org/opensearch/sql/planner/optimizer/Logica…

44f45dc

…lPlanOptimizerTest.java Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com>

Address PR feedback.

0de447a

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

Merge remote-tracking branch 'origin/feature/pagination/P1-one-optimi…

1ddd311

…zer' into feature/pagination/P1-one-optimizer

Remove PushPageSize optimizer rule

ba0b7a9

CreatePagingTableScanBuilder now does the same thing. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

GabeFernandez310 approved these changes Apr 13, 2023

View reviewed changes

acarbonetto approved these changes Apr 13, 2023

View reviewed changes

Address PR feedback.

9df41c7

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

MaxKsyunz merged commit b9cb0d0 into opensearch-project:feature/pagination/P1 Apr 14, 2023

Yury-Fridlyand mentioned this pull request Apr 14, 2023

Rework push down page size #1499

Closed

6 tasks

acarbonetto mentioned this pull request Apr 15, 2023

SQL ingestion and ops - standup #791

Open

Yury-Fridlyand added the pagination Pagination feature, ref #656 label May 1, 2023

Yury-Fridlyand deleted the feature/pagination/P1-one-optimizer branch June 16, 2023 22:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge paginated plan optimizer into the regular optimizer. #1516

Merge paginated plan optimizer into the regular optimizer. #1516

MaxKsyunz commented Apr 12, 2023

acarbonetto Apr 13, 2023

acarbonetto Apr 13, 2023

MaxKsyunz Apr 13, 2023

acarbonetto Apr 13, 2023

MaxKsyunz Apr 13, 2023

acarbonetto Apr 13, 2023

codecov-commenter commented Apr 13, 2023 •

edited

Loading

Merge paginated plan optimizer into the regular optimizer. #1516

Merge paginated plan optimizer into the regular optimizer. #1516

Conversation

MaxKsyunz commented Apr 12, 2023

Check List

acarbonetto Apr 13, 2023

Choose a reason for hiding this comment

acarbonetto Apr 13, 2023

Choose a reason for hiding this comment

MaxKsyunz Apr 13, 2023

Choose a reason for hiding this comment

acarbonetto Apr 13, 2023

Choose a reason for hiding this comment

MaxKsyunz Apr 13, 2023

Choose a reason for hiding this comment

acarbonetto Apr 13, 2023

Choose a reason for hiding this comment

codecov-commenter commented Apr 13, 2023 • edited Loading

Codecov Report

codecov-commenter commented Apr 13, 2023 •

edited

Loading