Redefine Drop Index as logical delete #2386

penghuo · 2023-10-27T01:13:52Z

Description

Redefine drop index semantic as logical deletion. align with [FEATURE] Redefine drop index semantic as logical deletion #2380
Refactor drop index result fetch logic
2.1. currently, encode drop index response state in query response.
2.2. change to, write drop index result in result index, then the query result fetch logic is unified.
Remove isDropIndex from jobmedata field
Add Index state based operation. align with [FEATURE] Add state maintain logic for drop index #2381
Add limit on refresh job. plugins.query.executionengine.spark.refresh_job.limit, default: 50
Enable session based query by default. Remove setting plugins.query.executionengine.spark.session.enable

Issues Resolved

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Peng Huo <penghuo@gmail.com>

penghuo · 2023-10-27T16:09:36Z

spark/src/test/java/org/opensearch/sql/spark/dispatcher/SparkQueryDispatcherTest.java

@@ -1029,255 +1018,6 @@ void testGetQueryResponseWithSuccess() {
    verifyNoInteractions(emrServerlessClient);
  }

-  // todo. refactor query process logic in plugin.
-  @Test


move to IndexDMLQuerySpecTest

Signed-off-by: Peng Huo <penghuo@gmail.com>

codecov · 2023-10-30T19:26:13Z

Codecov Report

Merging #2386 (85e98cc) into main (d3ce049) will increase coverage by 0.07%.
The diff coverage is 99.60%.

❗ Current head 85e98cc differs from pull request most recent head 6153f3e. Consider uploading reports for the commit 6153f3e to get more accurate results

@@             Coverage Diff              @@
##               main    #2386      +/-   ##
============================================
+ Coverage     95.55%   95.62%   +0.07%     
- Complexity     4922     4985      +63     
============================================
  Files           471      478       +7     
  Lines         13675    13854     +179     
  Branches        921      930       +9     
============================================
+ Hits          13067    13248     +181     
+ Misses          587      586       -1     
+ Partials         21       20       -1

Flag	Coverage Δ
sql-engine	`95.62% <99.60%> (+0.07%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...rch/sql/opensearch/setting/OpenSearchSettings.java	`100.00% <100.00%> (ø)`
...park/asyncquery/AsyncQueryExecutorServiceImpl.java	`100.00% <ø> (ø)`
.../spark/asyncquery/model/AsyncQueryJobMetadata.java	`86.88% <ø> (-3.74%)`	⬇️
...search/sql/spark/dispatcher/AsyncQueryHandler.java	`100.00% <ø> (ø)`
...search/sql/spark/dispatcher/BatchQueryHandler.java	`100.00% <100.00%> (ø)`
...ensearch/sql/spark/dispatcher/IndexDMLHandler.java	`100.00% <100.00%> (ø)`
.../sql/spark/dispatcher/InteractiveQueryHandler.java	`100.00% <ø> (ø)`
...rch/sql/spark/dispatcher/SparkQueryDispatcher.java	`100.00% <100.00%> (ø)`
...ch/sql/spark/dispatcher/StreamingQueryHandler.java	`100.00% <100.00%> (ø)`
...rch/sql/spark/dispatcher/model/IndexDMLResult.java	`100.00% <100.00%> (ø)`
... and 11 more

Signed-off-by: Peng Huo <penghuo@gmail.com>

* Redefine Drop Index as logical delete Signed-off-by: Peng Huo <penghuo@gmail.com> * merge from 2.x Signed-off-by: Peng Huo <penghuo@gmail.com> * add refresh_job limit and disable batch query Signed-off-by: Peng Huo <penghuo@gmail.com> * update doc Signed-off-by: Peng Huo <penghuo@gmail.com> --------- Signed-off-by: Peng Huo <penghuo@gmail.com> (cherry picked from commit cb8d953) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Redefine Drop Index as logical delete * merge from 2.x * add refresh_job limit and disable batch query * update doc --------- (cherry picked from commit cb8d953) Signed-off-by: Peng Huo <penghuo@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

… (opensearch-project#2397)" This reverts commit e939bb6. Signed-off-by: Eric <menwe@amazon.com>

… (opensearch-project#2397)" This reverts commit e939bb6.

* Revert "Add more metrics and handle emr exception message (#2422) (#2426)" This reverts commit b57f7cc. * Revert "Block settings in sql query settings API and add more unit tests (#2407) (#2412)" This reverts commit 3024737. * Revert "Added session, statement, emrjob metrics to sql stats api (#2398) (#2400)" This reverts commit 6e17ae6. * Revert "Redefine Drop Index as logical delete (#2386) (#2397)" This reverts commit e939bb6. * Revert "add concurrent limit on datasource and sessions (#2390) (#2395)" This reverts commit deb3ccf. * Revert "Add Flint Index Purging Logic (#2372) (#2389)" This reverts commit dd48b9b. * Revert "Refactoring for tags usage in test files and also added explicit denly list setting. (#2383) (#2385)" This reverts commit 37e010f. * Revert "Enable session by default (#2373) (#2375)" This reverts commit 7d95e4c. * Revert "Create new session if client provided session is invalid (#2368) (#2371)" This reverts commit 5ab7858. * Revert "Add where clause support in create statement (#2366) (#2370)" This reverts commit b620a56. * Revert "create new session if current session not ready (#2363) (#2365)" This reverts commit 5d07281. * Revert "Handle Describe,Refresh and Show Queries Properly (#2357) (#2362)" This reverts commit 16e2f30. * Revert "Add Session limitation (#2354) (#2359)" This reverts commit 0f334f8. * Revert "Bug Fix, support cancel query in running state (#2351) (#2353)" This reverts commit 9a40591. * Revert "Fix bug, using basic instead of basicauth (#2342) (#2355)" This reverts commit e4827a5. * Revert "Add missing tags and MV support (#2336) (#2346)" This reverts commit 8791bb0. * Revert "[Backport 2.x] deprecated job-metadata-index (#2340) (#2343)" This reverts commit bea432c. * Revert "Integration with REPL Spark job (#2327) (#2338)" This reverts commit 58a5ae5. * Revert "Implement patch API for datasources (#2273) (#2329)" This reverts commit 4c151fe. * Revert "Add sessionId parameters for create async query API (#2312) (#2324)" This reverts commit 3d1a376. * Revert "Add Statement (#2294) (#2318) (#2319)" This reverts commit b3c2e94. * Revert "Upgrade json (#2307) (#2314)" This reverts commit 6c65bb4. * Revert "Minor Refactoring (#2308) (#2317)" This reverts commit 051cc4f. * Revert "add InteractiveSession and SessionManager (#2290) (#2293) (#2315)" This reverts commit 6ac197b. --------- Co-authored-by: Vamsi Manohar <reddyvam@amazon.com>

Redefine Drop Index as logical delete

4cd3e44

Signed-off-by: Peng Huo <penghuo@gmail.com>

penghuo requested review from pjfitzgibbons, ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, rupal-bq, mengweieric, vmmusings, Swiddis, seankao-az, MaxKsyunz, Yury-Fridlyand, anirudha, forestmvey, acarbonetto and GumpacG as code owners October 27, 2023 01:13

penghuo commented Oct 27, 2023

View reviewed changes

penghuo mentioned this pull request Oct 28, 2023

add concurrent limit on datasource and sessions #2390

Merged

6 tasks

penghuo added 2 commits October 30, 2023 10:32

Merge branch 'main' into dropIndexFixPr

6e8d9dc

merge from 2.x

34a8917

Signed-off-by: Peng Huo <penghuo@gmail.com>

This was referenced Oct 30, 2023

[Backport 2.11] add concurrent limit on datasource and sessions #2395

Merged

[Backport 2.x] add concurrent limit on datasource and sessions #2394

Merged

penghuo added 2 commits October 30, 2023 13:53

add refresh_job limit and disable batch query

85e98cc

Signed-off-by: Peng Huo <penghuo@gmail.com>

update doc

6153f3e

Signed-off-by: Peng Huo <penghuo@gmail.com>

penghuo self-assigned this Oct 30, 2023

penghuo added the backport 2.x label Oct 30, 2023

penghuo added the backport 2.11 label Oct 30, 2023

dai-chen approved these changes Oct 30, 2023

View reviewed changes

ps48 approved these changes Oct 30, 2023

View reviewed changes

vmmusings approved these changes Oct 30, 2023

View reviewed changes

penghuo merged commit cb8d953 into opensearch-project:main Oct 30, 2023
20 of 21 checks passed

This was referenced Oct 30, 2023

[Backport 2.x] Redefine Drop Index as logical delete #2396

Merged

[Backport 2.11] Redefine Drop Index as logical delete #2397

Merged

mengweieric added a commit to mengweieric/sql that referenced this pull request Nov 8, 2023

Revert "Redefine Drop Index as logical delete (opensearch-project#2386)…

d737bfd

… (opensearch-project#2397)" This reverts commit e939bb6. Signed-off-by: Eric <menwe@amazon.com>

mengweieric added a commit to mengweieric/sql that referenced this pull request Nov 8, 2023

Revert "Redefine Drop Index as logical delete (opensearch-project#2386)…

7f2664b

… (opensearch-project#2397)" This reverts commit e939bb6. Signed-off-by: Eric <menwe@amazon.com>

vmmusings added a commit to mengweieric/sql that referenced this pull request Nov 13, 2023

Revert "Redefine Drop Index as logical delete (opensearch-project#2386)…

728eed2

… (opensearch-project#2397)" This reverts commit e939bb6.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Redefine Drop Index as logical delete #2386

Redefine Drop Index as logical delete #2386

penghuo commented Oct 27, 2023 •

edited

Loading

penghuo Oct 27, 2023

codecov bot commented Oct 30, 2023 •

edited

Loading

Redefine Drop Index as logical delete #2386

Redefine Drop Index as logical delete #2386

Conversation

penghuo commented Oct 27, 2023 • edited Loading

Description

Issues Resolved

Check List

penghuo Oct 27, 2023

Choose a reason for hiding this comment

codecov bot commented Oct 30, 2023 • edited Loading

Codecov Report

penghuo commented Oct 27, 2023 •

edited

Loading

codecov bot commented Oct 30, 2023 •

edited

Loading