Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Backport 2.11] Integration with REPL Spark job #2338

Merged
merged 1 commit into from
Oct 21, 2023
Merged

[Backport 2.11] Integration with REPL Spark job #2338

merged 1 commit into from
Oct 21, 2023

Conversation

opensearch-trigger-bot[bot]
Copy link
Contributor

Backport 7b4156e from #2327.

@github-actions
Integration with REPL Spark job (#2327)
6aa43d6 
* add InteractiveSession and SessionManager

Signed-off-by: Peng Huo <penghuo@gmail.com>

* add statement

Signed-off-by: Peng Huo <penghuo@gmail.com>

* add statement

Signed-off-by: Peng Huo <penghuo@gmail.com>

* fix format

Signed-off-by: Peng Huo <penghuo@gmail.com>

* snapshot

Signed-off-by: Peng Huo <penghuo@gmail.com>

* address comments

Signed-off-by: Peng Huo <penghuo@gmail.com>

* update

Signed-off-by: Peng Huo <penghuo@gmail.com>

* Update REST and Transport interface

Signed-off-by: Peng Huo <penghuo@gmail.com>

* Revert on transport layer

Signed-off-by: Peng Huo <penghuo@gmail.com>

* format code

Signed-off-by: Peng Huo <penghuo@gmail.com>

* add API doc

Signed-off-by: Peng Huo <penghuo@gmail.com>

* modify api

Signed-off-by: Peng Huo <penghuo@gmail.com>

* create query_execution_request index on demand

Signed-off-by: Peng Huo <penghuo@gmail.com>

* add REPL spark parameters

Signed-off-by: Peng Huo <penghuo@gmail.com>

* Add IT

Signed-off-by: Peng Huo <penghuo@gmail.com>

* format code

Signed-off-by: Peng Huo <penghuo@gmail.com>

* bind request index to datasource

Signed-off-by: Peng Huo <penghuo@gmail.com>

* fix bug when fetch query result

Signed-off-by: Peng Huo <penghuo@gmail.com>

* revert entrypoint class

Signed-off-by: Peng Huo <penghuo@gmail.com>

* update mapping

Signed-off-by: Peng Huo <penghuo@gmail.com>

---------

Signed-off-by: Peng Huo <penghuo@gmail.com>
(cherry picked from commit 7b4156e)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@codecov
Copy link

codecov bot commented Oct 20, 2023 •
edited
Loading

Codecov Report

Merging #2338 (6aa43d6) into 2.11 (4c151fe) will decrease coverage by 0.02%.
The diff coverage is 88.12%.

@@             Coverage Diff              @@
##               2.11    #2338      +/-   ##
============================================
- Coverage     96.55%   96.54%   -0.02%     
- Complexity     4899     4921      +22     
============================================
  Files           458      459       +1     
  Lines         13473    13565      +92     
  Branches        906      912       +6     
============================================
+ Hits          13009    13096      +87     
- Misses          449      452       +3     
- Partials         15       17       +2
Flag Coverage Δ
sql-engine 96.54% <88.12%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
.../spark/asyncquery/model/SparkSubmitParameters.java 100.00% <100.00%> (ø)
...earch/sql/spark/data/constants/SparkConstants.java 0.00% <ø> (ø)
...rch/sql/spark/dispatcher/SparkQueryDispatcher.java 100.00% <100.00%> (ø)
.../spark/execution/session/CreateSessionRequest.java 100.00% <100.00%> (ø)
...ql/spark/execution/session/InteractiveSession.java 100.00% <100.00%> (ø)
...nsearch/sql/spark/execution/session/SessionId.java 100.00% <100.00%> (ø)
...ch/sql/spark/execution/session/SessionManager.java 100.00% <100.00%> (ø)
...arch/sql/spark/execution/session/SessionState.java 100.00% <100.00%> (ø)
...earch/sql/spark/execution/session/SessionType.java 100.00% <100.00%> (ø)
...earch/sql/spark/execution/statement/Statement.java 100.00% <100.00%> (ø)
... and 4 more

... and 1 file with indirect coverage changes

@penghuo penghuo merged commit 58a5ae5 into 2.11 Oct 21, 2023
25 of 28 checks passed
@github-actions github-actions bot deleted the backport/backport-2327-to-2.11 branch October 21, 2023 00:27
mengweieric added a commit to mengweieric/sql that referenced this pull request Nov 8, 2023
@mengweieric
Revert "Integration with REPL Spark job (opensearch-project#2327) (op…
bcbffc3 
…ensearch-project#2338)"

This reverts commit 58a5ae5.

Signed-off-by: Eric <menwe@amazon.com>
mengweieric added a commit to mengweieric/sql that referenced this pull request Nov 8, 2023
@mengweieric
Revert "Integration with REPL Spark job (opensearch-project#2327) (op…
9237044 
…ensearch-project#2338)"

This reverts commit 58a5ae5.

Signed-off-by: Eric <menwe@amazon.com>
vmmusings added a commit to mengweieric/sql that referenced this pull request Nov 13, 2023
@vmmusings
Revert "Integration with REPL Spark job (opensearch-project#2327) (op…
a0114f4 
…ensearch-project#2338)"

This reverts commit 58a5ae5.
vmmusings added a commit that referenced this pull request Nov 13, 2023
@mengweieric @vmmusings
Revert commits back to after 2.11 release (#2427)
62d3940 
* Revert "Add more metrics and handle emr exception message (#2422) (#2426)"

This reverts commit b57f7cc.

* Revert "Block settings in sql query settings API and add more unit tests (#2407) (#2412)"

This reverts commit 3024737.

* Revert "Added session, statement, emrjob metrics to sql stats api (#2398) (#2400)"

This reverts commit 6e17ae6.

* Revert "Redefine Drop Index as logical delete (#2386) (#2397)"

This reverts commit e939bb6.

* Revert "add concurrent limit on datasource and sessions (#2390) (#2395)"

This reverts commit deb3ccf.

* Revert "Add Flint Index Purging Logic (#2372) (#2389)"

This reverts commit dd48b9b.

* Revert "Refactoring for tags usage in test files and also added explicit denly list setting. (#2383) (#2385)"

This reverts commit 37e010f.

* Revert "Enable session by default (#2373) (#2375)"

This reverts commit 7d95e4c.

* Revert "Create new session if client provided session is invalid (#2368) (#2371)"

This reverts commit 5ab7858.

* Revert "Add where clause support in create statement (#2366) (#2370)"

This reverts commit b620a56.

* Revert "create new session if current session not ready (#2363) (#2365)"

This reverts commit 5d07281.

* Revert "Handle Describe,Refresh and Show Queries Properly (#2357) (#2362)"

This reverts commit 16e2f30.

* Revert "Add Session limitation (#2354) (#2359)"

This reverts commit 0f334f8.

* Revert "Bug Fix, support cancel query in running state (#2351) (#2353)"

This reverts commit 9a40591.

* Revert "Fix bug, using basic instead of basicauth (#2342) (#2355)"

This reverts commit e4827a5.

* Revert "Add missing tags and MV support (#2336) (#2346)"

This reverts commit 8791bb0.

* Revert "[Backport 2.x] deprecated job-metadata-index (#2340) (#2343)"

This reverts commit bea432c.

* Revert "Integration with REPL Spark job (#2327) (#2338)"

This reverts commit 58a5ae5.

* Revert "Implement patch API for datasources (#2273) (#2329)"

This reverts commit 4c151fe.

* Revert "Add sessionId parameters for create async query API (#2312) (#2324)"

This reverts commit 3d1a376.

* Revert "Add Statement (#2294) (#2318) (#2319)"

This reverts commit b3c2e94.

* Revert "Upgrade json (#2307) (#2314)"

This reverts commit 6c65bb4.

* Revert "Minor Refactoring (#2308) (#2317)"

This reverts commit 051cc4f.

* Revert "add InteractiveSession and SessionManager (#2290) (#2293) (#2315)"

This reverts commit 6ac197b.

---------

Co-authored-by: Vamsi Manohar <reddyvam@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@YANG-DB YANG-DB YANG-DB approved these changes

@penghuo penghuo penghuo approved these changes

@pjfitzgibbons pjfitzgibbons Awaiting requested review from pjfitzgibbons

@ps48 ps48 Awaiting requested review from ps48 ps48 is a code owner

@kavithacm kavithacm Awaiting requested review from kavithacm kavithacm is a code owner

@derek-ho derek-ho Awaiting requested review from derek-ho derek-ho is a code owner

@joshuali925 joshuali925 Awaiting requested review from joshuali925 joshuali925 is a code owner

@dai-chen dai-chen Awaiting requested review from dai-chen dai-chen is a code owner

@rupal-bq rupal-bq Awaiting requested review from rupal-bq

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric is a code owner

@vmmusings vmmusings Awaiting requested review from vmmusings

@Swiddis Swiddis Awaiting requested review from Swiddis Swiddis is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az is a code owner

@MaxKsyunz MaxKsyunz Awaiting requested review from MaxKsyunz MaxKsyunz is a code owner

@Yury-Fridlyand Yury-Fridlyand Awaiting requested review from Yury-Fridlyand Yury-Fridlyand is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha is a code owner

@forestmvey forestmvey Awaiting requested review from forestmvey forestmvey is a code owner

@acarbonetto acarbonetto Awaiting requested review from acarbonetto acarbonetto is a code owner

@GumpacG GumpacG Awaiting requested review from GumpacG GumpacG is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@penghuo @YANG-DB