Skip to content

Conversation

@wForget
Copy link
Member

@wForget wForget commented Apr 26, 2022

Why are the changes needed?

close #2419

We need to clean up the ProcBuilder process and engine application when the session is closed.

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before make a pull request

@turboFei turboFei added this to the v1.6.0 milestone Apr 26, 2022
@codecov-commenter
Copy link

codecov-commenter commented Apr 27, 2022

Codecov Report

Merging #2482 (2690b5d) into master (7b70a6a) will increase coverage by 0.08%.
The diff coverage is 60.00%.

@@             Coverage Diff              @@
##             master    #2482      +/-   ##
============================================
+ Coverage     63.46%   63.55%   +0.08%     
  Complexity       69       69              
============================================
  Files           371      371              
  Lines         17582    17710     +128     
  Branches       2345     2358      +13     
============================================
+ Hits          11158    11255      +97     
- Misses         5392     5408      +16     
- Partials       1032     1047      +15     
Impacted Files Coverage Δ
...n/scala/org/apache/kyuubi/engine/ProcBuilder.scala 82.50% <0.00%> (+1.25%) ⬆️
.../org/apache/kyuubi/session/KyuubiSessionImpl.scala 75.34% <0.00%> (-1.05%) ⬇️
...ain/scala/org/apache/kyuubi/engine/EngineRef.scala 77.14% <85.71%> (+0.37%) ⬆️
...ache/kyuubi/operation/KyuubiOperationManager.scala 83.60% <0.00%> (-12.69%) ⬇️
...engine/trino/operation/TrinoOperationManager.scala 82.60% <0.00%> (-3.76%) ⬇️
...la/org/apache/kyuubi/session/AbstractSession.scala 97.11% <0.00%> (-2.89%) ⬇️
...ine/flink/operation/FlinkSQLOperationManager.scala 82.92% <0.00%> (-2.08%) ⬇️
...ine/spark/operation/SparkSQLOperationManager.scala 82.22% <0.00%> (-1.87%) ⬇️
.../apache/kyuubi/client/KyuubiSyncThriftClient.scala 88.58% <0.00%> (-1.24%) ⬇️
...apache/kyuubi/engine/hive/HiveProcessBuilder.scala 84.31% <0.00%> (-0.80%) ⬇️
... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b70a6a...2690b5d. Read the comment docs.

close(!waitCompletion)
}

def shutdown(): Unit = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do not need an another method, just

def close(destroyForcibly: Boolean = waitCompletion): Unit = {

if (shareLevel == CONNECTION && builder != null) {
try {
builder.shutdown()
engineManager.killApplication(builder.clusterManager(), engineRefId)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the ordering is a little confused: we have closed the builder, but get clusterManager from it again. It seems better:

val clusterManager = builder.clusterManager()
builder.close()
engineManager.killApplication(clusterManager)

@ulysses-you
Copy link
Contributor

@wForget thank you for the contribution, I think the title should be refine to:
Release engine during closing kyuubi server session if share level is connection ?

@wForget wForget changed the title [KYUUBI #2419] Destroy the ProcBuilder process and call killApplication during closing session [KYUUBI #2419] Release engine during closing kyuubi server session if share level is connection Apr 28, 2022
@ulysses-you
Copy link
Contributor

thanks, merging to master

ulysses-you added a commit that referenced this pull request Feb 6, 2023
### _Why are the changes needed?_

We do not need force close engine builder and application if open session succeded. The engine itself will shutdown in connection level.

This pr tries to fix regression of #2482

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4241 from ulysses-you/skip-close.

Closes #4241

0b28041 [ulysses-you] style
1636912 [ulysses-you] Only force close engine ref when open session failed

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
ulysses-you added a commit that referenced this pull request Feb 6, 2023
We do not need force close engine builder and application if open session succeded. The engine itself will shutdown in connection level.

This pr tries to fix regression of #2482

- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4241 from ulysses-you/skip-close.

Closes #4241

0b28041 [ulysses-you] style
1636912 [ulysses-you] Only force close engine ref when open session failed

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
(cherry picked from commit 8760eee)
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] For connection level engines if exception occurs in open session phase, it may be orphaned

4 participants