[Fix](catalog)Resources should be closed when dropping a Catalog. #59512
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
Member
Author
|
run buildall |
TPC-H: Total hot run time: 32013 ms |
TPC-DS: Total hot run time: 173785 ms |
ClickBench: Total hot run time: 27.03 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Member
Author
|
run buildall |
TPC-H: Total hot run time: 31365 ms |
TPC-DS: Total hot run time: 174157 ms |
ClickBench: Total hot run time: 27.04 s |
Member
Author
|
run external |
Member
Author
|
run buildall |
TPC-H: Total hot run time: 31913 ms |
TPC-DS: Total hot run time: 171533 ms |
ClickBench: Total hot run time: 26.83 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Member
Author
|
run cloud_p0 |
Contributor
FE Regression Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 31383 ms |
TPC-DS: Total hot run time: 172886 ms |
ClickBench: Total hot run time: 26.8 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Member
Author
|
run buildall |
morningman
approved these changes
Jan 7, 2026
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
TPC-H: Total hot run time: 31243 ms |
TPC-DS: Total hot run time: 172757 ms |
ClickBench: Total hot run time: 27.26 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
hubgeter
approved these changes
Jan 7, 2026
CalvinKirs
added a commit
to CalvinKirs/incubator-doris
that referenced
this pull request
Jan 7, 2026
16 tasks
yiguolei
pushed a commit
that referenced
this pull request
Jan 8, 2026
…Catalog. (#59642) … #59512 #59063 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
zzzxl1993
pushed a commit
to zzzxl1993/doris
that referenced
this pull request
Jan 13, 2026
…ache#59512) ### What problem does this PR solve? When creating an AWS SDK V2 async client (e.g. S3AsyncClient), the SDK requires a thread pool to manage asynchronous task scheduling, timeouts, and retries (e.g. ScheduledExecutorService or async executor). If no executor is explicitly provided, the AWS SDK creates its own internal thread pool, which is expected to be shut down when client.close() is invoked. ### Issue In Doris, when using Paimon Catalog, some catalog implementations provide an empty close() method. As a result, when a user executes DROP CATALOG: starting with Hadoop 3.4, the AWS SDK was upgraded to v2. Since apache#57226 upgraded HDFS to 3.4.2, the catalog runs into this issue. Some Catalog instance is discarded,AWS SDK client.close() is never called,The internally created thread pool cannot be shut down. This leads to a thread leak FYI aws/aws-sdk-java-v2#3746 ### Problem Analysis The lifecycle of Catalog creation and destruction is complex and managed internally by Paimon Doris cannot reliably intervene in the call chain to enforce AWS SDK client.close() If each Catalog creates its own AWS SDK client with an internally managed thread pool, threads will continue to leak as Catalogs are repeatedly created and dropped ### Solution This PR resolves the issue by introducing a shared executor strategy: A Doris-managed shared thread pool is explicitly passed when creating AWS SDK V2 clients This prevents the AWS SDK from implicitly creating per-client internal thread pools The lifecycle of the executor is fully controlled by Doris and no longer depends on Paimon Catalog’s close() implementation With this approach, even if a Paimon Catalog’s close() method is a no-op, the system will no longer leak threads. None
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
When creating an AWS SDK V2 async client (e.g. S3AsyncClient), the SDK requires a thread pool to manage asynchronous task scheduling, timeouts, and retries (e.g. ScheduledExecutorService or async executor).
If no executor is explicitly provided, the AWS SDK creates its own internal thread pool, which is expected to be shut down when client.close() is invoked.
Issue
In Doris, when using Paimon Catalog, some catalog implementations provide an empty close() method. As a result, when a user executes DROP CATALOG:
starting with Hadoop 3.4, the AWS SDK was upgraded to v2. Since #57226 upgraded HDFS to 3.4.2, the catalog runs into this issue.
Some Catalog instance is discarded,AWS SDK client.close() is never called,The internally created thread pool cannot be shut down.
This leads to a thread leak
FYI aws/aws-sdk-java-v2#3746
Problem Analysis
The lifecycle of Catalog creation and destruction is complex and managed internally by Paimon
Doris cannot reliably intervene in the call chain to enforce AWS SDK client.close()
If each Catalog creates its own AWS SDK client with an internally managed thread pool, threads will continue to leak as Catalogs are repeatedly created and dropped
Solution
This PR resolves the issue by introducing a shared executor strategy:
A Doris-managed shared thread pool is explicitly passed when creating AWS SDK V2 clients
This prevents the AWS SDK from implicitly creating per-client internal thread pools
The lifecycle of the executor is fully controlled by Doris and no longer depends on Paimon Catalog’s close() implementation
With this approach, even if a Paimon Catalog’s close() method is a no-op, the system will no longer leak threads.
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)