Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fail to use DebugProbes.dumpCoroutines with large amount of coroutines #1535

Closed
e5l opened this issue Sep 13, 2019 · 0 comments
Closed

Fail to use DebugProbes.dumpCoroutines with large amount of coroutines #1535

e5l opened this issue Sep 13, 2019 · 0 comments
Assignees
@qwwdfsad

Comments

@e5l
Copy link
Member

e5l commented Sep 13, 2019

We're trying to print stack traces of 52k coroutines in ktor stress tests and fail with OOM.
@qwwdfsad qwwdfsad self-assigned this Sep 13, 2019
qwwdfsad added a commit that referenced this issue Sep 16, 2019
@qwwdfsad
Do not aggregate all coroutine info into a single string before dumpi…
7964e32 
…ng it to the out in DebugProbes.dumpCoroutines

Previously it was done intentionally to reduce synchronization window of DebugProbesImpl, but still have a consistent view of the world and to avoid interleaving with other usages of out.

Apparently, applications with a significant amount of coroutines (e.g. Ktor) may spend a lot of time or even experience OOM in DebugProbes.dumpCoroutines. To make it usable in such applications, we dump coroutines incrementally without releasing a lock in DebugProbesImpl (thus significantly increasing synchronization window) while holding a lock on out to still have a continuous dump in out.

Fixes #1535
@qwwdfsad qwwdfsad mentioned this issue Sep 16, 2019
Merged
qwwdfsad added a commit that referenced this issue Sep 16, 2019
@qwwdfsad
Do not aggregate all coroutine info into a single string before dumpi…
8f3b484 
…ng it to the out in DebugProbes.dumpCoroutines

Previously it was done intentionally to reduce synchronization window of DebugProbesImpl, but still have a consistent view of the world and to avoid interleaving with other usages of out.

Apparently, applications with a significant amount of coroutines (e.g. Ktor) may spend a lot of time or even experience OOM in DebugProbes.dumpCoroutines. To make it usable in such applications, we dump coroutines incrementally without releasing a lock in DebugProbesImpl (thus significantly increasing synchronization window) while holding a lock on out to still have a continuous dump in out.

Fixes #1535
qwwdfsad added a commit that referenced this issue Sep 16, 2019
@qwwdfsad
Do not aggregate all coroutine info into a single string before dumpi…
5579516 
…ng it to the out in DebugProbes.dumpCoroutines

Previously it was done intentionally to reduce synchronization window of DebugProbesImpl, but still have a consistent view of the world and to avoid interleaving with other usages of out.

Apparently, applications with a significant amount of coroutines (e.g. Ktor) may spend a lot of time or even experience OOM in DebugProbes.dumpCoroutines. To make it usable in such applications, we dump coroutines incrementally without releasing a lock in DebugProbesImpl (thus significantly increasing synchronization window) while holding a lock on out to still have a continuous dump.

Fixes #1535
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@qwwdfsad qwwdfsad

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@qwwdfsad @e5l