Dynamodb enhanced client keeps spawning sdk-ScheduledExecutor threads

[vasylv1]

Describe the bug

I use sdk v2 2.19.26 and it keeps spawning threads in a WAITING state.
Thread dump shows thousands of threads in WAITING state like this:

"sdk-ScheduledExecutor-0-0" prio=0 tid=0x0 nid=0x0 waiting on condition
java.lang.Thread.State: WAITING
on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59b572af
at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)

sdk-ScheduledExecutor is defined in SdkDefaultClientBuilder in sdk2

Expected Behavior

There is no issue with threads

Current Behavior

Thread dump shows thousands of threads in WAITING state

Reproduction Steps

var dynamoDbClient = DynamoDbClient
.builder()
.region(region)
.credentialsProvider(() -> AwsBasicCredentials.create(accessKey, secretKey))
.overrideConfiguration(ClientOverrideConfiguration.builder()
.apiCallTimeout(DYNAMODB_CLIENT_TIMEOUT)
.retryPolicy(RetryPolicy.defaultRetryPolicy().toBuilder()
.numRetries(DYNAMODB_CLIENT_MAX_ERROR_RETRY)
.build())
.build())
.build();

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.19.26

JDK version used

11

Operating System and version

alpine-zulu

[debora-ito]

@vasylv1 thank you for reaching out.

Having java.lang.Thread.State: WAITING in the logs is not necessarily a bad thing, the threads can be in WAITING state only temporarily.

Thread dump shows thousands of threads

Are creating one new DynamoDB Client per request, or reusing the client?
We recommend you reuse the same instance of client whenever possible.

[vasylv1]

DynamoDB Client is created only once and then reused by all requests.

Having java.lang.Thread.State: WAITING in the logs is not necessarily a bad thing, the threads can be in WAITING state only temporarily.

That is exactly the problem, the threads are getting stuck in this state and are never released.
Here is grafana chart

The memory and threads in WAITING state go down only when the service is restarted on redeploy.
We started to observe this behavior when we migrated our service from aws sdk 1 to aws sdk 2 + enhanced client.

Here is what we see in memory dumps:

Comparing thread dumps after 3 days:

It looks like the threads are slowly getting stuck in WAITING state and those that are in this state are created by AWS SDK2 with "sdk-ScheduledExecutor" prefix.

[debora-ito]

Problem suspect 2 "play.shaded.ahc.io.netty.buffer" is not part of the Java SDK package.

And I can't get any conclusions by looking at the problem suspect 1 and the log snippet you provided.

Can you provide us the heap dump and a self-contained code sample we can use to reproduce the leak?

[vasylv1]

I cannot share the dump file as it was taken from a prod server. Is there anything specific in the dump file that I can give details about? I am open for providing more info if you could specify what is it.

[vasylv1]

My statement: "DynamoDB Client is created only once and then reused by all requests." wasn't true.
The issue is exactly with not closing the client.

[debora-ito]

Great, glad you found the cause.

Memory leak reports are usually associated with creating multiple clients and not closing them after they're not needed.

Marking this to close soon. Let us know if you have any other question.