MonitorConnectionContext queue cause memory leak

[YoungHu]

Describe the bug

AT 1.1.5+ version, in Monitor class, Queue<MonitorConnectionContext> just have add in startMonitoring and run method, no remove in stopMonitoring method, this will cause memory leak.
we can see 1.14 have remove action but missing in 1.1.5. maybe code lost at this version.

Expected Behavior

remove unused MonitorConnectionContext in queue.

Current Behavior

no remove action. dump file can see lots of MonitorConnectionContext instance。

Reproduction Steps

integrate aws driver and run application monitor the class instance

Possible Solution

No response

Additional Information/Context

No response

The AWS JDBC Driver for MySQL version used

1.1.7

JDK version used

JDK8

Operating System and version

Linux

[crystall-bitquill]

Hi @YoungHu,

Thanks for reaching out and raising this issue.

We'll take a look at this and keep you updated as we investigate.

Thank you for your patience!

[aaronchung-bitquill]

Hi @YoungHu

I saw your pull request and wanted to thank you for contributing!

Before we get it approve and merge in, I'd just like to do some more testing on our side.

Thank you!

[aaronchung-bitquill]

Hi @YoungHu

I took a closer look at the Monitor class and the uses of Queue<MonitorConnectionContext>. While the stopMonitoring() method does not call remove() to remove the context from the queues, the contexts are removed from the queues in the run() method by calling poll() on the queues.
https://github.com/awslabs/aws-mysql-jdbc/blob/main/src/main/user-impl/java/com/mysql/cj/jdbc/ha/plugins/Monitor.java#L150
https://github.com/awslabs/aws-mysql-jdbc/blob/main/src/main/user-impl/java/com/mysql/cj/jdbc/ha/plugins/Monitor.java#L184

I also tried reproduce a memory leak, but so far unable to.
I made a simple Spring Boot application using aws-mysql-jdbc:1.1.7 and profiled it using JProfiler.
Below is a graph of the number of instances of the MonitorConnectionContext class over time.

In the graph, there is a continuous pattern of the number of instances rising, then falling, then rising again. Not captured in the screenshot, but I did not observe any trends of the peaks getting higher over time.

I was hoping you could you provide the following details?

• Is your application using any framework, libraries, and/or connection pools? What are they?
• What type of queries is your application making? Are the long running? Transactional?
• Is your application using any non-default garbage collection settings/options?
• Anything else you think may be useful

Thank you for patience!

[YoungHu]

Hi @YoungHu

I took a closer look at the Monitor class and the uses of Queue<MonitorConnectionContext>. While the stopMonitoring() method does not call remove() to remove the context from the queues, the contexts are removed from the queues in the run() method by calling poll() on the queues. https://github.com/awslabs/aws-mysql-jdbc/blob/main/src/main/user-impl/java/com/mysql/cj/jdbc/ha/plugins/Monitor.java#L150 https://github.com/awslabs/aws-mysql-jdbc/blob/main/src/main/user-impl/java/com/mysql/cj/jdbc/ha/plugins/Monitor.java#L184

I also tried reproduce a memory leak, but so far unable to. I made a simple Spring Boot application using aws-mysql-jdbc:1.1.7 and profiled it using JProfiler. Below is a graph of the number of instances of the MonitorConnectionContext class over time. In the graph, there is a continuous pattern of the number of instances rising, then falling, then rising again. Not captured in the screenshot, but I did not observe any trends of the peaks getting higher over time.

I was hoping you could you provide the following details?

• Is your application using any framework, libraries, and/or connection pools? What are they?
• What type of queries is your application making? Are the long running? Transactional?
• Is your application using any non-default garbage collection settings/options?
• Anything else you think may be useful

Thank you for patience!

Hi @aaronchung-bitquill
we have 8 applications use 1.1.7 version, just one application have this problem.
following details as blew:

• SpringBoot 2.x, Druid 1.2.9, Mybatis 3.5.4.
• just simple sql select by key
• GC setting like this:-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+DisableExplicitGC -XX:+CMSParallelRemarkEnabled

normally， run method will call poll() to remove item in queue, but when debug the code, I find run method will break out sometimes, I have not idea what cause it. after break out, queue will just add not remove.

[aaronchung-bitquill]

Hi @YoungHu

A few additional questions for the application that is experiencing this issue

• Can you give me some details about the database? Is it MySQL or Aurora MySQL? What versions?
• How is the DB URL specified?
• What plugins are being used?
• What database and connection related configurations do you have?
• Are there any other details about this particular application that is different than the others are working without issue?

Also, would u be able to enable driver logging and provide some driver logs?

Thank you

[YoungHu]

Hi @YoungHu

A few additional questions for the application that is experiencing this issue

• Can you give me some details about the database? Is it MySQL or Aurora MySQL? What versions?
• How is the DB URL specified?
• What plugins are being used?
• What database and connection related configurations do you have?
• Are there any other details about this particular application that is different than the others are working without issue?

Also, would u be able to enable driver logging and provide some driver logs?

Thank you

we are using Aurora MySQL 5.7 version,DB url is cluster endpoint

[aaronchung-bitquill]

Hi @YoungHu

I suspect that it might be InterruptedException that is breaking the while loop in the Monitor#run method. But unfortunately, I am still unable to reproduce the issue.

If possible would you be able to provide driver logs of when the issue occurs? That would greatly help us be able to confirm the source of the issue. Otherwise, would you be willing to share a sample app that reproduces the issue?

Thank you

[YoungHu]

Hi @YoungHu

I suspect that it might be InterruptedException that is breaking the while loop in the Monitor#run method. But unfortunately, I am still unable to reproduce the issue.

If possible would you be able to provide driver logs of when the issue occurs? That would greatly help us be able to confirm the source of the issue. Otherwise, would you be willing to share a sample app that reproduces the issue?

Thank you

how to set logger? just like jdbc:mysql:aws://db-cluster-id:3306?key1=value&key2=value2& logger=com.mysql.cj.log.StandardLogger?

[aaronchung-bitquill]

HI @YoungHu

Apologies for the late reply.
Yes, that is correct.

[aaronchung-bitquill]

Hi @YoungHu

I wanted to check in to see how things are going? Were you able to obtain driver logs?

[aaronchung-bitquill]

Hi @YoungHu

We have not heard back from you for some time and wanted to check back in.

If there are no further updates on this ticket in the next 3 days, we will close this ticket. However, if you require further assistance, you can reopen this issue, or create a new one.

Thank you!