Memory leak in router

[mohitd11]

Describe the bug
Apollo Router is not releasing the allocated memory.

To Reproduce
Steps to reproduce the behavior:

1. Expose apollo router to the traffic.
2. Stop the traffic.
3. When the traffic starts, there is a dip in available memory(which is expected), but when the traffic is stopped, allocated memory is not freed. The available memory dip still remains even when no traffic is flowing through apollo router.

Expected behavior
After the traffic is stopped, the allocated memory should be freed.

Output
Below image shows the traffic pattern in the apollo router:

Below image shows the available memory in the apollo router:

As seen in the above images, the traffic starts around 20:00, and there is a dip in available memory, but when the traffic is stopped 20:30, the allocated memory is not freed. The available memory at this time is 65%.

Below image shows the TCP connection count:

Desktop :

• OS: centos
• Version: 7

Additional context
We are using apollo router version - 1.11.0

[abernix]

The steps that you wrote aren't quite enough to reproduce this for me, but thank you for including the version that you're using. Could you please update to the latest Router version and try again? We've fixed a couple memory-leak related things since 1.11.0 (we're on 1.15.1, now) related to leaks in recent versions and this could be resolved by that.

(As a request, in general when reporting all bugs — especially memory leaks — please try to reproduce using the latest version first!).

[abernix]

Related to the above, I should say that we have observed something like this before and we think we've improved some of it in recent releases. Worth pointing out that in some of the other cases, we've seen that memory indeed hasn't been aggressively freed, but instead has been getting freed in smaller amounts as new allocations are necessary.

Of course, this still isn't ideal. We're looking at putting #2882 into a alpha release for folks to test, though you could build it from source and try running it locally, and we'd appreciate the validation if you're able to reliably reproduce what you're encountering. We're staging some of that on #2969 for an upcoming alpha.

[jutaz]

We're running 1.15.1, but we're also seeing some memory accumulation over time:

You'll notice that at some point it reaches ~100% at which point k8s tells the pod to restart, which takes the memory usage back down.

Unsure what specifically might be causing this, though. Happy to provide more details if that would be helpful.

[Geal]

the next release will have #2882 which should help a lot with memory fragmentation

[garypen]

I think this has been fixed by our change to use jemalloc: #2882. Please re-open if there is still an issue.