Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experiencing memory leak in Grafana docker seemingly stemming from this plugin #116

Closed
emmanuelnk opened this issue Dec 28, 2020 · 3 comments · Fixed by #128
Closed

Experiencing memory leak in Grafana docker seemingly stemming from this plugin #116

emmanuelnk opened this issue Dec 28, 2020 · 3 comments · Fixed by #128
Assignees
Labels
bug Something isn't working
Milestone

Comments

@emmanuelnk
Copy link

emmanuelnk commented Dec 28, 2020

Hi I have Grafana running in a docker container (grafana/grafana) in ECS Fargate Service on AWS and I have been getting memory leaks when I (incorrectly?) enter custom redis commands in grafana.

Context

When I enter a custom command in redis timeseries datasource, there seems to be a memory leak that occurs immediately afterwards. Memory consumption goes from 20% to 95% in a few minutes while spitting out this log message. This is the log from ECS the moment before and while it happens:

t=2020-12-28T07:07:26+0000 
lvl=eror 
msg=PANIC 
logger=plugins.backend 
pluginId=redis-datasource 
command="should never get here" 
query="TS.RANGE stream:9DoF_0001:* 1609139215082 1609139245082"

I traced the error output it to this line of code:

log.DefaultLogger.Error("PANIC", "command", err, "query", qm.Query)

Is there any insight in what might be going on here? I may have written a bad query to redis timeseries (the query obviously doesn't accept wildcards) however I don't think the result should be such a severe memory leak... so I thought I'd bring up this issue.

@mikhail-vl mikhail-vl self-assigned this Dec 28, 2020
@mikhail-vl mikhail-vl added the bug Something isn't working label Dec 28, 2020
@mikhail-vl mikhail-vl added this to the Version 1.3 milestone Dec 28, 2020
@mikhail-vl
Copy link
Contributor

@emmanuelnk Thank you for submitting the issue. I will take a look at it.

You are correct, TS.RANGE command requires the key and does not support wildcards. To get time-series from multiple keys you should use TS.MRANGE command and query using labels.

If the command can't be executed properly, it's PANIC in Radix golang client. I will check with the developer on how I can improve it and get back to you.

@mikhail-vl mikhail-vl modified the milestones: Version 1.3, Version 1.3.1 Jan 3, 2021
@mikhail-vl
Copy link
Contributor

PANIC fixed in Radix 3.7.0: mediocregopher/radix#250

@mikhail-vl
Copy link
Contributor

mikhail-vl commented Jan 12, 2021

@emmanuelnk We were not able to reproduce memory leaks using docker containers, we have not tried RCS Fargate. The PANIC was fixed in the Radix client and now you should see the proper error coming from Redis in the data source.

The fix is a part of the upcoming release 1.3.1, which you can BUILD yourself following BUILD instructions to test.
Please let me know if you have any questions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants