You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi I have Grafana running in a docker container (grafana/grafana) in ECS Fargate Service on AWS and I have been getting memory leaks when I (incorrectly?) enter custom redis commands in grafana.
Context
When I enter a custom command in redis timeseries datasource, there seems to be a memory leak that occurs immediately afterwards. Memory consumption goes from 20% to 95% in a few minutes while spitting out this log message. This is the log from ECS the moment before and while it happens:
t=2020-12-28T07:07:26+0000
lvl=eror
msg=PANIC
logger=plugins.backend
pluginId=redis-datasource
command="should never get here"
query="TS.RANGE stream:9DoF_0001:* 1609139215082 1609139245082"
I traced the error output it to this line of code:
Is there any insight in what might be going on here? I may have written a bad query to redis timeseries (the query obviously doesn't accept wildcards) however I don't think the result should be such a severe memory leak... so I thought I'd bring up this issue.
The text was updated successfully, but these errors were encountered:
@emmanuelnk Thank you for submitting the issue. I will take a look at it.
You are correct, TS.RANGE command requires the key and does not support wildcards. To get time-series from multiple keys you should use TS.MRANGE command and query using labels.
If the command can't be executed properly, it's PANIC in Radix golang client. I will check with the developer on how I can improve it and get back to you.
@emmanuelnk We were not able to reproduce memory leaks using docker containers, we have not tried RCS Fargate. The PANIC was fixed in the Radix client and now you should see the proper error coming from Redis in the data source.
The fix is a part of the upcoming release 1.3.1, which you can BUILD yourself following BUILD instructions to test.
Please let me know if you have any questions.
Hi I have Grafana running in a docker container (grafana/grafana) in ECS Fargate Service on AWS and I have been getting memory leaks when I (incorrectly?) enter custom redis commands in grafana.
Context
When I enter a custom command in redis timeseries datasource, there seems to be a memory leak that occurs immediately afterwards. Memory consumption goes from 20% to 95% in a few minutes while spitting out this log message. This is the log from ECS the moment before and while it happens:
I traced the error output it to this line of code:
grafana-redis-datasource/pkg/redis-custom.go
Line 38 in cb60f57
Is there any insight in what might be going on here? I may have written a bad query to redis timeseries (the query obviously doesn't accept wildcards) however I don't think the result should be such a severe memory leak... so I thought I'd bring up this issue.
The text was updated successfully, but these errors were encountered: