Connection issue to redis deployed in k8s

[benjie91]

Summary

Connection issue to redis deployed in k8s.

Relevant information

I have setup a redis-sentinel database using bitnami/redis helm chart. However, on connecting to it, it always say plugin health check failed. No helpful logs as well to debug this issue.

[mikhail-vl]

Hello @benjie91. Thank you for submitting the issue.

You should be able to see the connection errors in the grafana log:

grafana    | t=2020-08-13T05:55:52+0000 lvl=eror msg="Metric request error" logger=context userId=0 orgId=1 uname= error="rpc error: code = Unknown desc = dial tcp 127.0.0.1:6379: connect: connection refused" remote_addr=172.23.0.1

Have you tried to connect to Sentinel port 26379 or to Redis port 6379? Datasource supports only Redis connection. Sentinel and Cluster API will be introduced in upcoming versions.

[benjie91]

Yep I had checked the log messages and it mentioned:
... uname=admin error="Failed to check plugin health: Health check failed"
My connection string is as follow: redis://xxx-redis-master-0.xxx-redis-headless.xxx.svc.cluster.local:26379/0
I have also tried the redis port but the result is still the same.

If you need more info, my grafana instance is also deployed on kubernetes and they are able to talk to each other as I tried pinging the redis url too from the grafana instance. Other things that I have tried include portforwarding the redis port to my local computer and using localhost to connect too in case the connection is made from my computer instead of the server.

[mikhail-vl]

@benjie91 Port 26379 is Sentinel port and it's not supported in the current version.

Sometimes, Grafana after failed health-check got stuck and may not accept the new URL parameter. Could you please try to recreate Datasource as redis://xxx-redis-master-0.conan-redis-headless.xxx.svc.cluster.local:6379/0 and check the log files.

Is it the correct link I can follow to install and try the bitnami/Redis helm chart? https://docs.bitnami.com/tutorials/deploy-redis-sentinel-production-cluster/.

Also, just want to confirm that your are using Grafana 7.X.

[benjie91]

Yes, I'm using Grafana v7.1.3! You can also follow that link you provided. I have tried connecting directly to the master redis which is 6379 and the log message is still the same as before.

[mikhail-vl]

@benjie91 Thank you. I will follow the link and will try to reproduce tomorrow.

[mikhail-vl]

Hello @benjie91. I was able to successfully connect to Redis port 6379 when forwarded port using this command

kubectl port-forward svc/SVC-NAME PORT:PORT

Did you provide a password when configured Data source? There was no correct error message displayed when the PING command failed, which is now fixed in master. You can try to build datasource following https://github.com/RedisTimeSeries/grafana-redis-datasource/blob/master/BUILD.md instructions to see if master will work for you. Meanwhile, I will add Sentinel support.

[benjie91]

I was able to resolve the issue. It was not related to the connection but rather because when I copied the plugin into the grafana plugin directory, I did not change the permission the redis driver to executable. It took a while for me to troubleshoot.

chmod +x redis-datasource_linux_amd64

Thanks for your help and great work. Let's close the issue! =)

[mikhail-vl]

@benjie91 Thank you for the update.
I will keep it open to add Sentinel support in the upcoming version.

[mikhail-vl]

@benjie91 Sentinel supported in the upcoming version 1.2.0 and it's available to test in master.
Please follow BUILD instruction if you would like to try it.