Failed to query data, rpc error

[tomcsi]

Describe the bug
Some graphs of grafana is broken, after I upgraded to grafana 8.1.1 and plugin version to 4.2.1. If I edit the panel, than the graphs are display, but If I save the panel and refresh the full page that panel is empty again. The logs are contains this errors:

t=2021-08-11T18:29:09+0200 lvl=info msg=Profiler logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource enabled=false
t=2021-08-11T18:29:22+0200 lvl=eror msg="Metric request error" logger=context userId=1 orgId=2 uname=user error="failed to query data: Failed to query data: rpc error: code = Unavailable desc = error reading from server: EOF" remote_addr=127.0.0.1
t=2021-08-11T18:29:22+0200 lvl=eror msg="Request Completed" logger=context userId=1 orgId=2 uname=user method=POST path=/api/ds/query status=500 remote_addr=127.0.0.1 time_ms=17 size=34 referer="https://xxxx.xxx.xxxx/d/000000033/example-dashboard4?orgId=2"
t=2021-08-11T18:29:22+0200 lvl=eror msg="Metric request error" logger=context userId=1 orgId=2 uname=user error="failed to query data: Failed to query data: rpc error: code = Unavailable desc = error reading from server: EOF" remote_addr=127.0.0.1
t=2021-08-11T18:29:22+0200 lvl=eror msg="Request Completed" logger=context userId=1 orgId=2 uname=user method=POST path=/api/ds/query status=500 remote_addr=127.0.0.1 time_ms=56 size=34 referer="https://xxxx.xxx.xxxx/d/000000033/example-dashboard4?orgId=2"
t=2021-08-11T18:29:22+0200 lvl=eror msg="Metric request error" logger=context userId=1 orgId=2 uname=user error="failed to query data: Failed to query data: rpc error: code = Unavailable desc = error reading from server: EOF" remote_addr=127.0.0.1
t=2021-08-11T18:29:22+0200 lvl=eror msg="Request Completed" logger=context userId=1 orgId=2 uname=user method=POST path=/api/ds/query status=500 remote_addr=127.0.0.1 time_ms=38 size=34 referer="https://xxxx.xxx.xxxx/d/000000033/example-dashboard4?orgId=2"

Software versions

Grafana	Zabbix	Grafana-Zabbix Plugin
8.1.1	5.2.1	4.2.1

[stdonato]

Same error.
Grafana 8.1.1
Zabbix 5.4.3
Plugin Zabbix 4.2.1

Zabbix log when load dashboard with error:
[Thu Aug 12 17:20:39.799683 2021] [php7:notice] [pid 32547] [client 10.20.30.40:58488] PHP Notice: Parameter "/user" is deprecated. in /usr/share/zabbix/include/classes/validators/CApiInputValidator.php on line 989

[frantakalab]

Similar issue:
Grafana 8.0.6
Zabbix: 5.4.2
Plugin: 4.2.1

t=2021-08-16T07:51:39+0000 lvl=eror msg="Failed to call resource" logger=context userId=1 orgId=1 uname=root error="failed to receive call resource response: rpc error: code = Unavailable desc = transport is closing"
t=2021-08-16T07:51:39+0000 lvl=eror msg="Metric request error" logger=context userId=1 orgId=1 uname=root error="plugin unavailable" remote_addr=xxx.xxx.xxx.xx
t=2021-08-16T07:59:59+0000 lvl=warn msg="Request Origin is not authorized" logger=live origin=https://xxxx appUrl=http://localhost:3000/ allowedOrigins=

[alexanderzobnin]

Hi! Can you set log level to debug and post grafana logs with error?

[stdonato]

Back to normal after update to 4.2.2
Thank's!

[ikogan]

I'm still getting this on 4.2.2 and Grafana 8.1.2. My Zabbix version is 4.4.10:

t=2021-08-25T16:46:58+0000 lvl=dbug msg="Writing result to cache" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource method=item.get
--
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="panic: runtime error: invalid memory address or nil pointer dereference" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xac62fa]" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg= logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="goroutine 131 [running]:" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="github.com/alexanderzobnin/grafana-zabbix/pkg/timeseries.AlignSeriesIntervals(0xc000574400, 0x2, 0x2, 0x11c1d00, 0x30662300, 0xed8b86b01)" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="\t/root/alexanderzobnin/grafana-zabbix/pkg/timeseries/align.go:90 +0x5a" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="github.com/alexanderzobnin/grafana-zabbix/pkg/datasource.(*ZabbixDatasourceInstance).applyDataProcessing(0xc0004b4870, 0xd63e38, 0xc000728000, 0xc0007520f0, 0xc000574400, 0x2, 0x2, 0x2, 0x2, 0xc000885000, ...)" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="\t/root/alexanderzobnin/grafana-zabbix/pkg/datasource/zabbix.go:129 +0x705" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="github.com/alexanderzobnin/grafana-zabbix/pkg/datasource.(*ZabbixDatasourceInstance).queryNumericDataForItems(0xc0004b4870, 0xd63e38, 0xc000728000, 0xc0007520f0, 0xc0001a13d0, 0x2, 0x2, 0xc00073a018, 0x13, 0xc000756060, ...)" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="\t/root/alexanderzobnin/grafana-zabbix/pkg/datasource/zabbix.go:105 +0x265" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="github.com/alexanderzobnin/grafana-zabbix/pkg/datasource.(*ZabbixDatasourceInstance).queryNumericItems(0xc0004b4870, 0xd63e38, 0xc000728000, 0xc0007520f0, 0x2, 0x2, 0x30662300, 0xed8b816a1, 0x11c1d00)" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="\t/root/alexanderzobnin/grafana-zabbix/pkg/datasource/zabbix.go:59 +0x150" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="github.com/alexanderzobnin/grafana-zabbix/pkg/datasource.(*ZabbixDatasource).QueryData(0xc00031a6c0, 0xd63e38, 0xc000728000, 0xc00072a0a0, 0xb71580, 0xd53e40, 0xbc1a00)" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="\t/root/alexanderzobnin/grafana-zabbix/pkg/datasource/datasource.go:121 +0x74b" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="github.com/grafana/grafana-plugin-sdk-go/backend.(*dataSDKAdapter).QueryData(0xc0003258b0, 0xd63e38, 0xc000728000, 0xc00072a000, 0xb71580, 0x0, 0xd53e40)" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="\t/root/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.98.1/backend/data_adapter.go:21 +0x63" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="github.com/grafana/grafana-plugin-sdk-go/backend/grpcplugin.(*dataGRPCServer).QueryData(0xc000325ac0, 0xd63e38, 0xc000728000, 0xc00072a000, 0xc000325ac0, 0x7fda58a76580, 0xc0000cade0)" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="\t/root/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.98.1/backend/grpcplugin/grpc_data.go:47 +0x51" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="github.com/grafana/grafana-plugin-sdk-go/genproto/pluginv2._Data_QueryData_Handler.func1(0xd63e38, 0xc000728000, 0xc02800, 0xc00072a000, 0x18, 0xc00072a050, 0x7fda5819c658, 0x20)" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="\t/root/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.98.1/genproto/pluginv2/backend_grpc.pb.go:191 +0x89" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1(0xd63e38, 0xc000728000, 0xc02800, 0xc00072a000, 0xc000750000, 0xc00071e018, 0xa68906, 0xbf2760, 0xc000750020, 0xc000750000)" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="\t/root/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107 +0xb0" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1(0xd63e38, 0xc000728000, 0xc02800, 0xc00072a000, 0xc000680000, 0xc000682480, 0xc000731b30, 0x40fdb8)" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="\t/root/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x63" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1(0xd63e38, 0xc000728000, 0xc02800, 0xc00072a000, 0xc000750000, 0xc00071e018, 0x473647, 0x49de05, 0xc09ec0, 0xc000728000)" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="\t/root/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xd7" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="github.com/grafana/grafana-plugin-sdk-go/genproto/pluginv2._Data_QueryData_Handler(0xb71580, 0xc000325ac0, 0xd63e38, 0xc000728000, 0xc000726000, 0xc000317a70, 0xd63e38, 0xc000728000, 0xc000720000, 0xa88)" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="\t/root/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.98.1/genproto/pluginv2/backend_grpc.pb.go:193 +0x150" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="google.golang.org/grpc.(*Server).processUnaryRPC(0xc0003296c0, 0xd6adb8, 0xc0003e6900, 0xc000692120, 0xc0003e4180, 0x11a9dc0, 0x0, 0x0, 0x0)" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="\t/root/go/pkg/mod/google.golang.org/grpc@v1.37.1/server.go:1217 +0x52b" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="google.golang.org/grpc.(*Server).handleStream(0xc0003296c0, 0xd6adb8, 0xc0003e6900, 0xc000692120, 0x0)" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="\t/root/go/pkg/mod/google.golang.org/grpc@v1.37.1/server.go:1540 +0xd0c" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc00031cb30, 0xc0003296c0, 0xd6adb8, 0xc0003e6900, 0xc000692120)" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="\t/root/go/pkg/mod/google.golang.org/grpc@v1.37.1/server.go:878 +0xab" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="created by google.golang.org/grpc.(*Server).serveStreams.func1" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:58 pm | t=2021-08-25T16:46:58+0000 lvl=dbug msg="\t/root/go/pkg/mod/google.golang.org/grpc@v1.37.1/server.go:876 +0x1fd" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource
Wed, Aug 25 2021 12:46:59 pm | t=2021-08-25T16:46:59+0000 lvl=dbug msg="plugin process exited" logger=plugins.backend pluginId=alexanderzobnin-zabbix-datasource path=/var/lib/grafana/plugins/alexanderzobnin-zabbix-app/zabbix-plugin_linux_amd64 pid=438 error="exit status 2"
Wed, Aug 25 2021 12:46:59 pm | t=2021-08-25T16:46:59+0000 lvl=eror msg="Failed to call resource" logger=context userId=2 orgId=1 uname=kogan error="failed to receive call resource response: rpc error: code = Unavailable desc = error reading from server: read unix @->/tmp/plugin599831882: read: connection reset by peer"

I've tried migrating this to another node and am getting the same behavior. It doesn't happen immediately but it does seem to happen after a lot of queries are run simultaneously (dashboard load).

Reading through the stack trace I thought maybe disabling data alignment might avoid this code path and yes, indeed, if I disable data alignment this error goes away.

[alexanderzobnin]

Ok, it looks like the reason is the same as for #1279 (bug in the AlignSeriesIntervals() function), so closing #1279.

[alexanderzobnin]

Hm, looks like an original issue is a bit different. But anyway, I fixed bug reported by @ikogan.