Query on type [traces] return error when searching with the word Unhealthy

[patrickdung]

What happened:
Searching on Traces sometimes return error
Works
namespace="default"
namespace="unhealthy"
reason="Info"
reason="unhealthy"

Query error
namespace="Unhealthy"
reason="Unhealthy"

What you expected to happen:
No error for the query

How to reproduce it (as minimally and precisely as possible):
Always

Anything else we need to know?:

1. The spans are from kspan https://github.com/weaveworks-experiments/kspan
2. Send to otel-collector
3. Send to Data-Prepper
4. Send to Opensearch 2.16.0
5. Grafana plugin query Opensearch

Environment:

• Grafana version: 11.2.1
• OpenSearch version: Opensearch 2.16.0
• Plugin version: 2.19.0

[patrickdung]

An error occurred within the plugin
Object
status:500
statusText:""
data:Object
message:"An error occurred within the plugin"
messageId:"plugin.downstreamError"
statusCode:500
traceID:""
config:Object
url:"api/ds/query?ds_type=grafana-opensearch-datasource&requestId=explore_s7u"
method:"POST"
data:Object
hideFromInspector:false
traceId:undefined
message:"An error occurred within the plugin"

checked the logs on the grafana pod

{"PPL index":"otel-v1-apm-span","indices":"otel-v1-apm-span","level":"info","logger":"plugin.grafana-opensearch-datasource","msg":"Creating new client","t":"2024-09-27T19:41:10.855211269Z","timeField":"@timestamp","version":"2.16.0"}
{"error":"runtime error: index out of range [0] with length 0","level":"error","logger":"plugin.grafana-opensearch-datasource","msg":"panic triggered","stack":"goroutine 442 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:26 +0x5e\ngithub.com/grafana/grafana-plugin-sdk-go/backend.handlePanic({0x142f860, 0xc000169d58})\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.2/backend/serve.go:115 +0x25\ngithub.com/grafana/grafana-plugin-sdk-go/backend.defaultGRPCMiddlewares.WithRecoveryHandler.func4.1({0x100?, 0xf8?}, {0x142f860?, 0xc000169d58?})\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/recovery/options.go:36 +0x27\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.recoverFrom({0x180ca58?, 0xc000b08450?}, {0x142f860?, 0xc000169d58?}, 0x43ae89?)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/recovery/interceptors.go:54 +0xea\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.UnaryServerInterceptor.func1.1()\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/recovery/interceptors.go:30 +0x67\npanic({0x142f860?, 0xc000169d58?})\n\t/usr/local/go/src/runtime/panic.go:785 +0x132\ngithub.com/grafana/opensearch-datasource/pkg/opensearch.processTraceListResponse(0x23e2ee0?, {0xc0000123a0, 0xe}, {0xc00012e8a0, 0x23}, {{0x23e2ee0, 0x0, 0x0}, {0x0, 0x0}, ...})\n\t/drone/src/pkg/opensearch/response_parser.go:431 +0xd65\ngithub.com/grafana/opensearch-datasource/pkg/opensearch.(*responseParser).parseResponse(0xc0008787b0)\n\t/drone/src/pkg/opensearch/response_parser.go:149 +0x828\ngithub.com/grafana/opensearch-datasource/pkg/opensearch.(*luceneHandler).executeQueries(0xc000182c80, {0x180ca58, 0xc000b08720})\n\t/drone/src/pkg/opensearch/lucene_handler.go:297 +0x347\ngithub.com/grafana/opensearch-datasource/pkg/opensearch.(*queryRequest).execute(0xc000622bc8, {0x180ca58, 0xc000b08720})\n\t/drone/src/pkg/opensearch/query_request.go:49 +0x574\ngithub.com/grafana/opensearch-datasource/pkg/opensearch.(*OpenSearchDatasource).QueryData(0xc000696208, {0x180ca58, 0xc000b08720}, 0xc00077c380)\n\t/drone/src/pkg/opensearch/opensearch.go:74 +0x1ab\ngithub.com/grafana/grafana-plugin-sdk-go/internal/automanagement.(*Manager).QueryData(0x180ca58?, {0x180ca58, 0xc000b08720}, 0xc00077c380)\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.2/internal/automanagement/manager.go:46 +0x3d9\ngithub.com/grafana/grafana-plugin-sdk-go/backend.(*dataSDKAdapter).QueryData.func1({0x180ca58, 0xc000b086c0})\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.2/backend/data_adapter.go:30 +0xb1\ngithub.com/grafana/grafana-plugin-sdk-go/backend.wrapHandler.errorWrapper.func1({0x180ca58, 0xc000b086c0})\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.2/backend/adapter_utils.go:49 +0x2b\ngithub.com/grafana/grafana-plugin-sdk-go/backend.wrapHandler.logWrapper.func2({0x180ca58, 0xc000b086c0})\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.2/backend/adapter_utils.go:123 +0x6a\ngithub.com/grafana/grafana-plugin-sdk-go/backend.metricWrapper.func2({0x180ca58, 0xc000b086c0})\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.2/backend/adapter_utils.go:76 +0x90\ngithub.com/grafana/grafana-plugin-sdk-go/backend.wrapHandler.tracingWrapper.func3({0x180ca58, 0xc000b08690})\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.2/backend/adapter_utils.go:105 +0x7ee\ngithub.com/grafana/grafana-plugin-sdk-go/backend.wrapHandler({0x180ca58?, 0xc000b08510?}, {0x1, {0xc0004a0180, 0x1d}, {0xc000012446, 0x6}, 0xc000b04340, 0x0, 0xc0004e5ad0, ...}, ...)\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.2/backend/adapter_utils.go:33 +0x146\ngithub.com/grafana/grafana-plugin-sdk-go/backend.(*dataSDKAdapter).QueryData(0xc0005be4e0, {0x180ca58?, 0xc000b08450?}, 0xc000182b90)\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.2/backend/data_adapter.go:27 +0x148\ngithub.com/grafana/grafana-plugin-sdk-go/backend/grpcplugin.(*dataGRPCServer).QueryData(0x2373330?, {0x180ca58?, 0xc000b08450?}, 0x3?)\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.2/backend/grpcplugin/grpc_data.go:48 +0x25\ngithub.com/grafana/grafana-plugin-sdk-go/genproto/pluginv2._Data_QueryData_Handler.func1({0x180ca58?, 0xc000b08450?}, {0x1409000?, 0xc000182b90?})\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.2/genproto/pluginv2/backend_grpc.pb.go:203 +0xcb\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.UnaryServerInterceptor.func1({0x180ca58?, 0xc000b08450?}, {0x1409000?, 0xc000182b90?}, 0xc000612c00?, 0x14e2af8?)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/recovery/interceptors.go:34 +0x8a\ngoogle.golang.org/grpc.getChainUnaryHandler.func1({0x180ca58, 0xc000b08450}, {0x1409000, 0xc000182b90})\n\t/go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1211 +0xb2\ngithub.com/grafana/grafana-plugin-sdk-go/backend.defaultGRPCMiddlewares.(*ServerMetrics).UnaryServerInterceptor.UnaryServerInterceptor.func9({0x180ca58, 0xc000b08450}, {0x1409000, 0xc000182b90}, 0xc0005e22a0?, 0xc000b04300)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/server.go:22 +0x28a\ngoogle.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0x180ca58, 0xc000b08450}, {0x1409000, 0xc000182b90}, 0xc0005e22a0, 0x78?)\n\t/go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1202 +0x85\ngithub.com/grafana/grafana-plugin-sdk-go/genproto/pluginv2._Data_QueryData_Handler({0x12fdfa0, 0xc0005be950}, {0x180ca58, 0xc000b08450}, 0xc00077c180, 0xc0005e3480)\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.2/genproto/pluginv2/backend_grpc.pb.go:205 +0x143\ngoogle.golang.org/grpc.(*Server).processUnaryRPC(0xc0001c5600, {0x180ca58, 0xc000b08360}, {0x1817300, 0xc00098e000}, 0xc0006090e0, 0xc000613290, 0x2373270, 0x0)\n\t/go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1393 +0xe11\ngoogle.golang.org/grpc.(*Server).handleStream(0xc0001c5600, {0x1817300, 0xc00098e000}, 0xc0006090e0)\n\t/go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1804 +0xe8b\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1()\n\t/go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1029 +0x7f\ncreated by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 40\n\t/go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1040 +0x125\n","t":"2024-09-27T19:41:10.867187537Z"}

[patrickdung]

Screenshot

[patrickdung]

sample-span-record.json
Attached a sample span record, hostname redacted.
It is captured from Grafana UI

[iwysiu]

Hi @patrickdung ! Thanks for the detailed bug report, I'm currently looking at this, and based on the error I suspect we're not handling what happens when we don't get results properly. I'll take a deeper look at this and see what I find

[iwysiu]

After more investigating, it looks like the line where we parse the trace group from the results assumes that trace_group will contain at least one bucket, and it's getting an index out of range error because it does not. We could handle this by checking the length beforehand and setting it to "" instead if there is no bucket.

I'll add this to our backlog to hopefully be scheduled soon.

[iwysiu]

Actually, this should be fixed by #445 , which will be in our next release!