v3/integrations/nrgrpc: Error message not found and Stack trace is incorrect

[Dilip-Nandakumar]

Description

On navigating to errors section under Events we are able to find the error response recored for grpc api calls.

However on clicking on a specific error we cannot find the error message and stack trace is incomplete.

We can only find the grpc error status code response code 2.

Steps to Reproduce

Simulate an error response from grpc handler in the example

Expected Behavior

1. Error message to be displayed along with grpc error status code
2. Full stack trace to be displayed

NR Diag results

Your Environment

Package	Version
github.com/newrelic/go-agent/v3	v3.8.1
github.com/newrelic/go-agent/v3/integrations/nrgrpc	v1.1.0
google.golang.org/grpc	v1.31.0

Reproduction case

Additional context

1. We are only fetching the status code from the error message returned by the hander in the nrgrpc interceptor

[newrelic-eheinlein]

Hi @Dilip-Nandakumar! Sorry for not getting back to you sooner. I know you also created a Support ticket for this issue, and I am currently working with our Support team to investigate. I would like to carry forward with future communication via the Support ticket, although I will come back here and add final details in case someone comes across this same issue in the future and wants to know what the resolution is.

[newrelic-eheinlein]

hi @Dilip-Nandakumar - I have been working on reproducing this issue and will be filing a bug for it. However, I don't think my stack trace seems to be problematic when I repro the issue - could you post what you are seeing and what you expect it to be so I can compare?

[Dilip-Nandakumar]

Current StackTrace In New Relic

go-agent/v3/integrations/nrgrpc/nrgrpc_server.go (nrgrpc.UnaryServerInterceptor.func2:85)
go-agent/v3/integrations/nrgrpc/example/sampleapp/sampleapp.pb.go (sampleapp._SampleApplication_DoUnaryUnary_Handler:265)
path/pkg/mod/google.golang.org/grpc@v1.15.0/server.go (grpc.(*Server).processUnaryRPC:982)
path/pkg/mod/google.golang.org/grpc@v1.15.0/server.go (grpc.(*Server).handleStream:1208)
path/pkg/mod/google.golang.org/grpc@v1.15.0/server.go (grpc.(*Server).serveStreams.func1.1:686)
/usr/local/go/src/runtime/asm_amd64.s (runtime.goexit:1373)

Expected StackTrace

The stack trace was printed using fmt.Sprintf("%+v", err)

main.(*Server).DoUnaryUnary
        go-agent/v3/integrations/nrgrpc/example/server/server.go:32
github.com/newrelic/go-agent/v3/integrations/nrgrpc/example/sampleapp._SampleApplication_DoUnaryUnary_Handler.func1
        go-agent/v3/integrations/nrgrpc/example/sampleapp/sampleapp.pb.go:263
github.com/newrelic/go-agent/v3/integrations/nrgrpc.UnaryServerInterceptor.func2
        go-agent/v3/integrations/nrgrpc/nrgrpc_server.go:85
github.com/newrelic/go-agent/v3/integrations/nrgrpc/example/sampleapp._SampleApplication_DoUnaryUnary_Handler
        go-agent/v3/integrations/nrgrpc/example/sampleapp/sampleapp.pb.go:265
google.golang.org/grpc.(*Server).processUnaryRPC
        path/mod/google.golang.org/grpc@v1.15.0/server.go:982
google.golang.org/grpc.(*Server).handleStream
        path/mod/google.golang.org/grpc@v1.15.0/server.go:1208
google.golang.org/grpc.(*Server).serveStreams.func1.1
        path/mod/google.golang.org/grpc@v1.15.0/server.go:686
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1373(45398)

Missing in Current StackTrace

main.(*Server).DoUnaryUnary
        go-agent/v3/integrations/nrgrpc/example/server/server.go:32

The expectation is to get the full stacktrace along with the handler function from where the error is thrown.

[newrelic-eheinlein]

Thank you for the additional information! That is exactly what I needed. I am adding this bug to our board to be prioritized. We are unfortunately somewhat shorthanded right at the moment so I can't make any promises about how it will be prioritized, but we will update this issue if and when we do a release addressing the bug.

[gunturaf]

having this same issue, can NewRelic give us update about this bug? thanks! cc @jodeev @newrelic-eheinlein

[RichVanderwal]

Thanks, @guntraf ! From your reports, it looks like skipping the stop of the stack trace, which generally keeps the New Relic Agent from needlessly appearing in the traces, is working against us in this case.

We'll get a reproduction going and determine the best path forward in the case of the nrgrpc integration.

[odannyc]

I am also running into this issue. I'd really like to see the grpc error message in the Errors tab in NewRelic. Currently I am only seeing the grpc code int.

@RichVanderwal Can you provide us with an update on this? It's been close to a month since it was stated that the team was looking into it.

Thanks!

[nr-swilloughby]

There are two separate issues reported here. We're working on the error message issue, but I do have an answer on the stack trace problem. After reviewing the code and running some tests we have determined that the error reporting code's handling of the call stack trace is working as intended, but we see where there may be some confusion about the intended behavior.

The Go agent removes the topmost set of calls that come from modules within the agent itself, so we report the functions in the application itself and don't clutter the report with the calls inside the agent that are merely reporting that error. You're seeing a number of your application's functions omitted from the reported stack trace as well, but that is because your application is the example code supplied along with the Go Agent, which is located in the Go Agent's source tree, and so gets automatically removed.

You should not see this in an actual production application (i.e., one whose source code is not located in the New Relic Go Agent source tree). Can you confirm that this is the case by trying with an application which is not located in the Agent's source tree?

Thanks

[nr-swilloughby]

With the changes we made, it seems error messages are being reported all the way out to the UI now. In the screenshots below, I had the DoUnaryUnary call return an error with the message string "this is an error". These changes will be in the next release.

[Ak-x]

Closing this issue as complete due to @nr-swilloughby 's fix.