Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gRPC streaming bugfix #260

Merged
merged 9 commits into from
Feb 8, 2021

Conversation

alertedsnake
Copy link
Contributor

@alertedsnake alertedsnake commented Dec 16, 2020

Description

Fix for a bug I introduced when rewriting the gRPC instrumentation where child spans weren't properly being tracked as part of the same trace. Also there weren't tests for streaming responses before, so I've added one, and several to verify child spans are handled properly in both unary and streaming responses.

Fixes #257

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Added new tests:

  • test_create_two_spans
  • test_create_span_streaming
  • test_create_two_spans_streaming

Also verified manually via exporting spans to Jaeger as part of a test application based on the sample server code.

Does This PR Require a Core Repo Change?

  • No.

Checklist:

See contributing.md for styleguide, changelog guidelines, and more.

  • Followed the style guidelines of this project
  • Changelogs have been updated
  • Unit tests have been added

@alertedsnake alertedsnake requested review from a team, aabmass and lzchen and removed request for a team December 16, 2020 05:21
Base automatically changed from master to main January 29, 2021 19:28
@alertedsnake
Copy link
Contributor Author

Is there anything you need me to do to help the process along here? This gRPC part of this is not really usable as-is, and I'd like to help with other things, but I'm blocked by these PRs.

Copy link
Contributor

@codeboten codeboten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code changes LGTM!

Michael Stella added 6 commits February 4, 2021 13:02
@alertedsnake alertedsnake force-pushed the grpc-streaming-bugfix branch from 86a5608 to 4332e7f Compare February 4, 2021 18:03
@alertedsnake
Copy link
Contributor Author

So for the record, it's also very frustrating when the lint check passes when the PR is made but sometime before it's merged, the linter rules get updated, and now a significant amount of the code fails the checks.

Is there an announcement of these changes somewhere I could subscribe to?

# Handle streaming responses separately - we have to do this
# to return a *new* generator or various upstream things
# get confused, or we'll lose the consistent trace
def _intercept_server_stream(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Single-use function here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what the issue is?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should have been Nit: single-use function here.

Having a function that is used only once has the disadvantage of making the reader make a mental copy of the arguments passed to it, having to look for the function code, mentally pasting the arguments, reading the function code and then going back to the starting point without any of the advantages of not having repeated code that a function provides.

Anyways, it is still a nitpick, you can leave it as is if you please.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this must be a function, for the reason explained in a the comment.


except Exception as error:
# pylint:disable=unidiomatic-typecheck
if type(error) != Exception:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please explain why is this necessary? record_exception expects an Exception. Even so, if this check was appropriate, wouldn't it be better to use isinstance?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes - in the gRPC world bare Exception instances are thrown by the system when you call grpc.abort(), and some of those shouldn't be treated as actual errors, for example the HTTP 404 series responses which are returned to the client but not recorded as errors in a trace.

This allows us to record anything except those as uncaught errors, since it's really unlikely anything else would throw a bare exception itself. It's not ideal but there just isn't any other way to distinguish these at this point in the program flow.

Using isinstance() would catch application exceptions here, which we want to raise normally.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I see. This is unusual code, but necessary because of a particularity of gRPC that the reader may not know about. I'll approve, but please add this explanation as a comment here. 👍

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same exact code is on line 263, with a comment describing why it was done this way. Happy to duplicate the comment here if that would be preferred.

self.assertEqual(parent_span.name, rpc_call)
self.assertIs(parent_span.kind, trace.SpanKind.SERVER)

# Check version and name in span's instrumentation info
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: redundant comment

@codeboten codeboten merged commit 5fc26d3 into open-telemetry:main Feb 8, 2021
@alertedsnake alertedsnake deleted the grpc-streaming-bugfix branch February 8, 2021 19:16
lzchen pushed a commit that referenced this pull request Feb 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

gRPC server instrumentation creates multiple traces on streaming requests
3 participants