Filter spans before sending them to the opentelemetry layer #2894
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment has been minimized.
This comment has been minimized.
the filter is not part of the reloadable tracer (which is on the opentelemetry side), so we use an atomic (lighter than a RwLock), and there's no AtomicF64 in std, so we transmute back and forth with a u64
This was referenced Apr 7, 2023
|
Perf test please |
|
Perf test please |
|
the PR should be good to review now. I merged dev, fixed the tests and removed some cruft, it's ice and small now. |
bnjjj
approved these changes
Apr 17, 2023
Co-authored-by: Gary Pennington <gary@apollographql.com>
SimonSapin
reviewed
Apr 24, 2023
SimonSapin
approved these changes
Apr 24, 2023
Geal
commented
Apr 24, 2023
the span might have been enabled by another layer
Geal
added a commit
that referenced
this pull request
Apr 25, 2023
The Opentelemetry layer accumulates data from all spans before sending it to the opentelemetry span exporters. This means that sampling decisions are done after a lot of data was already allocated. And that this work is done even if no exporter is set up. This adds a sampling filter around the opentelemetry layer, to do the sampling decision as early as possible, and avoid this work entirely if no exporter is configured
Geal
added a commit
that referenced
this pull request
May 15, 2023
Batch of performance improvements: * linux: replace default allocator with jemalloc (#2882) * Add a private part to the Context structure (#2802) * lighter path manipulation in response formatting (#2854) * prevent span attributes from being formatted to write logs (#2890) * Parse Accept headers once instead of three times (#2847) * use a parking-lot mutex in Context (#2885) * Filter spans before sending them to the opentelemetry layer (#2894)
BrynCooke
pushed a commit
that referenced
this pull request
May 18, 2023
If an otel context has been created then spans should be sampled regardless of if the sampler says to sample. The presence of a context means that we got a context from the client. Fixes #3103
BrynCooke
pushed a commit
that referenced
this pull request
May 18, 2023
If an otel context has been created then spans should be sampled regardless of if the sampler says to sample. The presence of a context means that we got a context from the client. Fixes #3103
BrynCooke
pushed a commit
that referenced
this pull request
May 18, 2023
If an otel context has been created then spans should be sampled regardless of if the sampler says to sample. The presence of a context means that we got a context from the client. Fixes #3103
BrynCooke
pushed a commit
that referenced
this pull request
May 18, 2023
If there is no otel context set up we let the otel layer do its thing. If there is a context set up then we can filter early depending on if the span is being sampeld or not. Fixes #3103
This was referenced May 18, 2023
BrynCooke
added a commit
that referenced
this pull request
May 18, 2023
…2894)" (#3111) This reverts commit ecd5d57. The reason for this is that parent based sampling definitely regressed and there are some ongoing questions around correctness. We will look again after the release as this did appear to give a good speedup. Related: #2894 #3103 <!-- start metadata --> **Checklist** Complete the checklist (and note appropriate exceptions) before a final PR is raised. - [ ] Changes are compatible[^1] - [ ] Documentation[^2] completed - [ ] Performance impact assessed and acceptable - Tests added and passing[^3] - [ ] Unit Tests - [ ] Integration Tests - [ ] Manual Tests **Exceptions** *Note any exceptions here* **Notes** [^1]. It may be appropriate to bring upcoming changes to the attention of other (impacted) groups. Please endeavour to do this before seeking PR approval. The mechanism for doing this will vary considerably, so use your judgement as to how and when to do this. [^2]. Configuration is an important part of many changes. Where applicable please try to document configuration examples. [^3]. Tick whichever testing boxes are applicable. If you are adding Manual Tests: - please document the manual testing (extensively) in the Exceptions. - please raise a separate issue to automate the test and label it (or ask for it to be labeled) as `manual test` Co-authored-by: bryn <bryn@apollographql.com>
BrynCooke
pushed a commit
that referenced
this pull request
May 18, 2023
BrynCooke
pushed a commit
that referenced
this pull request
May 18, 2023
BrynCooke
pushed a commit
that referenced
this pull request
May 18, 2023
BrynCooke
pushed a commit
that referenced
this pull request
May 18, 2023
BrynCooke
pushed a commit
that referenced
this pull request
May 19, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
the sampling configuration in the opentelemetry layer only applies when the span closes, so in the meantime a lot of data is created just to be dropped. This experiments with a filter than can sample spans before the opentelemetry layer