fix(tracing): ensure sampling decisions apply to correct trace ID #13275
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
core/pdk
core/tracing
cherry-pick kong-ee
3 tasks
nowNick
reviewed
Jun 24, 2024
nowNick
approved these changes
Jun 24, 2024
Kong generates a temporary trace ID when spans are created, this trace ID can be overwritten if tracing headers are passed in the request. Before this change, the global sampling rate from kong.conf was applied to the temporary trace ID, which was only valid for traces created by Kong, and not in distributed tracing scenarios. This commit ensures the sampling decision is only taken once, after headers have been read, when the trace id is final. This ensures Kong's sampling decision matches that of other tracing systems and therefore guarantees that traces are not broken when sampling rates are set to the same values.
|
I removed a small part of the diff (one less change to the tracer). Initially I also wanted to ensure the sampler is only ran once, and removed it on span creation so that the sampling decision is only taken by the OTel plugin. On second thought I realized that could be breaking for any users who rely on the PDK's logic and are not using the OTel plugin, so I reduced the scope of this PR. All we want to fix this is that, during the access phase, both the local (plugin) and the global (kong.conf) sampling rates are taken into account (with precedence on the plugin) to take the sampling decision. |
StarlightIbuki
approved these changes
Jun 26, 2024
ADD-SP
reviewed
Jun 26, 2024
Co-authored-by: Qi <add_sp@outlook.com>
|
Successfully created cherry-pick PR for |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Kong generates a temporary trace ID when spans are created, this trace ID can be overwritten if tracing headers are passed in the request. Before this change, the global sampling rate from kong.conf was applied to the temporary trace ID, which was only valid for traces created by Kong, and not in distributed tracing scenarios.
This commit ensures the sampling decision always happens during the OTel plugin's access phase, so that it is done after headers have been read, when the trace id is final. This ensures Kong's sampling decision matches that of other tracing systems, to prevent broken traces.
Checklist
changelog/unreleased/kongorskip-changeloglabel added on PR if changelog is unnecessary. README.mdIssue reference
KAG-4785