-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Load Balancing by Attribute #33660
Comments
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
I believe there are a couple of comments to this:
|
Thanks for the reply @jpkrohling. That's useful insight. I'd like to move our probabilistic sampling of sessions into the collector rather than having this client side. But the sampler configuration can only specify custom attributes for logs not traces. Our target solution is to have load-balanced telemetry across logs and traces, which is sampled based on complete sessions. We want to observe the users sessions so that we can understand the full journey. Do you have any recommendations for how this can be achieved with the current tooling? |
Take a look at the code for the probabilistic sampling processor at contrib. It could be changed to use specific attributes instead of trace ID, which would be sufficient for your use case, if I'm understanding it correctly. |
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
Component(s)
exporter/loadbalancing
Is your feature request related to a problem? Please describe.
In our project want to setup load balanced sampling based on a session ID an attribute.
In our project, we care about RUM and we have the concept of a user session with a session ID set as an attribute on our traces. We sample based on the session ID rather than the trace ID. This way we preserve telemetry of the whole user session.
The sampling is working fine but it is client-side head sampling. As we look to scale our solution we want to move this to the collector and introduce load balancing.
How do we achieve session sampling?
Right now we use the same ID generator for trace IDs as for session and we set it as an attribute. Then we have a head sampling strategy that uses the same logic as the probabilistic head sampler, but rather than applying the decision to the trace ID we apply it to the session ID. This ensures we make the same sampling decision for the whole user session.
What we would like to do is move these sampling decisions off the client and into the collector so that we have more flexibility. Our client is an Android application and making these decisions client side is not a long-term solution because we have to deal with application updates etc.
Following the recommended practice we would like to have a 2 layer collector setup, with the first layer load balancing the second. The issue is that the load balancer only supports decisions based on trace ID or service name.
Given that we want to sample based on session ID (an attribute), then making load balancing decisions on trace ID alone is not enough. We need to load balance telemetry with the same session ID to the same collector instance so that consistent sampling decisions can be made.
It doesn't look like the load balancer currently supports balancing based on an attribute. This is a friendly request to add it!
Describe the solution you'd like
The ability to route telemetry based on attributes in addition to service name and trace ID
Describe alternatives you've considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: