Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid redundant split/merge when roundtripping buckets through project cache #2517

Closed
Tracked by #2463
jan-auer opened this issue Sep 14, 2023 · 1 comment
Closed
Tracked by #2463
Assignees
Labels
maintenance Tasks with this label generally have a lower priority and due date onboarding

Comments

@jan-auer
Copy link
Member

jan-auer commented Sep 14, 2023

Relay splits large batches of buckets and large buckets down into smaller chunks so that the resulting requests and kafka messages never get larger than a configured threshold. This is centralized in the metrics aggregator so that it applies both to processing relays (producing to kafka) and other relays (sending envelopes).

The performance problem occurs in the split_at sub function, which splits a large distribution or set bucket into two smaller ones. This causes suboptimal memory allocations:

  • Splitting the bucket consumes significant time, since for the values a new container needs to be allocated
  • If the metrics cannot be sent, they need to be re-inserted, which consumes a similar amount of time in moving the values back and deallocating the old structure
  • If the metrics can be sent, allocating a temporary structure is redundant, since we could serialize from a view.
@jan-auer jan-auer added the maintenance Tasks with this label generally have a lower priority and due date label Oct 5, 2023
@jan-auer jan-auer self-assigned this Nov 6, 2023
@Dav1dde Dav1dde closed this as completed Dec 5, 2023
@Dav1dde
Copy link
Member

Dav1dde commented Dec 5, 2023

Implemented with #2682

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
maintenance Tasks with this label generally have a lower priority and due date onboarding
Projects
None yet
Development

No branches or pull requests

3 participants