[Ingest Investigation] Reservoir of samples #54449
Assignees
Comments
TBS1996
added a commit
to getsentry/relay
that referenced
this issue
Sep 29, 2023
Relay implementation of the reservoir project: getsentry/sentry#54449 Reservoir bias uses a type of `SamplingRule` which will sample all matches until a certain limit has been reached. This limit is tracked both locally on each relay, and with a global synchronized one in redis that procesisng relays can have access to. The redis counter will update the local counter if it's available. The counters are saved on the `Project` struct, with a Mutex<BTreeMap<Ruleid, i64>> . When we send an envelope for processing we send its corresponding project counters in the `ProcessEnvelopeState` to the `EnvelopeProcessorService`. There, in the `dynamic-sampling` crate, we introduce a `ReservoirEvaluator`, which will, when a reservoir rule is matching, check if the rule has been reached or not by using the local counters we sent or if applicable the redis global count. The `ReservoirEvaluator` also takes care of updating both redis and the local counter. After the limit is reached, the rule is no longer valid and will be ignored, so that the normal `SampleRate` and `Factor` variant of `SamplingValue` will apply. Sentry is responsible for removing the reservoir rule from the `SamplingConfig` when it has reached its limit. Whenever we receive a new `ProjectConfig`, we remove all the reservoir counters from its project that are no longer in the `DynamicSamplingConfig`. regarding the use of mutex: We use try_lock to avoid getting blocked in case the mutex is already in use. There's two reasons it might be blocked. 1. Another thread is handling a reservoir rule from the same project at the same time. 2. We are in the process of removing counters from that same project.
|
Ingest-side is finished 🤞 Next phase will be implementation of sentry-side and a test to see if it works, waiting for @RaduW for status here. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
We need to communicate to Relay that the user would like to upsample specific transaction with applied query filters.
E.g. upsample transaction X with custom tag value Y and transaction duration more than Z.
Ideally, this would not just multiply the base sample rate for a limited amount of time, but it would set it to 100% until we collect 10 samples. (10 is made up number)
Tasks
The text was updated successfully, but these errors were encountered: