feat(quotas): Require TransactionProcessed quota #1517
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
October 5, 2022 15:29
Serde was (de)serialising this wrongly leading to inconsistencies.
If metrics have not yet been extracted a transaction event needs to have DataCategory::TransactionProcessed quota available for ingestion. Otherwise we are in the quota enforcement part of the relay event processing pipeline and should check and count transaction indexing quota. Transaction processing quota however is only counted in the metrics part and not in event ingestion. INGEST-1581 INGEST-1653
flub
force-pushed
the
flub/billing-tx-quota
branch
from
ca7c553 to
9de287c
Compare
Instructions and example for changelogFor changes exposed to the Python package, please add an entry to For changes to the Relay server, please add an entry to
To the changelog entry, please add a link to this PR (consider a more descriptive message): - Require TransactionProcessed quota. ([#1517](https://github.com/getsentry/relay/pull/1517))If none of the above apply, you can opt out by adding #skip-changelog to the PR description. |
jjbayer
reviewed
Oct 6, 2022
added 2 commits
October 10, 2022 16:34
Even when metrics have already been extracted we want these two to be consistent.
|
I believe this is ready, please review. It does not yet have integration tests, I think for this it needs to be based on top of #1521 since it needs this to increment the used quotas of the metric buckets. |
jan-auer
reviewed
Oct 11, 2022
jan-auer
reviewed
Oct 11, 2022
jan-auer
changed the title
added 3 commits
October 14, 2022 14:05
This is the best we can do without also having metrics bucket counting happening.
jjbayer
added a commit
that referenced
this pull request
Oct 17, 2022
Apply rate limits to metrics buckets from the `transactions` namespace. # Background Dynamic Sampling introduces a new type of quota, `transactions_processed`, which defines how many transactions should be "processed", i.e. whether or not we should extract metrics from them. This quota is always >= the `transactions` quota, which defines how many transactions should be stored & indexed. The new data category should not only apply to incoming transaction payloads (see #1517), but also to metrics buckets, which may have been extracted by an upstream Relay. Moreover, processing Relays must _count_ the number of transactions for which metrics have been dropped, and create outcomes for them, in the same way that we create accepted outcomes (see getsentry/sentry#39236). # Data Flow Processing Relays check rate limits against Redis. Currently the only actor that has access to this rate limiter is the `EnvelopeProcessor`, and we want to make use of its thread pool to make the blocking calls to Redis. This makes it necessary to redirect metrics buckets to the processor before sending them to the `EnvelopeManager` for publishing: ## Old Flow (still applies under certain conditions) ``` ┌──────────┐ ┌────────────┐ ┌───────────────┐ │Aggregator│ │ProjectCache│ │EnvelopeManager│ └────┬─────┘ └─────┬──────┘ └──────┬────────┘ │ │ │ │ FlushBuckets │ │ ├───────────────►│ │ │ │ SendMetrics │ │ ├──────────────►│ │ │ │ ``` ## New Flow ``` ┌──────────┐ ┌────────────┐ ┌─────────────────┐ ┌─────┐ ┌───────────────┐ │Aggregator│ │ProjectCache│ │EnvelopeProcessor│ │Redis│ │EnvelopeManager│ └────┬─────┘ └─────┬──────┘ └────────┬────────┘ └──┬──┘ └──────┬────────┘ │ │ │ │ │ │ FlushBuckets │ │ │ │ ├─────────────►│ │ │ │ │ │ RateLimitFlushBuckets │ │ │ │ ├──────────────────────►│ │ │ │ │ ├────────────►│ │ │ │ │ │ │ │ │ │◄────────────┤ │ │ │ │ │ │ │ │ │ │ │ │ │ │ SendMetrics │ │ │ ├─────────────┬──────────►│ │ │ │ │ │ ``` # Business Logic 1. ProjectCache received a `FlushBuckets` from the metrics aggregator. 2. Count the number of transactions that contributed to these buckets (see below). 2. Check if cached rate limits apply. 3. If so, drop transaction-related buckets, generate outcomes, and send the buckets to `EnvelopeManager`. 4. Otherwise, if processing mode is enabled, send buckets to the `EnvelopeProcessor`. 5. The processor communicates the transaction count to redis and applies the rate limit if reached. 6. The processor sends a message to the project cache to update the cached rate limits. 7. Drop transaction-related buckets, generate outcomes, and send the buckets to `EnvelopeManager`. ## Counting processed transactions * For `d:transactions/duration@millisecond`, increment the counter in redis by the length of the bucket's value. This is an accurate count of the number of transactions that contributed to the bucket. * For any other metric, do not increment the counter in redis, but still enforce the rate limit if the quota is currently exceeded. # Not in this PR * Enforcement of rate limits on the fast path. * Enforcements of rate limits on metrics buckets in the `sessions` namespace. Co-authored-by: Jan Michael Auer <mail@jauer.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If metrics have not yet been extracted, a transaction event needs to
have
DataCategory::TransactionProcessedquota available for ingestion.Otherwise, we are in the quota enforcement part of the Relay event
processing pipeline and should check and count transaction indexing
quota. Transaction processing quota however is only counted in the
metrics part and not in event ingestion.
Background
Rate limiting and quota enforcement has to apply to legacy plans that
did not have processed transaction functionality, and to new plans
that introduce them.
Organization on Legacy Plan
There is a single quota for indexed transactions. We need to extract
metrics for indexed transactions so that there is backfilled history at
the time the customer switches to the new plan. This requires to extract
metrics for all accepted transactions.
Organization on New Plan
This plan has a dedicated quota for processed transactions. Indexing
transactions requires quota for processed transactions.
There are no metrics extracted for such dropped events.
this transaction. Otherwise, drop the transaction event and any
transaction metrics bucket.
Extracted metrics and the processed transaction quotas are not
impacted by this.
sampled transaction event. Otherwise, drop the transaction event.
Algorithm
Terminology:
tx: Existing quota for indexed transactions.txp: New quota for processed transactions.Preconditions:
txp:=txFlow:
CheckEnvelopein the fast-pathtxpis limited ortxp==0: drop; else: keeptx += 1,txp += 0txortxpare limited.txpis limited.Dependencies
TransactionProcessingquota to be set for all legacyplans.