ref(replays): increase chunking limit to 15MB #2032
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
olksdr
requested changes
Apr 14, 2023
relay-server/src/actors/store.rs
Outdated
| // Max message size is 1MB. | ||
| let max_message_size = 1000 * 1000; | ||
| // Max message size is 15MB. | ||
| let max_message_size = 1000 * 150000; |
There was a problem hiding this comment.
Shouldn't this be:
Suggested change
| let max_message_size = 1000 * 150000; | |
| let max_message_size = 1000 * 15000; |
if you want to have 15mb?
There was a problem hiding this comment.
Could we also introduce the configuration option for this and not hardcode it?
JoshFerge
force-pushed
the
jferg/increase-chunking-limit
branch
2 times, most recently
from
08e0171 to
fb5fc3d
Compare
olksdr
reviewed
Apr 18, 2023
CHANGELOG.md
Outdated
| @@ -47,6 +47,7 @@ Metrics: | |||
| - Remove global service registry. ([#2022](https://github.com/getsentry/relay/pull/2022)) | |||
| - Apply schema validation to all topics in local development. ([#2013](https://github.com/getsentry/relay/pull/2013)) | |||
| - Lower default max compressed replay recording segment size to 10 MiB. ([#2031](https://github.com/getsentry/relay/pull/2031)) | |||
| - Increase chunking limit to 15MB for replay recordings. ([#2032](https://github.com/getsentry/relay/pull/2032)) | |||
There was a problem hiding this comment.
Please, merge master in, and move this record to the Unreleased section.
JoshFerge
force-pushed
the
jferg/increase-chunking-limit
branch
from
fb5fc3d to
405b4b6
Compare
2 tasks
olksdr
approved these changes
Apr 19, 2023
This was referenced Apr 24, 2023
JoshFerge
added a commit
to getsentry/sentry
that referenced
this pull request
May 2, 2023
After merging getsentry/relay#2032, we've observed that we are no longer processing chunked messages (as intended). We can remove the logic for this as it is no longer used.
JoshFerge
added a commit
that referenced
this pull request
Jun 28, 2023
After merging #2032, we can confirm via our metrics that we no longer hit this logic, as we reject payloads over 10 MiB and set our chunking limit to 15 MiB. _#skip-changelog_ Co-authored-by: Tor <tor.saebjoernsen@sentry.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We were previously chunking kafka messages on the replay recording topic to 1 MiB. We are aiming to eliminate this chunking as it requires us to maintain a redis cluster / constrains us to in-order processing. Other infra at Sentry has successfully experimented with larger kafka messages, so we feel this is safe.
With #2031 this should pretty much mean we no longer chunk any messages. We can remove the chunking logic once this is confirmed after deploy.
We chose 15MiB as the limit, as we're eventually going to combine the replay event with the replay recording, and will want some extra headroom for this in addition to our 10 MiB recording limit.
Depends on ops increasing our kafka payload size limit first.