Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(replays): Add ReplayVideo envelope item type #3105

Merged
merged 87 commits into from
Feb 27, 2024
Merged
Show file tree
Hide file tree
Changes from 69 commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
e775838
feat(replay): combined envelope items
JoshFerge May 31, 2023
dc26a9d
only run in processing mode
JoshFerge May 31, 2023
7033ae0
add integration test. fix processing mode stuff, add store logic
JoshFerge Jun 7, 2023
2c9d294
remove processing mode business, use vec instead of Bytes
JoshFerge Jun 7, 2023
aa0c0ce
fix args in combined store func
JoshFerge Jun 7, 2023
9d21e80
fix tests
JoshFerge Jun 7, 2023
d7d18bb
remove unused test
JoshFerge Jun 7, 2023
5ce488b
merge
JoshFerge Jan 31, 2024
5ad1304
get it compiling
JoshFerge Jan 31, 2024
da62bb8
get tests running
JoshFerge Jan 31, 2024
8dbe24c
remove unused imports
JoshFerge Jan 31, 2024
e29cabf
make clippy even more happy
JoshFerge Jan 31, 2024
3b96c66
add changelog entry
JoshFerge Jan 31, 2024
6f7433e
move logic to store
JoshFerge Feb 6, 2024
6f825d4
fix
JoshFerge Feb 6, 2024
aa4d3be
fix merge
JoshFerge Feb 6, 2024
8d06df8
more cleanup
JoshFerge Feb 6, 2024
d8662fa
Update CHANGELOG.md
JoshFerge Feb 6, 2024
91eeda5
remove feature
JoshFerge Feb 6, 2024
848a513
use flag in header
JoshFerge Feb 8, 2024
6399ce8
merge
JoshFerge Feb 8, 2024
0540710
try to debug
JoshFerge Feb 8, 2024
562c5ef
tests working
JoshFerge Feb 8, 2024
43dd578
dont produce two messages
JoshFerge Feb 9, 2024
b9f7156
cleanup
JoshFerge Feb 9, 2024
c8726f3
remove println
JoshFerge Feb 9, 2024
27b508d
fix lint
JoshFerge Feb 9, 2024
f625668
combined_payload not used in non-processing, allow dead code
JoshFerge Feb 9, 2024
8d0e335
Update relay-server/src/envelope.rs
JoshFerge Feb 9, 2024
fa4afd2
simplify processing feature declare on env item flag
JoshFerge Feb 9, 2024
6383eb0
move logic into one func
JoshFerge Feb 9, 2024
9224202
clean refactor
JoshFerge Feb 9, 2024
f56ab35
add one final test / fix impl
JoshFerge Feb 9, 2024
3b52610
small refactor to produce function
JoshFerge Feb 13, 2024
50de8a2
Merge branch 'master' into jferg/combine-replay-env-items
JoshFerge Feb 13, 2024
b362d06
Merge branch 'master' into cmanallen/replays-add-video-envelope-item
cmanallen Feb 13, 2024
3a71696
Add ReplayVideo envelope item type
cmanallen Feb 13, 2024
1ae7767
Merge branch 'master' into jferg/combine-replay-env-items
JoshFerge Feb 13, 2024
bdb0345
Add video envelope item validator
cmanallen Feb 13, 2024
e7bc952
Add ReplayVideo item type processor
cmanallen Feb 13, 2024
0331a1f
Send optional replay_video field in Kafka payload
cmanallen Feb 13, 2024
8208ea4
Register outcome, rate-limit, and processing error
cmanallen Feb 13, 2024
3805bf5
Merge branch 'jferg/combine-replay-env-items' into cmanallen/replays-…
cmanallen Feb 13, 2024
67c0605
Compute payload through addition
cmanallen Feb 13, 2024
898da00
Add docs
cmanallen Feb 13, 2024
bab9202
Update changelog
cmanallen Feb 13, 2024
7baee11
Re-phrase changelog entry
cmanallen Feb 13, 2024
35dd886
only produce replay events if replay recording / event exists
JoshFerge Feb 14, 2024
acbaa9e
Merge branch 'jferg/combine-replay-env-items' into cmanallen/replays-…
cmanallen Feb 14, 2024
12ce170
Remove redundant boolean condition
cmanallen Feb 15, 2024
70e20f3
Flatten extraction and consider replay_event size when measuring payl…
cmanallen Feb 15, 2024
68c5a02
Replay-event variable is only populated if the flag is enabled
cmanallen Feb 15, 2024
306245a
Remove redundant method
cmanallen Feb 15, 2024
6ad66f9
Remove isize note
cmanallen Feb 15, 2024
2959b9d
Correct boolean condition
cmanallen Feb 15, 2024
b96ca07
Emit outcome if the payload is too large
cmanallen Feb 15, 2024
b363dbf
Merge branch 'jferg/combine-replay-env-items' into cmanallen/replays-…
cmanallen Feb 15, 2024
fb1480f
Rename error and update comments
cmanallen Feb 15, 2024
a44c578
Merge branch 'master' into cmanallen/replays-add-video-envelope-item
cmanallen Feb 15, 2024
0fa4678
Add replay-video test coverage
cmanallen Feb 15, 2024
7a3ee9d
Remove unused test case
cmanallen Feb 15, 2024
c492902
Add support for SDK merged replay-video envelope item
cmanallen Feb 20, 2024
279660e
Set processing items on the envelope item's headers
cmanallen Feb 20, 2024
c2227d4
Update store to accept combined ReplayVideo item type
cmanallen Feb 20, 2024
7db085e
Fix lint
cmanallen Feb 21, 2024
cb024ce
Add ReplayVideo to ReplayGroup
cmanallen Feb 21, 2024
8ffa74a
Add serde_bytes
cmanallen Feb 21, 2024
3d81a06
Add test_replay_videos test module
cmanallen Feb 21, 2024
b49f9d5
Remove unused import
cmanallen Feb 21, 2024
bed786b
Pass drop as an Outcome and allow the outerscope to manage how that o…
cmanallen Feb 21, 2024
ec4539e
Merge branch 'master' into cmanallen/replays-add-video-envelope-item
cmanallen Feb 22, 2024
c95f6a4
Revert "Pass drop as an Outcome and allow the outerscope to manage ho…
cmanallen Feb 22, 2024
e6e193c
ref(replays): video item (#3149)
jjbayer Feb 22, 2024
d2583bb
Merge branch 'cmanallen/replays-add-video-envelope-item' of https://g…
cmanallen Feb 22, 2024
22d2091
Update changelog
cmanallen Feb 22, 2024
10be625
Merge branch 'master' into cmanallen/replays-add-video-envelope-item
cmanallen Feb 23, 2024
10bfd11
Merge branch 'master' into cmanallen/replays-add-video-envelope-item
cmanallen Feb 23, 2024
8dc8503
Produce replay-event to the snuba consumer
cmanallen Feb 23, 2024
3417572
Merge branch 'cmanallen/replays-add-video-envelope-item' of https://g…
cmanallen Feb 23, 2024
152ceb3
Create a new Bytes struct with a copy of the replay-event bytes
cmanallen Feb 26, 2024
5109bb9
Use copy_from_slice
cmanallen Feb 26, 2024
79b0668
Add replay-video produced metric
cmanallen Feb 26, 2024
eb8343c
Merge branch 'master' into cmanallen/replays-add-video-envelope-item
cmanallen Feb 27, 2024
b5b90e5
Abstract replay-event assertion and use in replay-video coverage
cmanallen Feb 27, 2024
af2cde7
Assert retention days
cmanallen Feb 27, 2024
1d7ffdb
Emit a single metric
cmanallen Feb 27, 2024
45c2bb8
Remove copy
cmanallen Feb 27, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
**Features**:

- Add protobuf support for ingesting OpenTelemetry spans and use official `opentelemetry-proto` generated structs. ([#3044](https://github.com/getsentry/relay/pull/3044))
- Adds ReplayVideo envelope-item type. ([#3105](https://github.com/getsentry/relay/pull/3105))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move this to the ## Unreleased section.


**Internal**:

Expand Down
11 changes: 11 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions relay-replays/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ license-file = "../LICENSE.md"
publish = false

[dependencies]
bytes = { version = "1.4.0" }
flate2 = "1.0.19"
once_cell = { workspace = true }
relay-common = { path = "../relay-common" }
Expand Down
1 change: 1 addition & 0 deletions relay-server/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ reqwest = { version = "0.11.1", features = [
rmp-serde = "1.1.1"
rust-embed = { version = "8.0.0", optional = true }
serde = { workspace = true }
serde_bytes = { version = "0.11.14" }
serde_json = { workspace = true }
smallvec = { workspace = true, features = ["drain_filter"] }
sqlx = { version = "0.7.3", features = [
Expand Down
43 changes: 42 additions & 1 deletion relay-server/src/envelope.rs
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,8 @@ pub enum ItemType {
ReplayEvent,
/// Replay Recording data.
ReplayRecording,
/// Replay Video data.
ReplayVideo,
/// Monitor check-in encoded as JSON.
CheckIn,
/// A standalone span.
Expand Down Expand Up @@ -167,6 +169,7 @@ impl ItemType {
Self::Profile => "profile",
Self::ReplayEvent => "replay_event",
Self::ReplayRecording => "replay_recording",
Self::ReplayVideo => "replay_video",
Self::CheckIn => "check_in",
Self::Span => "span",
Self::OtelSpan => "otel_span",
Expand Down Expand Up @@ -213,6 +216,7 @@ impl std::str::FromStr for ItemType {
"profile" => Self::Profile,
"replay_event" => Self::ReplayEvent,
"replay_recording" => Self::ReplayRecording,
"replay_video" => Self::ReplayVideo,
jjbayer marked this conversation as resolved.
Show resolved Hide resolved
"check_in" => Self::CheckIn,
"span" => Self::Span,
"otel_span" => Self::OtelSpan,
Expand Down Expand Up @@ -509,6 +513,11 @@ pub struct ItemHeaders {
#[serde(default, skip)]
replay_combined_payload: bool,

/// The parsed replay-event and replay-recording payloads.
/// NOTE: This is internal-only and not exposed into the Envelope.
#[serde(default, skip)]
replay_video_events: Option<(Vec<u8>, Vec<u8>)>,
cmanallen marked this conversation as resolved.
Show resolved Hide resolved

/// Contains the amount of events this item was generated and aggregated from.
///
/// A [metrics buckets](`ItemType::MetricBuckets`) item contains metrics extracted and
Expand Down Expand Up @@ -597,6 +606,7 @@ impl Item {
routing_hint: None,
rate_limited: false,
replay_combined_payload: false,
replay_video_events: None,
source_quantities: None,
sample_rates: None,
other: BTreeMap::new(),
Expand Down Expand Up @@ -652,7 +662,9 @@ impl Item {
} else {
DataCategory::Profile
}),
ItemType::ReplayEvent | ItemType::ReplayRecording => Some(DataCategory::Replay),
ItemType::ReplayEvent | ItemType::ReplayRecording | ItemType::ReplayVideo => {
Some(DataCategory::Replay)
}
ItemType::ClientReport => None,
ItemType::CheckIn => Some(DataCategory::Monitor),
ItemType::Span | ItemType::OtelSpan => Some(if indexed {
Expand Down Expand Up @@ -771,6 +783,17 @@ impl Item {
self.headers.replay_combined_payload = combined_payload;
}

/// Returns the payload's replay video events.
#[cfg(feature = "processing")]
pub fn replay_video_events(&self) -> Option<(Vec<u8>, Vec<u8>)> {
self.headers.replay_video_events.clone()
cmanallen marked this conversation as resolved.
Show resolved Hide resolved
}

/// Set the replay video events attribute for this item.
pub fn set_replay_video_events(&mut self, replay_event: Vec<u8>, replay_recording: Vec<u8>) {
self.headers.replay_video_events = Some((replay_event, replay_recording));
}

/// Sets sample rates for this item.
pub fn set_sample_rates(&mut self, sample_rates: Value) {
if matches!(sample_rates, Value::Array(ref a) if !a.is_empty()) {
Expand Down Expand Up @@ -864,6 +887,7 @@ impl Item {
| ItemType::ClientReport
| ItemType::ReplayEvent
| ItemType::ReplayRecording
| ItemType::ReplayVideo
| ItemType::Profile
| ItemType::CheckIn
| ItemType::Span
Expand Down Expand Up @@ -898,6 +922,7 @@ impl Item {
ItemType::MetricMeta => false,
ItemType::ClientReport => false,
ItemType::ReplayRecording => false,
ItemType::ReplayVideo => false,
ItemType::Profile => true,
ItemType::CheckIn => false,
ItemType::Span => false,
Expand Down Expand Up @@ -1733,6 +1758,22 @@ mod tests {
assert_eq!(items[0].ty(), &ItemType::ReplayRecording);
}

#[test]
fn test_deserialize_envelope_replay_video() {
let bytes = Bytes::from(
"\
{\"event_id\":\"9ec79c33ec9942ab8353589fcb2e04dc\",\"dsn\":\"https://e12d836b15bb49d7bbf99e64295d995b:@sentry.io/42\"}\n\
{\"type\":\"replay_video\"}\n\
helloworld\n\
",
);

let envelope = Envelope::parse_bytes(bytes).unwrap();
assert_eq!(envelope.len(), 1);
let items: Vec<_> = envelope.items().collect();
assert_eq!(items[0].ty(), &ItemType::ReplayVideo);
}

#[test]
fn test_deserialize_envelope_view_hierarchy() {
let bytes = Bytes::from(
Expand Down
2 changes: 2 additions & 0 deletions relay-server/src/services/outcome.rs
Original file line number Diff line number Diff line change
Expand Up @@ -366,6 +366,7 @@ pub enum DiscardReason {
InvalidReplayEventNoPayload,
InvalidReplayEventPii,
InvalidReplayRecordingEvent,
InvalidReplayVideoEvent,

/// (Relay) Profiling related discard reasons
Profiling(&'static str),
Expand Down Expand Up @@ -413,6 +414,7 @@ impl DiscardReason {
DiscardReason::InvalidReplayEventNoPayload => "invalid_replay_no_payload",
DiscardReason::InvalidReplayEventPii => "invalid_replay_pii_scrubber_failed",
DiscardReason::InvalidReplayRecordingEvent => "invalid_replay_recording",
DiscardReason::InvalidReplayVideoEvent => "invalid_replay_video",
DiscardReason::Profiling(reason) => reason,
DiscardReason::InvalidSpan => "invalid_span",
}
Expand Down
2 changes: 1 addition & 1 deletion relay-server/src/services/processor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,7 @@ impl ProcessingGroup {
let replay_items = envelope.take_items_by(|item| {
matches!(
item.ty(),
&ItemType::ReplayEvent | &ItemType::ReplayRecording
&ItemType::ReplayEvent | &ItemType::ReplayRecording | &ItemType::ReplayVideo
)
});
if !replay_items.is_empty() {
Expand Down
1 change: 1 addition & 0 deletions relay-server/src/services/processor/event.rs
Original file line number Diff line number Diff line change
Expand Up @@ -458,6 +458,7 @@ fn is_duplicate(item: &Item, processing_enabled: bool) -> bool {
ItemType::Profile => false,
ItemType::ReplayEvent => false,
ItemType::ReplayRecording => false,
ItemType::ReplayVideo => false,
ItemType::CheckIn => false,
ItemType::Span => false,
ItemType::OtelSpan => false,
Expand Down
Loading
Loading