Handle replay events in a queue

[mydea]

Replay events depend a lot on the order they are sent, as the recordings are basically incremental.
this means that a whole bunch of issues can arise when one or multiple payloads are lost/not sent in the correct order.

Some related issues:

• [Replay] Ensure we avoid sending replays when hitting rate limits #6520
• [Replay] Improve Missing Segments #6533
• Replay SDK and client reports #6502

I propose to instead change how we send replays by start to use a queue of outstanding events instead. This could work something like this:

1. When a segment is ready to be sent, it is added to the bottom of the queue.
2. Each e.g. second, we try to send the top item of the queue. If it fails, it remains on the top of the queue, else it is removed.

This means that we can always ensure that all events are sent in order.

There are two additional wrinkles with this:

a) We need to make sure to send all queue items when the user is leaving the page
b) We may want to store unsent queue items in session storage, so we can recover them after e.g. the user was offline

These two are a bit contradictory, as if we send everything on unload, we can't be sure that is has been sent, so we can't really decide if we need to remove it from a local storage 🤔

[Lms24]

We talked about this a little more and we believe that using a queue would help us in multiple aspects (most of which are already outlined in the original pos)t. However, there are some challenges and open questions that are (tangentially) related to the queue we need to think about and reach a decision:

Queue Architecture

"Just building a queue" won't work here because we have to account for events of multiple replays being in the queue at the same time. So we need a way to prioritize for events based on recency vs. events from previous replays that might e.g. be retried (due to network errors or maybe also rate-limiting (see next section)).

We could go with a Map<replayId, {startTimestamp, queue}>, where each replay would get its own queue. When we send new events, we'd first send events of the most recent replay and only later on look at old replays.

Eviction Strategy

The queues need to be bound and hence we'd need to evict either older replays entirely or old events within one replay if it is overflowing.

Storing events Offline

To persist across page reloads, we could store the queues in the local storage (sessionstorage) or save the queues to IndexDB. This comes with its own set of challenges:

• keeping everything in sync
• handling too old events
• performance implications of serializing and storing/reading entries from sessionstorage
• API usability of IndexDB

I'd vote to revisit this topic at a later time.

Coincidentally, we're looking into creating an "offline transport" wrapper around our transports that takes care of storing events while the device is offline and attempting resending them when online again (#6403).

Rate Limiting

Extracted this to its own issue, as it's arguably only potentially related: #6710

[Lms24]

Update after a meeting regarding the queue and missing segments:

Let's prioritize Replay integrity (i.e. no missing segments) over extensively retrying and pausing replays: It's okay to cancel/stop a recording in case we hit an error or e.g. the queue is full. This will ofc shorten the replay but we don't get missing segments. So let's keep things simple and go with an in-memory, linear queue:

• We add events to send into this queue and pick events to send from it.
• If we get an error response (other than rate limit) or we can't send b/c we're offline, we retry sending this segment, before we move on to send the next event.
• We only retry for a lmited number of times (3x with exp. backoff).
• If we fail after retrying N times, we empty the entire queue and stop the session/recording
• If the queue is full, we empty the entire queue and stop the session/recording

[mydea]

After some investigation, we decided that we actually don't really need a queue for handling replay events, because this is already basically handled by flushing.

See #6765 for details.