Incomplete event with null data replaced by a different event type when replaying the event sequence

[godfrey-altmetric]

This may be related to #1367 but I raise this as a new issue so as to not detract attention from that issue if it is not related.

We're consuming from the bsky.network firehose, and observed a bsky repost event on 2024-11-01, which had null data (something that raised an exception in our code).

A short while later, when debugging this issue, we used the sequence id (minus 1) of the event prior as a cursor to replay the event when connecting to the firehose. When the event for that sequence id was returned, we saw that an entirely different event was returned (a create post, instead of the repost). This event was fully populated, and the only thing that was the same was the user's profile identifier (did:plc:..) and the sequence id.

We noticed that this user is using Bridgy-fed, by the inclusion of bridgyOriginalUrl and bridgyOriginalText. From what Bluesky have told us, this shouldn't be possible, as the relays are responsible for assigning sequence ids, and when subscribing with a previous event, it is retrieved from the event log. Has anyone come across this same issue?

[snarfed]

Hi @godfrey-altmetric, good to see you here! Altmetric is great, I'm a big fan.

Bridgy Fed did indeed have a bug where it would occasionally allocate a seq and store a commit, but not emit that commit event over its firehose to the relay. This was very rare, but it happened. Good find on #1367, that did track it.

I put a workaround/fix in place a few days ago, in snarfed/arroba#34, so hopefully this has stopped. However, I'm surprised that the official relay would change the historical event it emits for a given seq, that seems problematic, and agreed, shouldn't happen.

The relay does indeed allocate and manage its own seqs and events, so I don't know that I can do anything about that, and I don't have immediate plans to do anything more for #1367/snarfed/arroba#34. Let me know if I can help any other way though!

[snarfed]

@godfrey-altmetric any thoughts here? I'm happy to help debug etc, but not sure what I can do otherwise.

[godfrey-altmetric]

Hi @snarfed, we've had some further time to debug this issue, and from what we've discovered is that although we are still receiving events in the firehose with data: null, we managed to identify the root cause of why we were seeing a different event type when replaying the sequence id, which was caused by a subtle difference in how our main consumer processes operations vs how we replayed them.

I think we're good for now, and happy if you want to close this issue.

[snarfed]

Thanks for the followup! Good luck!