Any reason _not_ to use repo v3 rev as subscribeRepos sequence number?

[snarfed]

Hi all! So in repo v3, rev is a TID, ie a base32-encoded integer, and it has broadly the same semantics and requirements as the subscribeRepos sequence number. I like the separation between the two in the specs, esp since there are other subscription methods, eg subscribeLabels. As an implementation detail though, can you think of any reason not to reuse a commit's rev as its sequence number?

(As a hint, https://atproto.com/specs/event-stream#sequence-numbers says "For example, they might be timestamps," so I expect the answer is no. 😁)

[DavidBuchanan314]

So I'm currently using a microsecond-resolution timestamp counter as my sequence number, in picopds. It works quite nicely and it's convenient to implement, but I can think of a couple of things that might go wrong:

• It's not strongly guaranteed to be monotonic - for example, having the system clock drift forwards, followed by an NTP re-sync bringing it back to the correct time. (but I suppose this would also be an issue for your rev values in the first place)

• I forgot what the other thing was... If I remember, I'll come back and update this

[DavidBuchanan314]

It doesn't answer the question, but this thread might be handy for context: https://bsky.app/profile/retr0.id/post/3k5pvuta5pb2v

I'll reproduce it here for link-rot-avoidance purposes:

me, quoting a thread announcing repo v3:

If repo revisions are monotonic TIDs, what are we supposed to do about the 10-bit "clock identifier" part of the TID?

There are 3 options I can think of:

1. require it be all zeroes
2. allow arbitrary values but pretend they're zeroes for comparison purposes
3. allow arbitrary values, compare by magnitude (i.e. equivalent to lexicographic comparison of raw TID strings - nice and simple)

This also raises the question of "what do you do when you have two unique commits with the same TID?"

Bryan Newbold:

compare by overall TID.
if you have a commit for a did, and discover a new different commit with same did+tid, reject/ignore it. if you discover two different did+tid commits at the same time, something is fishy/hostile. i guess we should define something, but probably mark repo as suss

[snarfed]

Yes! It seems like the original goal for TIDs was collision resistance, since they're record keys. That makes sense. However, they aren't monotonically increasing across machines, so it's surprising to see them reused here for a use case that needs that, at least per repo.

[snarfed]

For now, to get my own code to repo v3, I guess I can implement a new, storage-based TID generator that provides per-repo monotonicity across machines. I'll probably have it always set clock id to 0.

[bnewbold]

clock identifier: we aren't really using it to prevent collisions as originally described

The recommendation/assumption for TIDs is that they be tracked by a "TID generator clock" which remembers the last value it gave out as state. When generating a new TID, take a timestamp. If it is lower or equal to the previous TID output, then ignore the timestamp and increment the last value by one and output that. This ensures monotonic output. It assumes the clock never gets too out of whack (like a 100 years). If you wanted to keep the clock ID part of the identifier clean, you'd increment the microseconds part instead of incrementing the overall TID value I guess. Often this is done as a thread-safe sequencing service, or even a microservice.

The above is a common pattern, but for atproto we don't require all this and are often chill. TIDs in rkey or commit rev are per-repo, and as a PDS you usually have the previous rev to compare against. The rkey values in records really don't matter that much (at least for now), and it doesn't matter if rev TID is equal, greater, lesser than any rkey TIDs.

The event stream sequence numbers are PDS-scoped (not repo-scoped). If you are doing a single repo PDS I guess it would be easy in some ways to use the rev? But otherwise I wouldn't bother. The model for repo event stream sequence numbers is definitely Kafka offsets, which start from zero and go up monotonically.

We may add the concept of a event stream sequence "epoch"/"era" to handle resets/resharding; and sharding of the stream, possibly using multiple (many?) parallel sequence numbers.

[snarfed]

Thanks! "as a PDS you usually have the previous rev to compare against" is a useful hint here.

Event stream sequencers (eg subscribeRepos connection handlers) are already single-homed (ie served from a single nodes) due to the websocket connection, so requiring them to serially generate sequence numbers PDS-wide seems fine, and easy to keep contained and out of the way.

There isn't a similar inherent constraint that repos are single-homed, though, and most straightforward distributed PDS designs probably wouldn't pin repos to specific nodes. So, they'd need some other place to synchronize generating revs across nodes.

We have to linearize commits per repo anyway though, and loading prev in that process to check that the new rev is higher is easy and makes sense. Thanks!

[DavidBuchanan314]

Another reason more specific to using the repo rev is that revs are only required to be monotonic per-repo (it would be legal for two repos on the same PDS to have the same rev, iiuc). Perhaps you have some logic that would prevent this from being the case, but something to be aware of at least.

[bnewbold]

Yup. I would recommend against using the rev as sequence number for this reason.

It is totally valid to think of using rev for the sequence number: it is weird to have basically sequence numbers per repo, and then again for the event stream. But the scope of ordering is different.