Unclear how to determine boundaries of backlog in /sync API?

[joepie91]

When using /sync, it's possible to receive a limited response with a prev_batch token, which can be used with /messages to fetch the omitted backlog. So far, so good.

However, how do I know whether I've reached the end of that omitted backlog?

For example: say that I do a sync "since A", and I receive a limited response of D,E,F, so B and C are omitted. When following the included continuation token, how do I know that I've received C and B, therefore reached the end of that backlog, and there's nothing more to obtain - so that I don't duplicate processing of event A?

I've been unable to find a clear answer to this in the spec, and asking in #matrix-dev also yielded no answers.

[florianjacob]

Related: https://github.com/matrix-org/matrix-doc/issues/574

[richvdh]

For example: say that I do a sync "since A", and I receive a limited response of D,E,F, so B and C are omitted. When following the included continuation token, how do I know that I've received C and B, therefore reached the end of that backlog, and there's nothing more to obtain - so that I don't duplicate processing of event A?

In this example, there are two /sync requests: the first returned events up to A, and a next_batch token: let's call it α. Then, you call /sync again with since=α, and it returns events=[D,E,F], limited=true, prev_batch=φ, next_batch=β.

You are looking to fill the gap between α and φ. To do so, you can call /messages with from=α, to=φ, dir=f (or vice-versa with dir=b, to go in the opposite direction); you may need to call it repeatedly with different from values to cover the whole gap.

I think you know this though, because you've implied you've tried it at #2251 (comment). So, is this just a duplicate of #2251?

[Dominaezzz]

The spec doesn't support passing sync tokens into /messages.

[richvdh]

The spec doesn't support passing sync tokens into /messages.

True. That's #2420.

[richvdh]

I can't understand why this isn't a duplicate of #2251.