Support wildcard/pattern subscribes

[TimEvens]

There are many use-cases where we would like to subscribe to a namespace tuple prefix to receive existing and new tracks that match that subscribe prefix. In these use-cases, the publisher knows the available tracks and does not need to rely on the subscribe full track information to choose which track to send. Instead, there is a controlled set of tracks that the publisher would send based on the subscribe match.

Example:

Subscribe to namespace tuple <meeting id>/participants/primary audio/* , name is null.

The relay would:

1. lookup existing published tracks that matches the subscribe, such as 1234/participants/primary audio/endpointId=456 and start forwarding to the subscriber this content. The track alias would reflect the full track name. The subscriber would be notified via control channel of what the full track name is for the given track alias.
2. Upon new announcement, relay would forward to publisher the subscribe prefix requested. The publisher would then choose which tracks to transmit. The publisher would send to the relay (aka subscriber) the track full name information via the control channel with a track alias along with data objects that include the track alias. The original subscriber would receive the new track alias via data objects and associate it to the control channel full name information that was sent upon initial match.

[afrind]

There was a previous issue #253 for "wildcard subscribe" that got closed when SUBSCRIBE_ANNOUNCES was landed #498. Perhaps the difference is that you expect only a single track to be forwarded per SUBSCRIBE?

Your proposal conflicts with the current design where the subscriber is in control of the Track Alias space - if the publisher or relay chooses one, it could be taken by an in-flight SUBSCRIBE. It also suffers from the HoL blocking problem where the objects with an unknown alias arrive at the subscriber before the control message linking them.

I'm curious what you mean by "controlled tracks" - are these fixed or will the publisher choose one of several based on some application logic.

[TimEvens]

There was a previous issue #253 for "wildcard subscribe" that got closed when SUBSCRIBE_ANNOUNCES was landed #498. Perhaps the difference is that you expect only a single track to be forwarded per SUBSCRIBE?

It's not about finding the announces to subscribe because as defined announces only contain namespace, sans name. Thus, the announce doesn't contain the full track name/alias. In the wildcard use case, the subscribe would result in receiving multiple track aliases (full track names) that matched a prefix for the subscribe. This is kind of a problem with MoQT design that has subscriber providing the track alias. In this case, it's the publisher needing to provide the track alias. Therefore, this doesn't really work with wildcard subscribes where the subscriber will have no idea of what the track alias will be.

The nice thing about track alias in data objects header is that it does allow for us to distinguish the different sources (aka tracks that matched) but it doesn't provide us the opaque details of the namespace tuple/name. Even if we subscribe announce, it would not help unless publisher announced full track names with alias. Even if we do announce full track names, I'm not sure subscribe announce is the right approach to get information on matching wildcard (or pattern/regex based) subscribes, which is really a transaction. A possible improved approach is to directly feedback to the subscriber the matching tracks with associated information (e.g., the namespace tuple and name opaque data). This would be a good use-case for subscribe ID. The subscription will have to be updated based on new announces that happen post subscribe... hence there is a need to update the information... until unsubscribe. Again, subscribe ID is a good use-case here to support the transaction/state.

Your proposal conflicts with the current design where the subscriber is in control of the Track Alias space - if the publisher or relay chooses one, it could be taken by an in-flight SUBSCRIBE.

Exactly, which is mentioned above.... it's a fundamental problem with MoQT design that has subscribers defining the track alias. IMO, it needs to change.

It also suffers from the HoL blocking problem where the objects with an unknown alias arrive at the subscriber before the control message linking them.

Not sure I follow your HoL issue here. Can you elaborate on why your implementation would HoL?.? An unknown track alias received that references a known subscribe ID will be dropped (metrics will count how many times this happens) in most cases till track alias information is known. Track alias information on matching subscribes would be sent via control messaging, which is over the control stream that has the highest priority. There are race conditions, which can be mitigated with short buffering of these unknown track aliases data objects... which does not result in HoL.

I'm curious what you mean by "controlled tracks" - are these fixed or will the publisher choose one of several based on some application logic.

Probably bad wording, but what I meant by "controlled tracks" is that we know them ahead of time via the catalog service. They are provisioned tracks, often conveyed via catalog service. Problem is that we have no idea what the track alias will be for these tracks. While each subscriber could subscribe to the 1 million tracks, the intent for this issue is to ask/callout that we find it inefficient to be required to subscribe to a million tracks when a simple wildcard/pattern subscribe could have sufficed.

There are discussions that we should be using a global track alias, which would help to mitigate the track alias issue, but that only works with an internal/custom implementation, which breaks interoperability... unless the draft is updated to handle this scenario.

[afrind]

Individual Comment:

announces only contain namespace, sans name

This is true, though there's nothing stopping an application from defining a namespace that is sufficiently unique and either a blank or otherwise fixed track name. See MoQ Chat. Would this meet your use case?

Exactly, which is mentioned above.... it's a fundamental problem with MoQT design that has subscribers defining the track alias. IMO, it needs to change.

[Chair Hat on]:

The working group has established consensus on this issue based on discussions in 2023. This was prior to the introduction of Subscribe ID, so the Track Alias served both to identify the subscription and as a compressed identifier for the track name. To meet the requirement that all downstream subscriptions could use the same identifier, we added the Retry Track Alias error code to Subscribe Error. I'll note there was skepticism that this single identifier would make a meaningful performance difference, compared to say, QUIC stream IDs and per-hop encryption. I'm individually curious to see a benchmark comparison.

There are other issues with discussion of Track Alias now that show others are also unhappy with it, but in a different way, see #597.

[Chair Hat off]:

An unknown track alias received that references a known subscribe ID will be dropped

It's not clear to me how that works, since Objects only contain Track Alias and not the Subscribe ID in the current version of the draft.

Can you elaborate on why your implementation would HoL?

The publisher would send to the relay (aka subscriber) the track full name information via the control channel with a track alias along with data objects that include the track alias.

The objects containing the new Track Alias will race this control message. If the objects arrive first, they are likely not usable until the control message arrives and have to be buffered.