New APIs for cascading VideoRoom publishers

[lminiero]

As the title says, this PR adds new APIs to cascade VideoRoom publishers: this means it allows you to publish in room A on Janus X, and be available as a publisher in room B on Janus Y as well. We've actually been doing something like this for a long time, even though we always used VideoRoom RTP forwarders and Streaming mountpoints for scalability purposes (see this presentation for more information): this PR has the key difference that we're not using different plugins, but only the VideoRoom, meaning that remote publishers created this way can be addressed exactly as local publishers (so as a participant you can be notified about them, subscribe to them, etc.). This also means these remote publishers can be served within the context of multistream subscriptions, which is something that with our previous approach cannot be done.

If you're wondering how this is supposed to work. it's pretty much like this:

1. someone publishes as normal in a room;
2. in a different room and/or on a different server, you call add_remote_publisher to create a fake publisher there, which will return connectivity information;
3. on the source server, you use publish_remotely to instruct the plugin to automatically relay all streams to the remote instance;
4. now the same publisher is also available on the remote instance.

This is it in a nutshell, even though it's a bit more complex than that of course. First of all, you may have noticed that the process is only partly automated, since you still have to invoke APIs yourself to actually tie the source to the remote sink: this was done on purpose, since while I had considered somehow making the VideoRoom able to do this automatically on its own, for this initial integration it made much more sense to rely on simple APIs an orchestrator could take care of instead. This will allow us to experiment with this form or cascading, and in case it makes sense (which may not necessarily be true, since a custom orchestrator can sometimes be better in terms of flexibility) come up with a more integrated approach.

The APIs we've added are the following:

1. add_remote_publisher: as the title says, this is what you can on the target server to setup a fake publisher on behalf of the real one; this API expects the list of streams that should be offered, and their characteristics (e.g., first m-line is an Opus stereo stream, second m-line is a VP8 simulcast stream, etc.), and returns the single port it will bind to for receiving RTP traffic from the source; other users in the room will be notified as if a new PeerConnection is available;
2. update_remote_publisher: this API can be used to update a remote publisher, e.g,, whenever the actual publisher is updated (e.g., audio was removed, a new video stream was added, etc.); this API only expects the list of changes, and doesn't return anything new, since the connectivity info will remain the same; other users in the room will be notified as if a local publisher just renegotiated their PeerConnection;
3. remove_remote_publisher: quite simply, this gets rid of a remote publisher, and is the equivalent of a local publisher hanging up their PeerConnection; as a result, other users in the room will be notified accordingly;
4. publish_remotely: this is used at the source to feed a remote publisher somewhere else, and mostly only needs the IP/port to send RTP packets to; it will automatically create forwarders for all active streams, and automatically do the same when a PeerConnection is renegotiated;
5. listremotes: you can use this API to list all the remotizations in use for a specific publisher (since you can actually make the same publisher available in multiple rooms/servers at the same time);
6. unpublish_remotely: this automatically destroys all forwarders tied to this remotization.

As you can evince from these descriptions, add_remote_publisher, update_remote_publisher and remove_remote_publisher are APIs you use on the targets, while publish_remotely, unpublish_remotely and listremotes on the source. You always configure a target first, and the source next, since the source needs to know where to send packets to. All these APIs are synchronous, which means they're easy to send via Admin API as well (which makes them easy to script in an orchestrator component).

Examples

Here's a simple example of how we can create a remote publisher on a target:

{
	request: "add_remote_publisher",
	room: 4321,
	secret: "adminpwd",
	display: "Remote publisher",
	streams: [
		{
			mid: '0',
			mindex: 0,
			type: 'audio',
			codec: 'opus',
			fec: true,
			stereo: true
		},
		{
			mid: '1',
			mindex: 1,
			type: 'video',
			codec: 'vp8'
		}
	]
}

We'll receive a unique publisher ID (e.g., 6261586119881072, which again will be valid in the specified room), a single port for RTP (as unlike the Streaming plugin, all packets will be multiplexed), and one for RTCP. Assuming the port for RTP is 10000 and RTCP is 10001, and that the original publisher we want to remotize has ID 98765432 in room 1234, this is how we can configure the source accordingly:

{
	request: "publish_remotely",
	room: 1234,
	publisher_id: 98765432,
	secret: "adminpwd",
	remote_id: "test",
	host: "<IP of target>",
	port: 10000,
	rtcp_port: 10001
}

If the source publisher is renegotiated to add a new video stream (e.g., a screensharing) beyond the existing audio and video ones we added already, the source will automatically start sending the packets for the new stream to the port, but the target will ignore them until we notify it accordingly, as this would be needed to also notify subscribers of the new stream being published. As such we can do something like this on the target:

{
	request: "update_remote_publisher",
	room: 4321,
	id: 6261586119881072,
	secret: "adminpwd",
	streams: [
		{
			mid: '2',
			mindex: 2,
			type: 'video',
			codec: 'vp8',
			description: 'Screen',
			disabled: false
		}
	]
}

Notice that update_remote_publisher isn't only used for new streams, but also to notify about streams being gone. In that case, we'd need to send an object that references an existing mindex/mid, and a disabled: true property.

When we want to get rid of the remote publisher, we have to notify both source and target. On the source we'll have to send unpublish_remotely to stop sending packets to the target, using the remote_id we received when we publisher remotely:

{
	request: "unpublish_remotely",
	room: 1234,
	publisher_id: 98765432,
	secret: "adminpwd",
	remote_id: "test"
}

We'll also have to send a remove_remote_publisher on the target, though, otherwise the PeerConnection there will remain alive but without any packets coming in anymore:

{
	request: "remove_remote_publisher",
	room: 4321,
	id: 6261586119881072,
	secret: "adminpwd"
}

As you can see, it's no rocket science, so it should be easy enough to manage. It's easy to also experiment with in the context of a single room on a single server, e.g., to duplicate a publisher in the same room they're in: silly and useless, but helpful to figure out how the APIs work.

What's missing?

This is still experimental (hence the draft PR) but functionally more or less complete. There are a few things that should be improved before it's ready, though.

• First of all, data channels don't currently work: while they may be remotized, they will not be sent by a remote publisher. This is because of how we currently structured the usage of the single RTP port for multiplexing: more specifically, we perform some arithmetic on SSRCs to know which m-index an incoming RTP packet is, so that we can relay it accordingly. That said, SSRCs only exist for RTP, which data channels don't use. A simple approach may be "relay anything you don't recognize as RTP via data channels" but that may be frail and prone to abuse. Another possible approach may be prefixing a data channel payload with an RTP header wihere we can put the SSRC we need, but that may risk getting the packet to exceed the MTU due to the RTP header overhead. As such I haven't decided how to handle that yet. This should now be fixed, using the fake RTP prefix that I described as a potential approach.
• RTP extensions don't currently work with remote publishers either. Due to the way the core now works, extensions are automatically stripped from RTP packets to relay, meaning it's up to plugins to fill in a struct with the extension properties to set in outgoing packets. Considering the intermediate plain RTP layer added for this remote publisher mechanism, we'd have to do something like the core does to turn extensions we can parse in packets to the struct we need to fill, which while doable is a bit of a pain. As such, for this initial version I chose to skip extensions altogether. This should now be fixed, as long as you specify the right extension IDs when adding a remote publisher.
• The remotization of RTCP for video (e.g., keyframe requests) will probably not work when there are multiple video streams being published and remotized. This is because we again use a single port for RTCP, and due to how forwarders (which we're reusing internally) work with latching, we may forward an incoming PLI from a target to the wrong m-line. I plan to fix this by using the same SSRC arithmetic I mentioned above. This should now be fixed, as we now check the SSRC in the PLI and find the right publisher stream it's meant for.
• The SSRC arithmetic itself may need to become more dynamic than it is now. At the moment, I have a fixed starting point (1000) and a fixed step (10). This means that m-line 0 will always have SSRC 1000, m-line 1 1010, etc. Simulcast substreams are additions to the m-line SSRC, so if m-line 1 is simulcast, then there will be three different SSRCs (1010, 1011, 1012) for the different substreams. Since the source and target agree on this convention, it's easy to demultiplex an incoming packet and know how to relay it. That said, it may make sense to make both the starting point and the step dynamic instead: in that case, we'll need to update the APIs so that both source and target know which values are used, so that demultiplexing can take place.

There's probably more that's slipping my mind, now, but I think these are the major things that need to be addressed before we can merge this. That said, please don't wait until that happens before testing this: as anticipated, it already does what it's supposed to do, and the APIs are mostly consolidated, so there's a good chance that when we'll merge very little will change there.

Feedback welcome!

[petarminchev]

Sounds exciting - it could be the solution for a question we had some months ago - the "dynamic" streaming plugin. Since currently streaming plugin has a fixed amount of port/slots and when new publishers come we either need to recreate/destroy streaming plugin or use a huge amount of predefined slots, which potentially wastes ports. With this new API we can add/remove dynamically streams for cascading on other servers. Is our understanding correct? Although not sure about performance, since our impression is Streaming plugin is a little more lean/performant than VideoRoom?

[lminiero]

Yes, that's the idea, even though it's worth pointing out that at the moment the VideoRoom is not optimized for broadcasting as the Streaming plugin. There's no such thing as helper threads, for instance. As such, this effort is helpful to distribute the same publisher across mutliple servers (or multiple rooms), but not really to enhance the performance of a publisher in a specific Janus instance (the VideoRoom plugin is limited in that regard). While we may address that in the future, it's not part of this effort.

[lminiero]

I took care of most of the things in the "what's missing?" list from the original description, so the lack of support for data channels, RTP extensions, and proper RTCP demultiplexing of PLIs. The only thing I didn't work on was making the SSRC arithmetic configurable, as that's way less important at the moment (it's more on the "nice to have"). As such, I consider this effort complete, and plan to merge it soon: please do test and provide feedback before that happens.

[lminiero]

Merging.

[petarminchev]

Does the VideoRoom cascading support SRTP? If not, does it mean the streams are sent not encrypted during the forward transit from Janus Instance A to Janus Instance B? In streaming plugin SRTP seems to be supported. Thanks for any suggestions!

[lminiero]

No, at the moment it doesn't. I don't even know if anyone is actually using the feature as it is. There are other things that may be added too, so maybe in the future, but not in the short term I think.

[shivanshtalwar0]

@lminiero I tested this pr and most of the things work great, but i am curious, what is required for simulcasting in terms of cascading api payload anything specific is needed to pass in add_remote_publisher to make simulcast work?