feat(multi-stream-support) Add the support for multiple local video streams. #1870

jallamsetty1 · 2022-02-03T21:54:30Z

This feature is behind 'sendMultipleVideoStreams' config.js flag and is currently supported only on clients running in Unified plan mode.

…treams. This feature is behind 'sendMultipleVideoStreams' config.js flag and is currently supported only on clients running in Unified plan mode.

paweldomas

Left few questions/comments. I'm trying to understand what's the final approach going to look like. Now it seems slightly different to what we were talking about.

JitsiConference.js

modules/RTC/TPCUtils.js

modules/sdp/LocalSdpMunger.js

modules/RTC/JitsiLocalTrack.js

paweldomas · 2022-02-04T02:22:35Z

modules/RTC/TPCUtils.js

+                && t.currentDirection === MediaDirection.INACTIVE);
+
+        // If a track of a given media type is added for the first time, replace the track on the first sender. This
+        // will be called when the user joins the call muted but unmutes during the call.


how do you know it's okay to just pick the first one in the multi track scenario? were we not supposed to generate dedicated mlines for each local track and re-use them as tracks are added/remove/replaced? I thought that the recvonly mlines will not be used for sending local tracks, but dedicate 'sendonly' mlines will be created?

When we add a transceiver by adding a new m-line in the remote description without a ssrc attached, the browser will create a transceiver with the direction as RECVONLY since the local track hasn't been added yet and the current direction will be INACTIVE since there is no media flowing yet. Current direction is the differentiator here. After the track is added and reneg is done, the transceiver direction will change to sendonly.

paweldomas · 2022-02-04T02:26:18Z

modules/sdp/SDP.js

+ * @param {Mediatype} mediaType media type of the new source that is being added.
+ */
+SDP.prototype.addMlineForNewLocalSource = function(mediaType) {
+    const mid = this.media.length;


As mentioned in another comment, I think if we put source names as mid for the sender mlines that would make it easier to find transceivers and recycle the mlines.

No I don't think this is a good idea. The same mids show up in both the local and remote description and these mids do not make sense when you are trying to generate new m-lines for new remote sources. They wouldn't follow a pattern and recycling them will not be possible.

They would be dedicated to local tracks and can be recycled for that purpose, but again if you think it's easier this way then that's ok. Still I think it will be more error prone. The if conditions that rely on ordering, finding media type and inactive or recvonly is hard to understand and follow without tons of comments.

The idea is to use the first available sender when you are adding/replacing it on the peerconnection. I still don't get what you mean by dedicated m-lines. Having one fixed per JitsiLocalTrack means there will be many unused ones since every device change for a given source produces a new JitsiLocalTrack and will be a new m-line.

modules/xmpp/JingleSessionPC.js

types/auto/modules/sdp/SDP.d.ts

types/auto/modules/xmpp/JingleSessionPC.d.ts

…nsceiver on unmute.

zsinba · 2022-02-08T10:53:04Z

There is no documentation on how to use multistreaming support.
Does that mean I can use the built-in camera and another external camera on one computer, and the remote user can see both of my videos?
How to invoke it now

paweldomas · 2022-02-08T14:33:22Z

@zsinba it's not finished yet, we'll post an update once it's ready for testing

zsinba · 2022-02-09T08:15:17Z

Do you have an estimated release date? @paweldomas

zsinba · 2022-02-22T00:51:31Z

Has this been released?

paweldomas · 2022-02-22T13:38:00Z

@zsinba the estimated release date is around end of March

zsinba · 2022-02-27T07:07:36Z

@zsinba the estimated release date is around end of March
I mentioned this feature at the end of 2019.
I'm really looking forward to this。

Thanks。

zsinba · 2022-03-03T13:50:00Z

HowAbouIt

zsinba · 2022-03-25T07:52:14Z

@zsinba the estimated release date is around end of March
I mentioned this feature at the end of 2019.
I'm really looking forward to this。

Thanks。

Hello， Can We Have a try now?

zsinba · 2022-05-16T07:11:29Z

Hello all. how about it now?

paweldomas · 2022-05-16T15:45:38Z

Hello all. how about it now?

Yes, it is possible to start testing it now. I will put more detailed instructions in this thread:

https://community.jitsi.org/t/multiple-video-stream-per-endpoint-changes-to-the-presenter-mode/105850

The instructions should be there within a day or two, but long story short you need to:

pull every repo to the latest version
enable "multi stream mode" in the JVB: https://github.com/jitsi/jitsi-videobridge/blob/master/jvb/src/main/resources/reference.conf#L318
enable "source name signaling" in jitsi-meet config.js: https://github.com/jitsi/jitsi-meet/blob/master/config.js#L91
enable "send multiple streams" in jitsi-meet config.js: https://github.com/jitsi/jitsi-meet/blob/master/config.js#L95

If you have any followup questions please use the community forum and the thread linked above.

zsinba · 2022-05-17T14:04:56Z

你太棒了。

I have updated the Jitsi of Docker version to the latest version, can it be used for testing?

zsinba · 2022-05-18T06:36:59Z

I tried it out and it was great.
It combines desktop streams and local camera streams into one screenshot to push out.
More than that, if I have a computer with two screens (Windows display extension, one master screen, one secondary screen), is there any way I can share both screens with my remote friend ? I want to give two screen to remote.

paweldomas · 2022-05-18T12:52:07Z

I tried it out and it was great. It combines desktop streams and local camera streams into one screenshot to push out.

Uhm, I'm confused about this part. It is actually not supposed to combine local camera and desktop into one stream anymore.

More than that, if I have a computer with two screens (Windows display extension, one master screen, one secondary screen), is there any way I can share both screens with my remote friend ? I want to give two screen to remote.

You could eventually do that, but you'd have to modify jitsi-meet yourself.

zsinba · 2022-05-18T13:11:57Z

I may not have made myself clear.

I downloaded and ran the latest version of Jitsi of docker; I could share My desktop and open my camera at the same time. They will be pushed to the remote end in the same stream. as follows:

My question is:
My Windows 11 (OS) has two screens, is it possible for me to share both screens and use both streams

Thank you very much for your constant reply. Thank you for your answer too.

zsinba · 2022-05-18T13:15:10Z

ou could eventually do that, but you'd have to modify jitsi-meet yourself.

I consulted about this about two years ago. The response: No. You can't push two streams out at the same time.

paweldomas · 2022-05-18T17:19:11Z

I consulted about this about two years ago. The response: No. You can't push two streams out at the same time.

Right, but this changes with the multi stream mode supported on the lib-jitsi-meet/JVB level. You should be able to do that, but you need to modify jitsi-meet.

About your screenshots, that is not the new presenter mode, that's how it works currently and it's a single stream. Maybe it's not available in the docker release yet.

feat(multi-stream-support) Add the support for multiple local video s…

76f0748

…treams. This feature is behind 'sendMultipleVideoStreams' config.js flag and is currently supported only on clients running in Unified plan mode.

paweldomas reviewed Feb 4, 2022

View reviewed changes

jallamsetty1 added 2 commits February 4, 2022 16:14

squash: Address review comments.

e511513

squash: Use the track index from source name to find the matching tra…

810b688

…nsceiver on unmute.

jallamsetty1 requested a review from paweldomas February 4, 2022 22:43

paweldomas approved these changes Feb 7, 2022

View reviewed changes

jallamsetty1 merged commit c3ab13d into jitsi:master Feb 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(multi-stream-support) Add the support for multiple local video streams. #1870

feat(multi-stream-support) Add the support for multiple local video streams. #1870

jallamsetty1 commented Feb 3, 2022

paweldomas left a comment

paweldomas Feb 4, 2022

jallamsetty1 Feb 4, 2022

paweldomas Feb 4, 2022

jallamsetty1 Feb 4, 2022

paweldomas Feb 4, 2022

jallamsetty1 Feb 4, 2022 •

edited

Loading

zsinba commented Feb 8, 2022

paweldomas commented Feb 8, 2022

zsinba commented Feb 9, 2022

zsinba commented Feb 22, 2022

paweldomas commented Feb 22, 2022

zsinba commented Feb 27, 2022

zsinba commented Mar 3, 2022

zsinba commented Mar 25, 2022

zsinba commented May 16, 2022

paweldomas commented May 16, 2022

zsinba commented May 17, 2022

zsinba commented May 18, 2022

paweldomas commented May 18, 2022

zsinba commented May 18, 2022

zsinba commented May 18, 2022

paweldomas commented May 18, 2022

feat(multi-stream-support) Add the support for multiple local video streams. #1870

feat(multi-stream-support) Add the support for multiple local video streams. #1870

Conversation

jallamsetty1 commented Feb 3, 2022

paweldomas left a comment

Choose a reason for hiding this comment

paweldomas Feb 4, 2022

Choose a reason for hiding this comment

jallamsetty1 Feb 4, 2022

Choose a reason for hiding this comment

paweldomas Feb 4, 2022

Choose a reason for hiding this comment

jallamsetty1 Feb 4, 2022

Choose a reason for hiding this comment

paweldomas Feb 4, 2022

Choose a reason for hiding this comment

jallamsetty1 Feb 4, 2022 • edited Loading

Choose a reason for hiding this comment

zsinba commented Feb 8, 2022

paweldomas commented Feb 8, 2022

zsinba commented Feb 9, 2022

zsinba commented Feb 22, 2022

paweldomas commented Feb 22, 2022

zsinba commented Feb 27, 2022

zsinba commented Mar 3, 2022

zsinba commented Mar 25, 2022

zsinba commented May 16, 2022

paweldomas commented May 16, 2022

zsinba commented May 17, 2022

zsinba commented May 18, 2022

paweldomas commented May 18, 2022

zsinba commented May 18, 2022

zsinba commented May 18, 2022

paweldomas commented May 18, 2022

jallamsetty1 Feb 4, 2022 •

edited

Loading