Roadmap for mediasoup v3 #227
Comments
|
Hi! Best regards. |
|
hi，guys， i have no idea with the roadmap. i'm using kurento now. i really want to use mediasoup instead,because nodejs seems friendly for me.and mediasoup seems more stable than kurento. but,theres something i really not like. second, i choose kurento ,because i need some CV(computer vision) function. it make technoligy meanable. if we just want VC, theres dozens solution . infact i don't think google can make webrtc project right.it seems someday,they have an idea,and say let's do it. so many "maybe change" api,behavior in the webrtc project. so,i don't think "improve performence" in V3 is important. to be honest ,who will use nodejs do a heavy job? and,if we talk VC, the biggest problem is bandwidth.before we exceed mediasoup performance limit, the bandwidth or udp package crash the server 99999999 times. personally, i want mediasoup has more function, can export "stream/rtp" object /data ... etc.but,it is your project, you are the god. |
There is an open issue for this: versatica/mediasoup-client#6 However, we have not received any contribution. Building a native SDK requires time and expertise. We, the mediasoup developers, can rely on I wrote libsdptransform, which is a C++ port of the |
This si completely false. We've NEVER recommended using ANY signaling protocol. The documentation clearly says that mediasoup does not mandate any sig protocol, and there is even a Examples section with mediasoup examples using different signaling solutions: https://mediasoup.org/documentation/examples/
No idea. May be Netflix? PayPal? are those good examples of applications running on top of Node that also require a very good performance? BTW
Bandwidth is not a problem if you can run parallel servers. That's one of the features that mediasoup v2 will enable.
You are welcome to propose those features you wish. Have you requested them before? |
In this case, how the producer side handle PLI/FIR ? If we have several consumers has very poor network, there will be many PLI/FIR requests I think. |
|
hi, ibc , take easy . I read the whole document about mediasoup . I know mediasoup "NEVER recommended using ANY signaling protocol." . But ,in real life ,if somebody want his project online ASAP, it's better to using "Pxxx server". i forgot the name. So it 's why i said "force to using" ..maybe ,i should say " for lazy man". haha. I applaud mediasoup exposes its API in Node but it manages audio/video in C/C++. It's a good design. In fact, i have test using pure nodejs to proceed "media stream regarding" task. such as RTMP /FLV. You know the result. That's why i say "who will use nodejs do a heavy job". Have you requested them before? --- OK , i just told you yesterday ..hahah. from my post ,you know my english is poor. so , usually , i will not say much. Regarding the "bandwidth" , it's a huge topic. From my experience, in following scenario. Just my opinion , V2 and mediasoup-client is enough for application development. It's time to make it stable , provide more function/api . for example : BTW: i really really want to say "you do a good job". sincerely . |
Currently the Said that, there is a simple rule here: just the encoder can process a PLI/FIR and generate the corresponding keyframe. So, when in a SFU scenario, that is the remote client. |
The API documentation does not even mention protoo, so what you mean is the
What we (also) want to promote in mediasoup v3 is the ability to create server-side media producers that inject streams into a
The fact that 20 participants are sending a video stream to mediasoup does not mean that mediasoup is forwarding 20-1 streams to each participant. It's up to the application to decide which streams are sent and which one are not in each instant. For example, if your app UI just renders 10 remote videos, your JS may pause all the others and resume them when they become "visible".
Performance is not the only reason for v3. If you read the 4 bullets above, just one of them talks about "performance".
We want to keep mediasoup as simple and small as possible. We don't want to create a full monster that does everything. One of our goals in v3 is allow developers to easily plug their own stuff into mediasoup.
OpenCV does not fit well within a SFU. A SFU receives, selects and forwards streams, "just that". If you want to do computer vision or AI or whatever, you need a "media server". What we want to allow in mediasoup v3 is making it easy for developers to plug those "media servers" into mediasoup (in the same or in different servers).
Thanks a lot. |
That's ok . Do you mean anyone using mediasoup can take streams out , do some job ,and then put it back?? or , insert/replace streams into "server side peer" ? Anyway ,these ideas is great. So, the filter/mixer/CV/AI can be true. up to application developer. |
It's not mediasoup the entity who will do such as "job". The picture would be as follows:
|
|
Just piping in my 2 cents here, but I would like to add some thoughts to the V3 horizon. I started using Mediasoup because I wanted to broadcast from a browser. I had 3 choices for architecture: mesh, peer-to-peer downstream distribution, and server based. Because I also wanted to record all streams on the server, Mediasoup was the best choice. I really liked it's simple and universal API on the client, and ability to run without installing a separate server daemon. Furthermore, the node wrapper around a C-based core installs seamlessly. If I had any other use-case, I would not have gone with Mediasoup. I am happy to see that the 'broadcast' use case is high on the priorities! What I do not understand is the discussion around the limiting factor when broadcasting. It seems to me that the focus should be on running multiple instances of MediaSoup on multiple networks to scale horizontally. No matter how much we refactor and/or streamline the API, the biggest issue when it comes to broadcasting will always be performance on production systems. So I would like to see a system to easily create, configure, and monitor a network Mediasoup servers that provides load-balancing and fault tolerance. |
Please re-read bullet 4 in the issue description. mediasoup v2 uses N C++ subprocesses, but each room just runs in a single subprocess, so before "scaling horizontally" it would be simpler to just scale within the same machine by allowing a room to use N C++ subprocesses. mediasoup v3 will allow that and also horizontal scaling via separate servers.
The API refactor has a different motivation. The 4 bullets above do not have direct relationship between them, they are orthogonal features/changes. |
|
Hello @ibc. Thank you for responding. I did read bullet #4. My issue is that my bandwidth, not my CPU, is the limiting factor. Once my application has a certain number of (publishers x subscribers), I will need to run on another physical network. On my LAN, I have 8 servers, each with 12 CPUs and 100 GB RAM. No matter how much you optimize your code, my WAN network is the limiting factor.
That's my concern. I was hoping we could consider all the options before you start your journey.
I don't see this 'horizontal scaling' in the original post or anywhere else for that matter.. If that's the case, great! |
|
We cannot do magic to increase WAN network bandwidth. However streaming servers do exist. I assume they solve the network issue somehow. |
|
I hope this will help people here understand the motivation for the new broadcasting design in v3: https://github.com/versatica/mediasoup/blob/master/doc/Charts.md |
|
@notedit, please take a look to the issue description, I've updated it with info about "PLIs every two seconds". |
|
For all those interested in how the v3 API will look like, here a test unit that makes use of all the API in mediasoup-client v3: https://github.com/versatica/mediasoup-client/blob/master/test/test.js?ts=3 |
|
please don't make a protoo/reactjs gas factory for demo. |
|
There are already N mediasoup example apps running vanilla JS. Please don't reopen such a request here again. |
|
I am not asking for example apps. I am happy some users added finally some samples. |
This is the API in
You can use them if you are fine with the API they expose. Also, those apps you mention provide a specific signaling protocol, |
In mediasoup-client v3: // Send a track.
const audioProducer = await sendTransport.send({ track });
// Receive a track.
const audioConsumer = await recvTransport.receive({ producerId });
// Play the received track into an HTML <audio> element.
audioElem.srcObject = new MediaStream([ audioConsumer.track ]);And, of course, communicate those actions (when appropriate) to the server to call |
|
that's brilliant. If could could embed that into you demo, that would be perfect. |
That will be documented once mediasoup (server) v3 is done :) |
|
Hey, @ibc , it would be nice to have a demo part in vanilla js. In the whole the demo is great. Nice work. Sorry. |
Oh sure! |
|
:) But would be also great if we can have smaller snippets to start with |
|
Guys, still working in v3 which, indeed, will include much better doc and code snippets. It's just that we have not finished it yet. |
|
The demo is more like a showcase: this is what you could achieve ... if you have good doc :) |
|
Hi, do you already now, when could the v3 be released? At least approximately, because we decided to use mediasoup for our project, which is gonna be more complex, so if we could start with the v3, that would be definitely great, as there are really good improvements and simplifications. |
|
v3 will be ready (if everything goes well) in some weeks. You can start using it already since the API is not gonna change much (in both mediasoup and mediasoup-client). However the API documentation is not yet written. You can use the mediasoup-demo project (branch "v3") as reference for now. |
|
@ibc thanks for your quick response, I'm glad to heat, that it's gonna be released in a few weeks |
|
Hi, firstly thanks a lot for this great project ! I really look forward for the completion of the v3 :) Maybe you could help me on this one : I wish to use mediasoup together with an extra machine learning layer to analyse the video streams in real time, which would make use of the currently experimental workers feature of node.js (thus node's single thread base architecture shouldn't be a performance issue). For this I would need to extract each frame of the streams (to pass them to the video analysis workers). Is there a way to do so with mediasoup v3 (as it seemed to be possible in the v1, referring to your answer here if I understood correctly) and if so, any hint on how I could do it ? Thank you ! |
|
With mediasoup v3 is terribly easy to forward a video stream to whichever endpoint (using plain RTP over UDP) and the other way around: injecting plain RTP from any endpoint. Getting RTP packets via JavaScript events was dropped in v2. It's much more efficient to route them to any destination via UDP. Probably, by using GStreamer or similar you can consume those RTP packets and use them as your wish. |
|
Okay, that's what I understood from exploring the v2 but I wasn't sure about the reasons why it was dropped. I'll try working with gstreamer then ! Thanks for answering so quickly ! |
|
We could add it back. However it's really CPU expensive since all RTP packets must be serialized from C++ to the mediasoup Node process via IPC (connected Unix Sockets). |
Is this feature ready? It is possible to use plain RTP to achieve the goal. |
|
Yes, it's ready, but not yet documented. Working on the v3 API documentation these days. We'll announce it in the mediasoup mailing list when ready. |
Our test result show CPU almost exhausted when participants reaches 400+. We already fixed a performance issue on std::list::size() caused by old gcc. gprof result shows most consuming api is sha related. Is it reasonable? I can only assume the test report your present is conducted without dtls ? |
|
@antidigital how much CPUs on board? |
|
Current status of mediasoup v3Almost done. We are just adding sender-side bandwidth estimation (for now with REMB, Transport-CC will be added next) for automatically upgrade/downgrade simulcast spatial and temporal layers based on the endpoint's available bandwidth. v3 documentation is work in progress but some sections are already written. Those are:
|
|
A new section has been added explaining some important differences https://mediasoup.org/documentation/v3/differences-between-v2-and-v3/ |
|
Hi!
Maybe what I'll say is unfounded, but the other day I read about Node C++ addons which seem to offer a way to share memory between C++ programs and Node. Have you already heard of or considered this? I'm asking because I was thinking this could be a way to share packets directly with Node. But as I'm far from being comfortable with all the concepts involved with RTC, I can just ask you about this. I hope it makes some sense. Cheers |
|
That's true, but honestly I do not want to depend on Node addons internals for media processing. As the FAQ says:
Making the mediasoup worker a Node addon would also mean that both, the signaling plane and the media process, run in the same process. In mediasoup we can launch N workers (subprocesses). Said that, in v3 is easy to forward a RTP stream to a UDP destination (can be localhost). And such a UDP destination can even be a Node.js UDP socket that reads the RTP packets being received on it. |
BTW you can already try it. Just the sender-side bandwidth estimation is missing (working on it). The API is finished and documented. The mediasoup-client v3 API is not yet documented, but you can figure it out by checking the mediasoup-demo app. |
|
Okay, I understand the reasons for not using node addons, I wasn't sure whether you could create subprocesses with it. Anyway the RTP stream to UDP destinations seems to be a more than perfect solution for what I plan to do.
True, that's just because I don't have the time yet to experiment with it :) |
Just in case, I've found a RTP parser for Node (I've not tested it): |
|
How about npm packet to test mediasoup v3? Or is better to clone it from this github repo? |
NPM packets will be published when v3 is released, not before. You can clone from GitHub for testing purposes. |
|
mediasoup v3 is almost ready. Sender side bandwidth estimation has been implemented, meaning that mediasoup now properly adapts the sending spatial and temporal layers based on the network capability of each remote consumer, which is a huge improvement. As a side note, the API documentation of mediasoup v3 is already done: v3 documentation is work in progress but some sections are already written. Those are:
Working on the remaining documentation before releasing v3 packages. Let's close this issue since everything. |
After internal discussions we have decided to start the design of the future
mediasoupv3. The v2 version has been a great advance and had made possible many scenarios that were impossible in v1 (it introducedmediasoup-clientwhich provides a proper and unified API for all the browsers). However we are aware of a big room for improvement in the current v2 design.Changes for v3 and their rationale
mediasoupv2 is based on peers. A peer is just a wrapper that holds transports, producers and consumers. However those peers just exist at Node.js level, they do NOT exist at C++ level. Those peers also impose some unnecessary internal design constraints that provide no real benefit. In v3 we'll get rid of "peers". It will be up to the application to manage the concept of peers (if required by the app) or not.mediasoupv2 depends too much on opaque messages (the MEDIASOUP_PROTOCOL) generated by both,mediasoup-clientandmediasoup(server), that must be blindly exchanged between both without the application being aware of what those opaque messages exactly mean. In v3, we'll get rid of those opaque messages and, instead, will provide real JS API calls for bothmediasoup-clientandmediasoupto create and manage transports, producers and consumers.Make it possible to easily inject external audio/video in
mediasoupfrom sources such asffmpeg,gstreamer, etc. without requiringmediasoup-client. This is easily doable in v3. An example using example usingffmpeg: https://github.com/versatica/mediasoup/blob/master/doc/ffmpeg.mdmediasoupv2 can just listen for RTP in a single IPv4 and/or IPv6, making it not possible to run different transports in different IPs (needed in scenarios in which there are external clients but also clients running within backend hosts that may accessmediasoupusing a different IP). In v3, the Node.js app will be responsible of indicating the exact IP or IPs each transport must use.mediasoupv2 is multi-process. It runs into a single Node.js process and launches N (typically same number as CPU cores) C++ subprocesses:mediasoupworkers that handle the media layer. Those media workers are single thread (based onlibuvIO library). However, each room in mediasoup just uses a single worker, which means a single CPU with a single thread. This works very well for some hundreds of participants, but it does not scale well for broadcasting scenarios (a few media producers and many many many consumers).mediasoupv3 will make broadcasting scenarios possible by allowing a room to use N media worker subprocesses. It will also make it possible to extend a room into N separate hosts running different instances ofmediasoup.The text was updated successfully, but these errors were encountered: