Audio/Video demo is broken

[SakulFlee]

WebRTC is intended to be a real-time P2P communication protocol.
It's mainly used for VoIP (Voice-over-IP, i.e. audio/video calling).
Having support for proper Audio/Video communication via the plugin is a wanted goal.

However, while trying to implement this now for the third time, I've encountered some issues with different parts.
Below, I will describe what exactly the approach and consequent issues were.
Once any of the issues get resolved, we can properly add the Video/Audio demo.
Until then, the Demo will be scrapped.

SIPSorcery's Media libraries

SIPSorcery (maker of the WebRTC library used) actually provide a separate library for Media related things.
Mainly, audio and video streaming & communication.

The issue with their library is that it still is a pre-release and has been for years at this point.
When it works, it works beautifully and is relatively simple to use, but in most cases it doesn't work out of the box.
Mostly, the issue can be traced back to some broken library file, which may or may not be included for your target platform.
Building this library yourself does fix the issue, but you are required to build it for every platform x architecture you are targeting. This is simply out of scope for this project.

Once SIPSorcery properly releases their libraries and fixes issues with the media encoding/decoding, this may be the most viable solution.

Using Godot's VideoStream nodes/classes

Godot itself comes with video playing support via VideoStreamPlayer.
You can read more about it here: Godot: Playing videos.

In theory, this sounds ideal!
A Godot-"native" video support would mean we just have to stream (or transfer) the target video and display it.

However, many issues arose:

1. Godot only supports the Ogg Theora format and nothing else.
2. Ogg Theora isn't properly streamable -> you have to basically transfer the whole file first.
3. Godot refused to properly load and playback and Ogg Theora converted file for me. Even after following their guide about using FFMPEG to convert videos.

The transfer itself (2.) isn't a big issue. It just means that we have to wait first for the file transfer to be finished, which, depending on the size of the video file, can take a bit.
But Godot basically refusing to open the video file at all is a big problem.

You are also forced to use the .ogv (Ogg Theora) format.

Using 3rd party libraries to extract frames

Lastly, I had the idea to split the video into frames and an audio stream (audio actually never was an issue!).
This approach would dissect the video into frames (e.g. 30FPS == 30 frames per second) that would get send to the peer, then displayed as a texture on a rectangle or something like it.
This sounds ideal, since we aren't required to use the VideoStream nodes from the above approach and are "just" displaying images and playing back some audio.

However, the issue is actually similar to the first approach: Any library I could find that enables us to dissect a video into frames is based on something like FFMPEG, GStreamer, VLC, etc. and none of those come with it bundled.
Meaning, you have to have the said tool installed locally on your machine, thus on every device running the app/game.
Furthermore, it would probably be impossible to have this working on Android or iOS, given that most of those based upon libraries don't build for those platforms.

This approach is working, but you will have to do some encoding manually first:

1. Extract any audio you need from the video
2. Dissect the video into individual frames (images)
3. Load this all into Godot and somehow keep it organized and efficient
4. Transmit the audio stream and the images
5. Upon receiving a transmission, start playback of the audio and display the images.

Steps 4-5 are easy to do and do work.
Step 1-3 are the essential problem.

---

Like stated above, there currently is no good and proper way to get this working with the current version of Godot (4.2) and state of libraries for C#, given dependencies.
I will use this issue branch to revert some changes being done for each approach (stripping out unnecessary things) and will close this issue for now.
If in the future there is demand for this AND a proper solution can be found (ideally with Godot tools ...), please reopen this issue and let us discuss.

As a side note:
While toying around, I couldn't really find a proper use-case for this except for literal calling.
Any asset you may wanna stream to a peer should already be a game asset, thus no streaming required as it already exists locally per peer.
Video-Calling remains the only use-case I could identify for this, however:
Godot, as of writing this, also doesn't provide proper Webcam support.
There isn't an easy way to grab a video feed from your webcam or smartphone camera and then stream that to other peers.

---

Update:

A few things have changed, this issue is being developed on again.
The tasks are:

• One-Side video streaming
• Bi-Directional video streaming
• Audio streaming
• Documentation

Note

Video streaming quality is relataively low due to a packet size limitation with WebRTC.
Potential solutions for this are out-of-scope, but are discussed in the documentation.

[GeorgeS2019]

is this related to this sipsorcery-org/sipsorcery#698

[SakulFlee]

is this related to this sipsorcery-org/sipsorcery#698

Yes, indeed!

However, neither version of the included library would work for me. Neither on Windows, nor Linux.
It also fails to load the library entirely, not just some part of it.

Building the library from source works if you then package it correctly when exporting the app.
Although, I couldn't find a way to automatically include the build library files as it always replaces it with the providen, broken, library from the NuGet package.

The whole package is still in pre-release.
I assume with some time and an eventual actual release this will be fixed.

In the mean time here is what you can do:
A audio/video track is just another data channel under the hood with some added events for format negotiation.
If you know the source video/audio file, you can skip negotiation. If it's some common format that Godot can display, you can even skip transcoding (alternatively: find some video/audio transcoding library. FFMPEG is probably a good point to start).
Then simply either transfer or, better, stream the file over a data channel.

Compared to using the "official" way there is barely any difference. The only thing you are loosing is automatic event handling and transcoding, which currently is also broken for most given the library won't work out of the box.
I was thinking about changing the audio/video to do exactly that until that part of the library is stable/released/working.

On a side node, the main issue with that library seems to be VP8 transcoding which is also still fairly new. Using some proofen transcoder like x256 might be a better choice for now.

[GeorgeS2019]

A audio/video track is just another data channel under the hood with some added events for format negotiation.
If you know the source video/audio file, you can skip negotiation. If it's some common format that Godot can display, you can even skip transcoding (alternatively: find some video/audio transcoding library. FFMPEG is probably a good point to start).
Then simply either transfer or, better, stream the file over a data channel.

SIPSorcery provides unit tests and examples, but I still fail to see why the provided e.g. video streaming examples by SIPSorcery are allowed to fail without a fix or PR from users (I have not check that yet)

[SakulFlee]

SIPSorcery provides unit tests and examples, but I still fail to see why the provided e.g. video streaming examples by SIPSorcery are allowed to fail without a fix or PR from users (I have not check that yet)

They do! But their current version unfortunately still is a pre-release and the issue seems to be documented.
Honestly, giving the complexity of having to rebuild the library for each target platform + architecture, it may be worth using something else and streaming video/audio data that way.

I've also read that it may be a packaging issue. Simply by manually downloading Rh NuGet package and swapping library files around also seems to fix it.
However, only for windows and only for x86 or something.
Other platforms aren't included at all from what I can see.

Using SIPSorcery's package seemed to be a logical choice, but in hindsight not all the best.

Is there high demand to get the Audio/Video Demo fully working?

[GeorgeS2019]

@SakulFlee
i see no more audio/video demo

[SakulFlee]

@SakulFlee i see no more audio/video demo

Yes, I've updated the issue description, it describes the issue in more details, but here is a shorter form:

• The Video/Audio demo wasn't properly working, and a third attempt at getting it to work also failed.
• Godot only support OGV files, which aren't properly streamable
  • Meaning: The video/audio file has to be fully transferred first before playback. Limitation, not an issue.
• Godot fails to read OGV's most of the time for an unknown reason I couldn't figure out
  • Big issue, other formats are unsupported as of 4.0
• There isn't a single library for C#/.NET that doesn't require some external application to be installed
  • All libraries I could find rely on something like FFMPEG, MPV, VLC, GStreamer, etc. to be installed on the target system
  • We can't really ship them with the demo/project due to license issues, nor do I want to tell people to "go to this other repository, build it from source, copy the files over, etc. ..."
  • Furthermore, most of those required applications don't export for platforms like Android or iOS, breaking compatibility with those systems
• And lastly; I couldn't find a proper use-case for this unfortunately, except for actual calling (with or without video) which would require webcam or phone camera access which:
  • Godot also doesn't give you a proper way of accessing at this moment.

Once this is achievable, I would love to have this demo re-added (after all, WebRTC is a communication technology).
However, as of now, there isn't a good way I could come up with or find that will work:

• Regardless of the target platform (i.e. cross-platform, including mobile devices)
• Doesn't require extra steps like compiling your own library files
• Matches our license
• Matches our quality standards

If there is high demand, I can only encourage you to play around and try some things out.
Maybe there is a solution I am missing that actually could work.
If so: Keep me updated! I am happy to assist/help :)
As for now I will focus on getting the two other big tasks (support for more than two peers & integration with high-level Godot nodes).

[GeorgeS2019]

Thx for clarifying. I will try asking around for solution

[GeorgeS2019]

https://github.com/search?q=Godot%20mp3&type=repositories

[GeorgeS2019]

@fire

This is the closest demo on WebRTC for audio and video.

Would like your view on the challenges discussed here

[GeorgeS2019]

https://github.com/Pieeer1/GodotVoipNet

[GeorgeS2019]

https://docs.godotengine.org/en/stable/tutorials/animation/playing_videos.html

HandBrake (GUI) and FFmpeg (CLI) are popular open source tools for this purpose. FFmpeg has a steeper learning curve, but it's more powerful.

Here are example FFmpeg commands to convert a MP4 video to Ogg Theora. Since FFmpeg supports a lot of input formats, you should be able to use the commands below with almost any input video format (AVI, MOV, WebM, …).

https://github.com/ReferenceType/P2PVideocall

[SakulFlee]

https://github.com/Pieeer1/GodotVoipNet

That's a voice/audio only solution.
Audio never has been an issue.

[GeorgeS2019]

I need more time to have the overview. Thx for any feedback U can provide

[SakulFlee]

https://docs.godotengine.org/en/stable/tutorials/animation/playing_videos.html

HandBrake (GUI) and FFmpeg (CLI) are popular open source tools for this purpose. FFmpeg has a steeper learning curve, but it's more powerful.

Here are example FFmpeg commands to convert a MP4 video to Ogg Theora. Since FFmpeg supports a lot of input formats, you should be able to use the commands below with almost any input video format (AVI, MOV, WebM, …).

https://github.com/ReferenceType/P2PVideocall

That's exactly what I was saying.
You are required to use and distribute FFMPEG (or similar solutions with GStreamer, VLC, MPV, etc.) with the application or require users to have it installed locally and in path.

From what I can tell, this isn't really possible on all platforms and would require other people either compiling a lot (Cross-Platform ...) Libraries before even opening the project, or, we would need to bundle a precompiled binary.

What we could do is make a demo out of this, seperate to the main project, and have it with a lot of requirements/limitations.
Not really the standard I am going for, but it seems like the only solution as of now?

[GeorgeS2019]

For Godot, it is important to demo this is possible in Godot. This helps getting more people excited and contribute.

WebRTC video streaming has been high on request. See Godot demo issues.

Although it is imperfect, not possible for all platforms, it is still important to show in what way it works in Godot.

In time, some one would implement the missing cross platform in Gdextension c++ or rust

I can't tell why there is no WebRTC video streaming yet by Godot team

It is still good idea, like u said to demo to inspire others.

[SakulFlee]

Although it is imperfect, not possible for all platforms, it is still important to show in what way it works in Godot.
In time, some one would implement the missing cross platform in Gdextension c++ or rust

Are you willing to spend time on this and porting something like FFMPEG to GDExt.?

---

I can't tell why there is no WebRTC video streaming yet by Godot team
It is still good idea, like u said to demo to inspire others.

Godot, being a game engine, doesn't have a high focus on VoIP/Calling/Video Calling.
Furthermore, in my opinion at least, the Godot networking stack is also very outdated, inefficient and badly designed. This further complicates such issues (and thus this project was born actually!).
I understand the problem though, and I do want a demo like that integrated as well.

However, there are multiple issues like I stated. Not just the library issue:

1. We need a way to properly decode/encode videos OR splice a video into frames + audio and transmit those.
   This requires some kind of library or Godot functionality.
   Unfortunately, Godot only supports OGV and their support seems to be broken when I try to extract individual frames.
   Additionally, any 3rd party solutions require FFMPEG or similar to be installed which again: Not possible on all platforms.

Alternatively: We need a "Video Player Node" for Godot that supports some other format like .m3u8 or .mp4.
Though, as stated by the Godot wiki, those formats are licensed and can't just be added like that.

2. We need a way to actually stream that data (video) stream.
   I assume we are actually talking about real-time communication (as in VoIP) and something like grabbing the feed of a Webcam + Microphone.
   This data needs to be in the correct format (OGV ...), streamable (which OGV isn't) or needs to be transcoded, which brings us back to issue 1.
   Secondly, Godot currently doesn't have any proper way of actually retrieving a Webcam stream, as far as I can tell.
   Another solution would be needed to grab this data, which most likely also won't be cross-platform.

3. We need a way to actually display this data inside Godot.
   Like stated in 1.: Godot only supports OGV, but OGV isn't streamable and thus not viable for VoIP.
   Thus, a solution would be needed to either display frames in quick succession to basically make our own "Video Player Node", or, a way to decode a for example, MP4 stream, into OGV segments which then will be queued inside Godot.
   Both ways require a library or support like stated in 1.

---

Transmitting data over WebRTC is possible and the main goal of this project.
This part works and exists already. A video can also be easily streamed or transferred over the connection being made (with some added protocol layer ...).

If we are just talking about transmitting a video+audio over WebRTC and then displaying it, that's easily done.
But if we are talking about real-time communication like in VoIP we have to deal with streamable formats, which OGV isn't, and find ways to properly en- and decode this data.
Apart from grabbing it from e.g. a webcam in the first place.

[SakulFlee]

Since Audio was never an issue, we could compromise at implementing VoIP (audio-only).
That should be fully supported by Godot, as far as I can tell.
Microphone access can also be easily achieved, it seems: Godot-Wiki: "Recording with microphone"

And once there is some development around, either:

• Godot's VideoPlayer Node supporting better formats
• FFMPEG (and similar) aren't required to be installed or easier to ship (incl. cross-platform!)
• Proper Webcam access inside Godot

We could add Video support.

[GeorgeS2019]

https://github.com/rosenbjerg/FFMpegCore

working-with-raw-video-frames

https://github.com/Ruslan-B/FFmpeg.AutoGen

[GeorgeS2019]

FFmpeg GDExtensiom
https://github.com/EIRTeam/EIRTeam.FFmpeg
godotengine/godot-proposals#3286 (comment)

@makemefeelgr8 <= FIY
@mattkanwisher <= FIY

godotengine/godot-proposals#813 (comment)

[SakulFlee]

https://github.com/rosenbjerg/FFMpegCore

working-with-raw-video-frames

https://github.com/Ruslan-B/FFmpeg.AutoGen

Both require bundling of FFMPEG or libraries related to it which I want to avoid it possible.

[SakulFlee]

@GeorgeS2019 I might have found a solution for the FFMPEG issue, but it had to wait until 2.0.0 is done.

Still no solution for webcam though.
Also won't be Cross-Platform.

[GeorgeS2019]

webcam => godot GDMP

[makemefeelgr8]

Have you considered embedding some kind of a webview? This would resolve 99% of "feature X is not supported" issues on all platforms, including video streaming and playback, webrtc, GUI scaling, and so on. One could add a node to render a browser surface, sprinkle it with some JS interop, and have a reasonably stable workaround in place.

[SakulFlee]

Have you considered embedding some kind of a webview? This would resolve 99% of "feature X is not supported" issues on all platforms, including video streaming and playback, webrtc, GUI scaling, and so on. One could add a node to render a browser surface, sprinkle it with some JS interop, and have a reasonably stable workaround in place.

I like the idea in concept, however there currently doesn't seem to be any webview implementation for Godot.
I found GDCEF (and various forks of it ...), but none of them have fully updated to Godot 4.x yet, and it doesn't seem like they will soon.

Porting CEF (Chromium Embedded Framework) to Godot certainly seems possible, but would be a massive overhead just to get this demo working.

sharpWebview & webview might be worth a try, though from a quick look it doesn't seem like there is an easy way to make this work with Godot. Especially, [STAThread] seem problematic, and I don't see a way of easily letting it render to a texture (or something like it) inside Godot, instead of being its own window.

---

Live de- & encoding using something like FFMPEG might be the only way (for now at least).
I might have found a solution in that regard, at least for Desktop-platforms.
Further testing will be required for mobile platforms and may simply not work there.
MediaFileProcessor seems to support both: Having FFMPEG installed & in PATH, and downloading it.
This would remove the need to bundle FFMPEG, but may not work on every platform. Further testing is required.

---

webcam => godot GDMP

I don't know anything about GDMP and MediaPipe, but it looks more like a Machine-Learning solution than a pure webcam access.
However, if we are going with FFMPEG anyways:
FFMPEG supports capturing webcams natively!
At least from the command line:

$~> ffmpeg -list_devices true -f dshow -i dummy
(...)
[dshow @ 000001f76251a300] "Integrated Camera" (video)
[dshow @ 000001f76251a300]   Alternative name "@device_pnp_\\?\usb#vid_5986&pid_212b&mi_00#7&155234e1&0&0000#{65e8773d-8f56-11d0-a3b9-00a0c9223196}\global"
[dshow @ 000001f76251a300] "OBS Virtual Camera" (none)
[dshow @ 000001f76251a300]   Alternative name "@device_sw_{860BB310-5D01-11D0-BD3B-00A0C911CE86}\{A3FCE0F5-3493-419F-958A-ABA1250EC20B}"
[dshow @ 000001f76251a300] "Microphone (Realtek(R) Audio)" (audio)
[dshow @ 000001f76251a300]   Alternative name "@device_cm_{33D9A762-90C8-11D0-BD43-00A0C911CE86}\wave_{46D75241-389F-4203-A402-106E94028BDC}"
[dshow @ 000001f76251a300] "Microphone Array (Realtek(R) Audio)" (audio)
[dshow @ 000001f76251a300]   Alternative name "@device_cm_{33D9A762-90C8-11D0-BD43-00A0C911CE86}\wave_{1E21DAC9-A62F-482A-AFB8-4D041D03BEA7}"
[dshow @ 000001f76251a300] "Microphone (Steam Streaming Microphone)" (audio)
[dshow @ 000001f76251a300]   Alternative name "@device_cm_{33D9A762-90C8-11D0-BD43-00A0C911CE86}\wave_{3F4AC327-D414-4856-AB0C-D9BEA83F1B7D}"

more here

I can't give an ETA yet, but I will start working on this after v2.0.0 which should be around the corner! :)
Thanks for the inputs and ideas. If anyone wants to port CEF to Godot please do.