Allow reading Multichannel Opus streams with WebRTC

[aler9]

Describe the feature

Follows #3355

[ditogam]

Hi @aler9

Thank you so much for your amazing work and findings.

I'm testing new behave

My origin file is from Surround sound test files in (almost) every format
, Google Drive folder

I run this command tu publish srt (setting b-frames to 0 and transcode to opus)
ffmpeg -re -stream_loop 100 -i "/home/dito/Downloads/AAC 5.1.mp4" -c:a libopus -c:v libx264 -bf 0 -f mpegts "srt://localhost:8890?streamid=publish:mystream"

   Stream #0:0(und): Video: h264, yuv420p(progressive), 800x600, q=2-31, 8 fps, 90k tbn (default)
    Metadata:
      creation_time   : 2012-05-17T00:54:33.000000Z
      handler_name    : GPAC ISO Video Handler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc60.31.102 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
  Stream #0:1(und): Audio: opus, 48000 Hz, 5.1, flt, 320 kb/s (default)
    Metadata:
      creation_time   : 2012-05-17T00:56:13.000000Z
      handler_name    : GPAC ISO Audio Handler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc60.31.102 libopus

Receiver
ffplay -rtsp_transport tcp "rtsp://localhost:8554/mystream"

 Stream #0:0: Video: h264 (High), yuv420p(progressive), 800x600, 8 fps, 16 tbr, 90k tbn
  Stream #0:1: Audio: none, 48000 Hz, 5.1

MtxServer logs

2024/05/20 18:03:24 INF [SRT] [conn 127.0.0.1:49250] opened
2024/05/20 18:03:24 INF [SRT] [conn 127.0.0.1:49250] is reading from path 'mystream', 2 tracks (H264, Opus)

No audio

ffplay "srt://localhost:8890?streamid=read:mystream"

Stream #0:0[0x100]: Video: h264 ([27][0][0][0] / 0x001B), none, 90k tbr, 90k tbn
Stream #0:1[0x101]: Audio: opus (Opus / 0x7375704F), 48000 Hz, 5.1, fltp

Audio works with srt recewiver

MediaMtx
2024/05/20 18:03:49 WAR [SRT] [conn 127.0.0.1:41095] frame is too big

WebRtc
http://localhost:8889/mystream/

Audio works, but sound is demaged (even thought you can hear unclean some words and signals)

2024/05/20 18:06:16 INF [SRT] [conn 127.0.0.1:54250] is publishing to path 'mystream', 2 tracks (H264, Opus)
2024/05/20 18:06:33 INF [WebRTC] [session 0de41c5f] created by [::1]:43468
2024/05/20 18:06:33 INF [WebRTC] [session 0de41c5f] peer connection established, local candidate: host/udp/172.19.0.1/8189, remote candidate: prflx/udp/192.168.2.9/39560
2024/05/20 18:06:33 INF [WebRTC] [session 0de41c5f] is reading from path 'mystream', 2 tracks (H264, Opus)
2024/05/20 18:06:36 WAR [SRT] [conn 127.0.0.1:54250] frame is too big
2024/05/20 18:06:40 WAR [SRT] [conn 127.0.0.1:54250] frame is too big
2024/05/20 18:06:42 WAR [SRT] [conn 127.0.0.1:54250] frame is too big

[aler9]

Hello @ditogam, for your tests did you use the latest release (v1.8.2) or a nightly binary that includes #3375?

Anyway, streaming multichannel Opus with WebRTC is still a work-in-progress and #3375 is yet to be fully tested and merged, in particular, the error you reported

2024/05/20 18:03:49 WAR [SRT] [conn 127.0.0.1:41095] frame is too big

is not covered by the patch yet and is due to an excessive size of the audio packets. I'll see what i can do about it before merging the patch.

[ditogam]

Hi @aler9 .
I tested against PR 3375, feature/webrtc-multiopus branch cheechout and run from GoLand

about frame is too big I just included in logs

[aler9]

hello @ditogam, i found a bug in parsing timestamps of Opus packets coming from SRT and i fixed it (#3388). This should fix the intermittent audio issue on Chrome.

Regarding the "frame is too big issue", from my tests it happens when bitrate is very high. High bitrate Opus tracks can be streamed with SRT indeed, but they cannot be streamed with WebRTC or RTSP since the specification tells that "An RTP payload MUST contain exactly one Opus packet as defined by that document.", therefore the maximum Opus packet size must be lower than the maximum RTP packet size (which is the UDP MTU). Fortunately the issue only happens with unreasonably high bitrates (higher than 512k), therefore it should be enough to set an explicit and reasonable bitrate when performing the encoding:

ffmpeg -stream_loop -1 -re -i 'AAC 5.1.mp4' -map 0:a:0 -c:a libopus -b:a 256k -f mpegts "srt://localhost:8890?streamid=publish:mystream"

Please test the latest feature/webrtc-multiopus and let me know whether the streaming to the browser is successful. Thanks.

[ditogam]

This is fantastic news, thank you so much. I tested, everything works perfectly. I run server from this branch and I can hear all the channels

[ditogam]

Even 7.1 works perfectly

[ditogam]

I tested locally and it worked perfectly on 5.1 and 7.1 (Downloaded from same source AAC 7.1.mp4 Google Drive folder)
Then I deployed it on external server with public IP.
5.1 worked as expected, but on 7.1 with WebRTC got deadline exceeded while waiting connection, even thought I put webrtcAdditionalHosts.
here is debug log (Masked IPs)
debug.txt

Publish command
ffmpeg -re -stream_loop 100 -i "/home/dito/Downloads/AAC 7.1.mp4" -c:v libx264 -bf 0 -c:a libopus -b:a 256k -f mpegts "srt://MEDIAMTX_PUBLIC_IP:8890?streamid=publish:mystream"

output

 Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp42isom
    encoder         : Lavf60.16.100
  Stream #0:0(und): Audio: opus, 48000 Hz, 7.1, flt, 256 kb/s (default)
    Metadata:
      creation_time   : 2013-05-09T11:31:26.000000Z
      handler_name    : soun
      vendor_id       : [0][0][0][0]
      encoder         : Lavc60.31.102 libopus

srt reciever

ffplay "srt://MEDIAMTX_PUBLIC_IP:8890?streamid=read:mystream"

output

Input #0, mpegts, from 'srt://MEDIAMTX_PUBLIC_IP:8890?streamid=read:mystream':
  Duration: N/A, start: 58.325322, bitrate: N/A
  Program 1 
  Stream #0:0[0x100]: Audio: opus (Opus / 0x7375704F), 48000 Hz, 7.1, fltp

works.

Then I downloaded dolby-demo-trailers-hd
And it worked with WebRTC as well
ffmpeg -re -stream_loop 100 -i "/home/dito/Downloads/7.1.mkv" -c:v libx264 -bf 0 -c:a libopus -b:a 256k -f mpegts "srt://MEDIAMTX_PUBLIC_IP:8890?streamid=publish:mystream"

output

Stream #0:0(eng): Video: h264, yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 29.97 fps, 90k tbn (default)
   Metadata:
     encoder         : Lavc60.31.102 libx264
   Side data:
     cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
 Stream #0:1(eng): Audio: opus, 48000 Hz, 7.1, flt, 256 kb/s (default)
   Metadata:
     encoder         : Lavc60.31.102 libopus

So it seams working
Thank you very much

[github-actions]

This issue is mentioned in release v1.8.3 🚀
Check out the entire changelog by clicking here