Pretty bad H264 performance after upgrading to iOS 15.4

[derMani]

Description

Hi everyone.

after upgrading our phones to iOS 15.4, we are experiencing a pretty bad H264 performance.
Versions before iOS 15.4 with the same source code are not affected by this provlem

I can provide a simple peer 2 peer example if needed

Console output looks normal.

Versions affected

• Cordova version 11
• Cordova iOS version: cordova-ios 5 and cordova-ios 6
• Plugin version 6 and 8.01
• iOS version 15.4
• Xcode version 13.3.1 (13E500a)

Steps to reproduce

Establish an iPhone to Chrome video connection via H264 and compare it with VP8.
A lot of jitter and bad FPS situations will occur, if you observe chrome://webrtc-internals

Both H264 profiles in the SDP with cordova iosrtc are affected

Expected results

Fluid video image, like in previous versions.
Maybe an upgrade of the webrtc lib could help?

Actual results

Stuttering video quality with H.264

[hthetiot]

Versions before iOS 15.4 with the same source code are not affected by this provlem

I don't think compilation of WebRTCLib will help here, and BTW, you don't need that plugin to use WebRTC using iOS 15.4.

Try without the plugin (comment register global) to see if Native iOS WebRTC also have the same performance issue.

[derMani]

Update: Without using the globals of cordova-iosrtc: H264 works just fine with "native" WebRTC of Webkit.

Pros:

• Good performance
• The Video is an actual html element and the streams are actual javascript objects
• One plugin less in your project

Cons:

• You get the camera prompt in every session. You can work around this behaviour in WKWebView with "requestMediaCapturePermissionForOrigin"
• Works only with iOS 15+
• You will run into some weird WKWebView WebRTC bugs

The plugin had a good run, but I think most users of this plugin can now move on to the "native" WebRTC of Webkit

Anyway: The libwebrtc bug remains for H264 performance. This also happens in non cordova based projects when using libwebrtc

[hthetiot]

Thank you @derMani

The plugin had a good run, but I think most users of this plugin can now move on to the "native" WebRTC of Webkit

I agree, since iOS 15+, you don't really need the plugin if you don't have to put speaker on or support old iOS versions. Still you will have to deal with know ios safari webrtc issues that the plugin handle to manage (auto play, speaker, background).

[hthetiot]

In any case I will leave this issue open to.see if updating to latest libwebrtc fix thr issues.

[hthetiot]

@derMani see webrtc-100 PR that update webrtc to v100 that pass the CI (include full build test https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/.github/workflows/main.yml#L59)

[shrhoads]

I think this project may still be useful if the H264 encoding issue can be fixed. It would be nice to be able to provide higher quality encoding than is available in the safari webrtc support.

[hthetiot]

@shrhoads

I think this project may still be useful if the H264 encoding issue can be fixed. It would be nice to be able to provide higher quality encoding than is available in the safari webrtc support.

Then, can you then try #748 to see if iosrtc rendering from h264 is better?

[hthetiot]

Ping @derMani @shrhoads

#744 (comment)

[dl4mmers]

@shrhoads

I think this project may still be useful if the H264 encoding issue can be fixed. It would be nice to be able to provide higher quality encoding than is available in the safari webrtc support.

Then, can you then try #748 to see if iosrtc rendering from h264 is better?

Hey @hthetiot,

I've tested PR #748 on iPhone 14 Pro (iOS 16.3.1) and it fixes the H264 performance and fps issues.

[derMani]

Hi together, sorry für the late reply. I can confirm, that this works now without stuttering (also on iOS16) 👍

[hthetiot]

Hi together, sorry für the late reply. I can confirm, that this works now without stuttering (also on iOS16) 👍

@derMani using #748 also ?

In any case I'm going to release it as soon as possible.

[derMani]

Yep, #748 works fine 👍

[hthetiot]

Yep, #748 works fine 👍

Thx this help a lot to validate build.

[dl4mmers]

Hey @hthetiot, I continued testing PR #748 with my application and I noticed that, while H264 works fine, some WebRTC API's do not work like with iosrtc@6.0.21. I then tested iosrtc@8.0.4 and it's the same case:

After initiating a call and sending audio + video from iPhone 14 Pro to chrome browser

• RTCPeerConnection.getSenders() 
• RTCPeerConnection.getReceivers() 
• RTCPeerConnection.getTransceivers()

are always returning empty arrays in PR #748 and iosrtc@8.0.4, while with version iosrtc@6.0.21 there are correct RTCRtpSender objects inside. In chrome there are also correct values. So, when upgrading to latest iosrtc, I would have to use deprecated functions RTCPeerConnection.getLocalStreams() and RTCPeerConnection.getRemoteStreams().

Is this a bug?

[hthetiot]

Thank you @dl4mmers that really helpful.
I'm a bit surprised of this issue.
Let me investigate.

@RSATom would you mind check the comment above ?

[RSATom]

@hthetiot we are using WebRTC library from Chrome v102 pretty long time without any significant issues. But I have to check if we are using getSenders() | getReceivers() | getTransceivers()... I'll try to find time for that soon.

[hthetiot]

@RSATom can you do a PR for WebRTC 102?
We have PR for 100 already here #748

[hthetiot]

@dl4mmers do you know if you use webrtc-adapter that may be the cause of the issue, I have seen thr shim F* iosrtc sham in the past ?

[RSATom]

@hthetiot I think I did PRs for all my changes to iosrtc (and all of them was accepted if I remember correctly). What about WebRTC lib - we are using patched one
https://github.com/WebRTSP/WebRTC/commits/m102-patched
https://github.com/WebRTSP/libWebRTC/releases
And I don't know if it's good idea to use that patched library version in iosrtc.

[hthetiot]

@RSATom OK I will build or may be move to the jisti one use by react-native-webrtc and made original maintainers of iosrtc.

[dl4mmers]

@hthetiot Yeah I'm using webrtc-adapter, but I've looked through the swift code of RTCPeerConnection (https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/src/PluginRTCPeerConnection.swift#L324) and saw the check IsUnifiedPlan() and then noticed that I'm still using plan-b on iOS. I set it to unified-plan. Now with sdpSemantics: 'unified-plan' I get correct objects via RTCPeerConnection.getSenders(), RTCPeerConnection.getReceivers() and RTCPeerConnection.getTransceivers().

[hthetiot]

I will set unified-plan by default in next release I think it is time. Will avoid wasting people time.

[dl4mmers]

In any case I'm going to release it as soon as possible.

Hey @hthetiot, any news on releasing 10.x with WebRTC M100 or M102?

[hthetiot]

@dl4mmers yes, just need testing on that PR #748 see testing instructions in description. I will rebase master this wk and re ping here. I don't want to release without community feedback to risky.