Add support for WebRTC on iOS (WKWebView)

[pichillilorenzo]

WKWebView doesn't support WebRTC natively:

• https://forums.developer.apple.com/thread/88052
• https://stackoverflow.com/questions/45055329/does-webkit-in-ios-11-beta-support-webrtc/49467964#49467964
• https://stackoverflow.com/questions/54319924/access-camera-or-webcam-without-webrtc
• https://stackoverflow.com/questions/46988450/wkwebview-or-uiwebview-in-ios-app-does-this-webview-support-webrtc

Useful links:

• https://stackoverflow.com/questions/39639181/add-webrtc-in-webview-in-ios
• https://webrtc.org/native-code/ios/
• https://webrtchacks.com/guide-to-safari-webrtc/
• https://developer.apple.com/documentation/avfoundation/cameras_and_media_capture/requesting_authorization_for_media_capture_on_ios

How to solve this problem?

• Taking swift code from here and make it work on this Flutter plugin: https://github.com/cordova-rtc/cordova-plugin-iosrtc
• https://stackoverflow.com/questions/43916436/enable-camera-and-mic-access-in-wkwebview
• https://github.com/react-native-webrtc/react-native-webrtc

This is the unique solution I have found (this requires a lot of work).

[marinat]

Is this task added to plans?

[pichillilorenzo]

@marinat at the moment there aren't any plans about it. Also, another problem is that I don't have any real iPhone/iPad on where I can test (on iOS Simulator it won't work!).

[preetham-kena]

When will the WebRTC support come to WkWbeView Folks? This is hurting a lot in our inability to record music inside WkWebView.

[noahmehl]

Hey everyone, we wrote an open source swift shim for the missing WebRTC functionality for WKWebView:

https://github.com/OpenTelecom/WKWebViewRTC

Big thanks to the Cordova project!

[pichillilorenzo]

@noahmehl Thanks! Is there a way to import it as a dependency for iOS or should I clone/fork it?

[noahmehl]

@pichillilorenzo were hoping to make it into a cocoa pod, but for the moment, you would just have to clone it.

[marinat]

@noahmehl does camera work in your project? Samples from here didn't work at all

[pichillilorenzo]

@marinat did you give permission to the camera and microphone? I didn't test it, also because I don't have any real Apple smartphone/tablet.

[marinat]

@pichillilorenzo I gave permission to the microphone, but for camera permission unfortunately I got error dialog with empty description

[noahmehl]

@marinat we did not test video or the camera. I’m sure it could be done, but would need sponsorship.

[marinat]

@noahmehl should i create an issue in your project?

[noahmehl]

@marinat absolutely, feel free. I just want to be honest, that we currently have no immediate plans to update for that.

[noahmehl]

@marinat @pichillilorenzo the repo is now a cocoa pod. The original code is in a branch called prototype.

[pichillilorenzo]

@noahmehl So, if the camera is not working, I'm sorry but I won't add/use this dependency.
WebRTC is about both audio and video. If the camera is not working, it doesn't make sense to me to use it because this is a basic feature of WebRTC.
Unfortunately, I cannot test or fix your library because, as I said, I don't have any real iOS phone.

Let me know for any update about it! Thanks again!

[noahmehl]

@pichillilorenzo I'm totally fine with this. I imagine that it will get updated for video, it's just that, in life, all things are limited. If someone in the community wants to contribute/sponsor, this issue is here for reference. If we update it for video, I will update my comments here as well.

There is another concern with this implementation: which is bitcode has to be disabled. If we address this as well, I will comment here.

If this gets resolved in the OS, then this will be moot. Either way, the pod is available for people if they want to implement themselves (as wrapping wkwebview is trivial).

[breebee]

@pichillilorenzo and @noahmehl
I will sponsor! Development on my project lacks 2 things:

1. Webrtc support
2. Flutter desktop target support
   Please be my super hero's, how much do you guys need for this?

[noahmehl]

@breebee feel free to email me at: noahmehl@gmail.com.

[pichillilorenzo]

@breebee Flutter desktop target support is not yet available because PlatformView support is not yet implemented by the Flutter team, unfortunately! See:

• MacOS: Implement PlatformView support on macOS flutter/flutter#41722
• Linux: Implement PlatformView support on Linux flutter/flutter#41724
• Windows: Implement PlatformView support on Windows flutter/flutter#31713

[breebee]

@noahmehl just write [ time / price / paypal email ] and if I can afford to help inappwebview you will receive the payment in full trust. Thank you

[noahmehl]

@breebee I'm not discussing a price publicly. The only thing we are interested in working on is updating the shim to support video. It's likely to happen else wise, I just can't give a time frame.

[Manit-Chanthavong]

you can't post a non-working implementation of a library. You're costing people wasted time. At least put the known bugs or issues in the read me page.

[skakwy]

anyone got it to work ? please tell me someone did

[noahmehl]

Good news! We are extending the Pod for video support. I will update everyone when we have something.

[skakwy]

Good news! We are extending the Pod for video support. I will update everyone when we have something.

How long do you think it will take to add ?

[noahmehl]

Good news! We are extending the Pod for video support. I will update everyone when we have something.

How long do you think it will take to add ?

We're hoping for mid-December.

[MeherBa]

@noahmehl Did you update the pod to the latest version of cordova-plugin-iosrtc ?
is your pod support the latest Sip.js ? 0.17.1 it didn't work for me and i'm stuck for one week
Check this issue ticket out.

[noahmehl]

@MeherBa we haven't done anything yet. We do intend to update from cordova-plugin-iosrtc also.

[Faiizii]

web rtc is working in iOS 14.3. but inside inapp_webview, the page https://appr.tc is not loading on android side as expected, and on iOS side the call view/screen is not layout as it is layout in safari.

[noahmehl]

@Faiizii I think they only added getUserMedia(): https://webkit.org/blog/11353/mediarecorder-api/

I don't think they added the rest.

That being said, https://github.com/OpenTelecom/WKWebViewRTC has been updated to version 0.3.0 which should now support audio and video. We used: https://cordova-rtc.github.io/cordova-plugin-iosrtc-sample/index.html as the test.

@MeherBa @skillgamer @Manit-Chanthavong @breebee @pichillilorenzo ^^

Also, we're still actively working on it, especially against the twilio options. If you find anything please open an issue on the repo.

[MeherBa]

@noahmehl Thank you for the reply, i got this error ERROR: pluginMediaStream with id already exist with WKWebViewRTC plugin here is the issue link

[Faiizii]

@noahmehl the samples provided by WebRTC are working in iOS 14.+ safari browser and WKWebView. Still, WebRTC layout have few issues, but functionality is working perfectly.

[milzamhibatullah]

Any updates for this issue ? Bcz i've this problem too. Thank you ^^

[mixdesign]

Just updated to iOS 14.3, and now webRTC is working in WKWebView 🙂

[milzamhibatullah]

@mixdesign thanks for information ^^

[toklive]

is there an example for wkwebview (with IOS 14.3) with flutter_inappwebview ? same as the one we have for android with all permission implementation : https://medium.com/flutter-community/inappwebview-the-real-power-of-webviews-in-flutter-c6d52374209d

[pichillilorenzo]

@noahmehl Is there a way to add it with cocoapods without setting the plugin minimum ios platform version to 11.0?
I just want to "enable" the WebRTC feature only and only if the version is 11.0+ in order to support also older ios versions with my plugin.

I mean something like this:

public class InAppWebView: WKWebView, .... {
  if #available(iOS 11, *) {
    // enable WebRTC, add WKWebViewRTC custom user scripts, etc.
  }
}

[noahmehl]

@pichillilorenzo I think the minimum version in the pod is probably arbitrary at the moment. Honestly, we have no way of testing old versions (as Apple pretty much makes it impossible). What is the minimum version your app support?

Separately, the newest version on of the pod: 0.4.0 should handle any version from 11-14.3+. Even though some WebRTC features were added in WKWebView in 14.3, not all features have been added, etc.

[pichillilorenzo]

I'm not talking about a specific iOS App, but about the flutter plugin. My Plugin supports ios 8.0+.

Honestly, we have no way of testing old versions (as Apple pretty much makes it impossible).

There is no problem for me with that. It's ok to enable WebRTC to iOS 11.0+, but the problem is that when I add it to my cocoapods dependencies and run pod install, it fails with:

Analyzing dependencies
[!] CocoaPods could not find compatible versions for pod "WKWebViewRTC":
  In Podfile:
    flutter_inappwebview (from `.symlinks/plugins/flutter_inappwebview/ios`) was resolved to 0.0.1, which depends on
      WKWebViewRTC (~> 0.4.0)

Specs satisfying the `WKWebViewRTC (~> 0.4.0)` dependency were found, but they required a higher minimum deployment target.

That's obvious because the minimum ios version for WKWebViewRTC is 11.0.
So, I'm asking if there is a way to add it whatever my Plugin minimum version platform is and then "enable" it using something like:

if #available(iOS 11, *) {
  // enable WebRTC, add WKWebViewRTC custom user scripts, etc.
}

Is it possible? Thanks again!

[pichillilorenzo]

Ok, I solved it by myself using a custom modification of https://github.com/cordova-rtc/cordova-plugin-iosrtc to make it work on my plugin and, also, I solved the problem related to the Google WebRTC minimum ios version (11.0) adding a specific subspec to my plugin flutter_inappwebview.podspec file.

However, @noahmehl I tried also your WKWebViewRTC ios plugin but there is a memory leak: the class WKWebViewRTC can set an instance of WKWebView using the WKWebViewRTC constructor but, after that, cannot be set to nil.
For example, this happens when the WebView should be removed from the current View but it remains on memory (just check on Safari developer tools) because WKWebViewRTC continues to have a reference on it (the WKWebViewRTC.webView attribute).
I think you should create a public dispose method in order to be able to make it nil and for removing the user scripts using WKUserContentController.removeScriptMessageHandler() method, otherwise, the WKWebView instance will never be removed from the memory.

[noahmehl]

@pichillilorenzo thanks for the update. And we will look into the memory issue. I really appreciate the help!

[pichillilorenzo]

After hours of tests using the cordova plugin iosrtc swift code workaround, I decided to not use it because there are a lot of limitations in general. For example, one limitation is that the streams are not really part of the webpage document, they are just other native UIViews on top of the native WKWebView, so video HTML element that should contains the video stream is just a placeholder.
Also, because the streams are native UIView, it could happen that you need to refresh them manually using custom javascript code, for example, because of WebView scrolling or other things.
Other problems come if you need to have HTML buttons over the video streams. This is a mess because you need to change the CSS of the webpage to make it transparent and somehow put the UIView below the WKWebView.
For other things check the cordova plugin FAQ page: https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/FAQ.md and the issue page: https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues.

For cordova/ionic apps, probably the cordova plugin iosrtc works well because it is all wrapped inside a single WebView, but here in Flutter, where the WebView is just one of many widgets of an App, it doesn't work very well and it gets a bit messy quickly.

I have just found that the WebRTC feature has been added to WKWebView starting from iOS 14.3+ (https://blog.bitsrc.io/ios-14-3-brings-webrtc-to-wkwebview-closing-gap-on-ios-accessibility-90a83fa6bda2), so I will not add this feature using the cordova plugin workaround for iOS < 14.3.
On iOS, WebRTC will work automatically starting from 14.3+.

iOS Usage

To make it work with this plugin, you need to set the cross-platform option mediaPlaybackRequiresUserGesture to false and set the iOS-specific option allowsInlineMediaPlayback to true, for example:

initialOptions: InAppWebViewGroupOptions(
  crossPlatform: InAppWebViewOptions(
    mediaPlaybackRequiresUserGesture: false,
  ),
  android: AndroidInAppWebViewOptions(
    useHybridComposition: true
  ),
  ios: IOSInAppWebViewOptions(
    allowsInlineMediaPlayback: true,
  )
),

Note that on iOS, to be able to play video inline, the video HTML element must have the playsinline attribute, for example:

<video autoplay playsinline src="..."></video>

Also, you need to request camera and microphone permissions, for example using the permission_handler flutter package:

import 'package:permission_handler/permission_handler.dart';

Future main() async {
  WidgetsFlutterBinding.ensureInitialized();

  await Permission.camera.request();
  await Permission.microphone.request();

  runApp(MyApp());
}

In your Info.plist file, you need to add also the following properties:

<key>NSMicrophoneUsageDescription</key>
<string>Flutter requires acess to microphone.</string>

<key>NSCameraUsageDescription</key>
<string>Flutter requires acess to camera.</string>

If you open this file In Xcode, then the NSMicrophoneUsageDescription property is represented by Privacy - Microphone Usage Description and
NSCameraUsageDescription is represented by Privacy - Camera Usage Description.

To test it, you can try to visit https://appr.tc/.

[Faiizii]

The named parameter 'useHybridComposition' isn't defined. I am using flutter_inappwebview: ^4.0.0+4. It is the latest version on pub.dev

[Faiizii]

allowsInlineMediaPlayback: true, is working for camera previews positioning, but call controls (mute,fullScreen etc) are still not visible in iOS build.

[pichillilorenzo]

@Faiizii useHybridComposition is on the new version. It isn't published yet, so for flutter_inappwebview: ^4.0.0+4, you can just ignore that option.
Instead, about call controls, the <video> HTML element must have the controls attribute, otherwise media controls won't show up.

[Faiizii]

control attribute enables the videos controls. My issue is web RTC controls (enable/disable camera/mic, hang call etc) are not appearing on iOS (IPhone7 OS 14.3)

[pichillilorenzo]

@Faiizii It seems there are some html/css issues on https://appr.tc/ website if you use iOS. So, it is not related to my plugin.
If you inspect the web page using Safari Developer Tools, you can find the svg icons related to call controls
(under <div id="icons"> <svg id="mute-audio" ... /> ... </div>) and remove the css style transform: translateX(calc(-6vw - 96px)).
For some reason, on iOS, they don't appear when you touch the video, but as I said, it is not related to my plugin.
In fact, if you go to that website using also the Safari iOS App, you will see the same problem.

[busyhe]

Every time you open a web page, you need to reauthorize it