Drawing HTML on top of a UIView is possible #38

telemakhos · 2015-08-03T18:55:45Z

Q: Can I place HTML elements (buttons and so on) on top of active

Is this really accurate?

Not sure if it'll work with a video layer, but for the Google Maps phonegap plugin drawing HTML on top of a native UIView is not a problem. Actually it works wonders...

Here's the trick.

So just passing by and putting this on your radar guys.

Hope it helps!

ibc · 2015-08-03T19:09:26Z

The aim of the cordova-plugin-iosrtc is to provide the WebRTC API without requiring custom stuff. If the video element is placed behind the Cordova HTML UIView and the background of the HTML view is not transparent, then the video would not be shown. How would that work if a <body> background is desired (for example a bitmap texture or a gradient)?

Said that, I may add an option to set this behavior per <video> element. Any suggestion? Perhaps a custom CSS field? a custom data-xxxx="true" attribute?

ibc · 2015-08-03T19:10:04Z

Not sure if it'll work with a video layer

Yes, sure.

telemakhos · 2015-08-03T19:19:17Z

How would that work if a background is desired (for example a bitmap texture or a gradient)?

Users can still set a desired background. Here's how the gmaps plugin solves the issue:

Map.setBackgroundColor()

Yes, sure.

That's great news then. Might be worth exploring more in depth this option...

Any suggestion?

Not really, you know better than anyone what solution would fit better in your project and I just know the basic internals of the gmaps plugin. As a user I think that gmaps makes a great job providing a dummy transparent layer where you can draw freely any HTML element. That'd be undoubtedly the best solution since the users could draw a proper UI on top of the vid elements...

In any case I'm going to continue researching about this issue. I'll get back to you if I find anything worth looking at or if I have any interesting idea...

murillo128 · 2015-08-03T19:37:23Z

How about?

z-index: -1;

ibc · 2015-08-04T09:45:02Z

Using "z-index: -1" is what I consider a non appropriate usage of the CSS property. This is, setting "z-index: -1" may have very different results when in a web browser with native WebRTC support and when in an iOS Cordova device.

I would prefer having a custom data-xxxx attribute that would be ignored in the desktop browser or Android browser:

<video data-iosrtc-rear="true" autoplay></video>

Thoughts?

murillo128 · 2015-08-04T10:00:46Z

We are already using the z-index in a non-appropriate way to place the videos relative to other videos, so " may have very different results when in a web browser with native WebRTC" is already happening.

murillo128 · 2015-08-04T10:01:39Z

Also, that way we ensure that the app has to put a transparent background in order the video to be seen.. ;)

telemakhos · 2015-08-04T15:45:08Z

The issue targeting CSS to only iOS (or web or Android) has several solutions. Ionic uses classes like…

.platform-browser video_div { z-index: x; }
.platform-android video_div { z-index: y; }
.platform-ios video_div { z-index: z; }

and also allows you to use dynamic targeted CSS using platform and ng-class (this is for angular but can be done with vanilla js).

The main issue here is abstracting everything and letting the users easily draw custom controls. I think this could be a very valuable differentiation feature for iosrtc. I’ve thought that an interesting option could be providing default CSS controls with the plugin (like the HTML5 video tag does) and allow the users to enable/customise them via a JSON object and overriding CSS properties… something like

{
  "enableDefaultControls": true,
  "controls": {
    "startCall": true,
    "stopCall": true,
    "holdCall": true,
    "fullScreen": true,
    "switchCamera": false,
    "mute": true,
    etc...
  }
}

Then provide default CSS classes (.stopCallButton, .startCallButton, .fullScreen, .fullScreenButton, etc) for both iOS and Android (xwalk):

SCSS example:

.fullScreenButton {
    position: absolute; 
    bottom: 0;
    float: right;

    /* PNG Icon */
    background-image: url('path/to/fullScreenButton.png');
    background-repeat: no-repeat;
    background-position: 25px 0px;

    /* Scalable Vector Icon */
    i {
        font-size: 25px;
        color: white;
        vertical-align: middle;
    }

    /* Platform Specifics */
    .platform-android { 
        // Android Stuff
    }

    .platform-ios { 
        // iOS Stuff
    }

    /*  Modify appearance depending on the resolution */
    @media (max-width: $screen-sm-max) {
        display: none;
        etc…
    }
}

The defaults could result in a badass UI like:

This way the users would have a straightforward out-of-the-box solution to use controls on top of the video, and also a starting point with simple logic in case they want to go further customising their UI’s.

Just an idea...

ibc · 2015-08-05T08:52:38Z

I’ve thought that an interesting option could be providing default CSS controls with the plugin (like the HTML5 video tag does) and allow the users to enable/customise them via a JSON object and overriding CSS properties… [...]
Then provide default CSS classes (.stopCallButton, .startCallButton, .fullScreen, .fullScreenButton, etc) for both iOS and Android

There are no start/stop/hold/fullscreen "buttons" in the WebRTC spec, and there shouldn't be. That assumes "softphone vision" and WebRTC is beyond that.

I agree that rear video UIView is useful, but just that. It's up to the app/developer to place "control" buttons on top of the video view when needed.

Regarding the usage of CSS classes to tell the plugin whether to put the video UIView in front/rear, I don't like that approach. CSS classes are up to the application and they are supposed to provide custom style/design to the app. I'd rather custom data-xxxxx attributes as suggested above, as that is exactly the purpose of data-xxxx attributes: to add custom properties to the HTML element. Or the negative z-index...

ibc · 2015-08-05T09:06:03Z

Said that, I will implement the feature this morning XD

telemakhos · 2015-08-05T09:14:03Z

There are no start/stop/hold/fullscreen "buttons" in the WebRTC spec, and there shouldn't be. That assumes "softphone vision" and WebRTC is beyond that.

Yeah I know, I was just trying to get around the UI problem and the question of placing HTML on top. My suggestion was just an idea for convenience.

Said that, I will implement the feature this morning XD

Fair enough... 😁

1N50MN14 · 2015-08-05T09:26:29Z

Hi, first thanks for the wonderful work @ibc (and eface2face for supporting it), where's the BUY_ME_BEER.md in the repo? ;)

To shime on this issue (if I may):

The "rear" video UIView was the first thing I personally looked for when reading through the docs yesterday. A gazillion +1 from me on this one I believe it's a fundemental thing to have.
I feel everything-CSS that is otherwise for the purpose of demonstration (and you already have a an external demo repo) should remain outside the scope of this project. Sorry @telemakhos, nice screenshot though :)
I personally perfer data-xxxx, but that's just taste imho. If you already rely on z-index:-1 kind-of-thingie then might just as well stick to that, both approaches are equally fine, as long as the functionality is there :)

Gload to see someone did actually nail this project for a change :)

telemakhos · 2015-08-05T09:58:53Z

I feel everything-CSS that is otherwise for the purpose of demonstration (and you already have a an external demo repo) should remain outside the scope of this project. Sorry @telemakhos, nice screenshot though :)

I didn't know about the demo app!... I guess that would be definitely a better option for the reference client with controls that I was suggesting.

1N50MN14 · 2015-08-05T10:01:07Z

https://github.com/eface2face/iOSRTCApp :)

ibc · 2015-08-06T08:31:27Z

Guys, I tried yesterday with zero success. Don't know why but I was not be able to place the video UIView behind the web view and make the web view transparent. I also had issues when using the zPosition attribute of the layer video layer.

I will try it again next week.

telemakhos · 2015-08-06T13:08:25Z

No worries. Are you trying to replicate the gmaps technique stacking the subviews programatically or are you trying to come up with a custom solution?

ibc · 2015-08-06T14:08:12Z

Let's say I must expend more time looking into phonegap-googlemaps-plugin...

RangerMauve · 2015-08-14T13:47:15Z

This would be super useful. Right now I've had to manually hide the video view when using the phonertc plugin whenever I wanted to render anything over the video.

ibc · 2015-08-14T13:57:40Z

It would be really useful, but my time is limited. PR are welcome.

oNaiPs · 2015-08-14T23:47:57Z

@ibc I could try to help on that issue. This is one of the few issues I have to fix before using your project. Did you actually create a branch and started working on it?

Thanks

ibc · 2015-08-15T23:46:05Z

I started working on it with no success. I will upload a branch on Monday and notify it here.
Thanks a lot.

ibc · 2015-08-18T11:09:18Z

I've upload a new branch "rear-video" to implement this feature. For now it has the same content as master since nothing that I tried did work so I've removed it (it was useless).

Notes and problems I remember:

My idea is that a negative z-index makes the video element UIView to be placed behind the HTML view (the Cordova UIView).
Stuff must be done in src/MediaStreamRenderer.swift.
Note that I use two UIView:
- elementView: It has same size as its corresponding HTML <video> element.
- videoView: It is placed over the elementView. Its purpose is to have a different size (which is cropped by the parent view) so CSS object-fit can be implemented (already done).
So we must work on the elementView leaving the videoView untouched.
In my attempts I was not able to place the elementView behind the Cordova view (it became invisible even if the HTML has no background color, not sure if something else must be done in the HTML UIView).
I expect issues due the usage of self.elementView.layer.zPosition = CGFloat(zIndex) since it seems that zPosition may alter how different layers are rendered. Currently if a negative z-index is set for the video element, the elementView just disappears.

Hope somebody can help on this :)
If so, please fork the "rear-video" branch and let work on it.

yocontra · 2015-08-18T19:57:38Z

This is the last outstanding issue preventing us from being able to use this in production. If you need anything at all, I'm down to help.

ibc · 2015-08-18T20:01:21Z

How to help is explained in my previous comment.

derMani · 2015-09-11T10:26:12Z

Hi IBC, in previous projects, we implemented an RTMP (flash) Video component for iOS as a native video component (without Cordova though), which runs in the background of a WebView component.
With transparent body settings it was possible to use it in the background and it worked pretty well vor us.

We want to move away from flash now by using WebRTC for future projects. I don't know if I can help you with this, but all we had to do in our project was this:

- (void)viewDidLoad
{
  /*** other stuff ***/ 

// set WebView Transparent
 webView.opaque = NO;
 webView.backgroundColor = [UIColor clearColor];
}

In the HTML, i used

body {
  background-color: transparent
}

If you knew this already, sorry for bothering you ;) I just came along this thread and thought this could help

makc · 2015-09-11T18:00:07Z

maybe a bit of off-topic, but can video with src = URL.createObjectURL(stream) be used with gl.texImage2D as here? seeing all these UIView refs I guess it can't, correct?

edit: #65

ibc · 2015-09-12T09:54:20Z

@derMani thanks a lot for pointing that out. I will check it soon.

@makc sorry, I don't know what that is about. Anyhow please feel free to comment about it in another issue.

1N50MN14 · 2015-09-21T17:28:24Z

@ibc Did you ever get the chance to look into derMani's workaround?

ibc · 2015-09-21T17:29:32Z

Not yet, sorry.

acylum · 2015-10-05T16:35:20Z

Maybe this can work: Include a local HTTP server in the webRTC plugin to serve the video content to the video tag - this can hopefully allow both overlayed HTML elements along with full CSS styling control. We used this technique when we created an RTMP plugin...

1N50MN14 · 2015-10-26T21:49:09Z

This is in the works, so far the video successfully playing the background (HTML body must have transparent background), z-index support with negative values will be completed this week so that multiple videos (UIViews) are stacked on top of each other in order. Anyone that wishes to help testing this prior to sending a PR please feel free to PM me.

acylum · 2015-10-30T13:39:11Z

Sweet! I'd like to test it. Currently, my work around is to launch a wizviewmanager child window over the main view - so basically the webrtc video is "sandwiched" in-between. You have to use the window messaging to allow button interaction from wizview back to your main view.

1N50MN14 · 2015-10-30T15:57:53Z

@acylum awesome thanks! just replied you :)

jkuri · 2015-11-12T02:14:42Z

+1

ibc · 2015-11-12T22:19:12Z

+1 are good, but pull requests are even better :)

1N50MN14 · 2015-11-26T16:57:27Z

@ibc I have this working in my fork, see my last commit here: https://github.com/1N50MN14/cordova-plugin-iosrtc/commit/30ccdaa583b1f33f790bb8db3a5789fcdd8088e4 I'm not able to send a PR just yet, but feel free to take it for a spin in the meantime

ibc · 2015-11-26T17:00:14Z

Great! Let me handle that on next week.

1N50MN14 · 2015-11-26T17:02:25Z

Take your time

derMani · 2015-12-21T17:03:35Z

Hey guys, just wanted to know if you have an update on this.

1N50MN14 · 2015-12-23T11:13:54Z

@derMani I submitted a PR for this one, waiting for ibc to look into it and eventually approve it, otherwise you can use my fork for now.

ibc · 2015-12-23T11:16:32Z

Guys, I won't be able to merge the PR until the second week of the new year.

1N50MN14 · 2015-12-23T11:19:30Z

@ibc No worries, Merry Christmas! ;)

ibc · 2015-12-23T11:19:56Z

Same! :)

derMani · 2016-01-11T15:47:51Z

Happy new year everyone! @1N50MN14 your PR works superb 👍 Thx :)

1N50MN14 · 2016-01-11T16:29:42Z

@derMani Du bist willkommen und Frohes Neues Jahr :)

telemakhos · 2016-01-11T17:16:24Z

Glad to see this is finally coming to fruition. Great job @1N50MN14 . I guess the next goal is supporting the trick on both UIWebView and the new WKWebView... see #90

1N50MN14 · 2016-01-11T21:47:33Z

@telemakhos Yup I can make that work once @saghul submits his PR for WKWebView support

bassemBMI · 2016-02-22T21:46:58Z

I used @1N50MN14 branch but the video doesn't work.
@ibc there is any fix planned for this version ?

ibc · 2016-02-24T15:28:36Z

Hi, I'm sorry I couldn't properly check the status of the PR #98. I commented some issues regarding the PR (for example: /lib/libWebRTC-LATEST-Universal-Release.a can NOT be removed from the repo) and I'm not sure whether that PR handles all the required use cases without breaking working code (in fact I've read some users reporting wrong behavior).

So, if somebody can spend time to test #98 and provide proper conclusions I will merge it.

xpluscal · 2016-04-26T15:34:29Z

Sorry to bother you again @ibc, but are there any news regarding this issue?

bitjson · 2016-04-26T15:45:33Z

Not directly related to this repo, but a plugin I work on just released support for rendering the camera preview layer behind a transparent UIWebView or WKWebView: bitpay/cordova-plugin-qrscanner

It's just a simple plugin, but might help if someone is trying to implement the same thing here: https://github.com/bitpay/cordova-plugin-qrscanner/blob/master/src/ios/QRScanner.swift

bassemBMI · 2016-04-26T16:04:38Z

Hi @ibc I join m'y voici to @xpluscal in order to have an idea when it can be fixed.
Thx.

ibc · 2016-04-26T17:09:45Z

I think the status regarding this PR is already explained in comments above, but I'll try to summarize here:

The author of the PR has already said that he's working in his own project so it's expected that he's no longer maintaining the code in this PR.
There are also comments from some users that tested the PR and they reported some issues in some cases.
I don't have time to test all the required use-cases nor to look at how other projects implement a similar feature.

So, if someone wants to contribute, take this PR, apply it into a fork of the official master branch of the plugin, check all the use-cases and document them, then I'll be very happy to accept a new PR.

derMani · 2016-05-18T16:47:53Z

Just upgraded to 3.0.0 of the Plugin and the PR still works for me.

The steps are basically the same as before (and suggested in the PR)

Hide Video Controls for video element in Javascript
Edit PluginMediaStreamRenderer.swift to make sure the webview is transparent.

I think I could provide a small example project, if you wish.

@ibc What exactly are "all use cases" ? :-)

I just created a web page, made sure everything had the correct CSS and after that I was able to see the video through my transparent elements.

Maybe that's the problem for some people here, you have to make everything transparent on your page, if you want to look through it. It's easy to forget about a few elements, which might still have a background-color (like body, html, ...)

chris087 · 2016-05-21T09:53:21Z

Actually is not working for me.

Using 3.0.0;

Having all transparant. (verified several times).

body {
background-color: transparent;
}
html {
background-color: transparent;
}
.pane, .view {
background-color: transparent;
}

[nav-view-transition="ios"][nav-view-direction="forward"], [nav-view-transition="ios"][nav-view-direction="back"] {
background-color: transparent;
}

#myMiniVideo {

width:30%;
height:30%;
position:absolute;
right:0;
bottom:0;
z-index:0;

}

#videoscreen .video_controls {

position:absolute;
right:50px;
bottom:50px;
z-index:15;

}

Video overlays the .video_controls... :(
no matter what z-index the myMiniVideo have. When setting -1 z-index to myMiniVideo, the video is not visible.

Changes in PluginMediaStreamRenderer.swift :

// The browser HTML view.
self.webView = webView
self.eventListener = eventListener
self.webView.opaque = false
self.webView.backgroundColor = UIColor.clearColor()
// The video element view.
self.elementView = UIView()
// The effective video view in which the the video stream is shown.
// It's placed over the elementView.
self.videoView = RTCEAGLVideoView()
    self.webView.superview?.addSubview(self.elementView)

    self.elementView.userInteractionEnabled = false
    self.elementView.hidden = true
    self.elementView.backgroundColor = UIColor.clearColor()
    self.elementView.addSubview(self.videoView)
    self.elementView.layer.masksToBounds = true

    self.videoView.userInteractionEnabled = false

ibc · 2016-05-23T16:16:31Z

@derMani

What exactly are "all use cases" ? :-)
I just created a web page, made sure everything had the correct CSS and after that I was able to see the video through my transparent elements.

I mean how it works. Which values do z-index accept and what happens if z-index is negative. And what happens if a <video> is placed on top of another <video> with different z-index, etc.

To be clear: I don't have an iOS device so I won't check all the cases, that's why I need complete documentation regarding how this PR works.

BTW as you can see, a comment after your says that it fails. If I accept this PR as it is, I'll become responsible of such a problem.

ibc · 2016-06-14T11:04:24Z

Done in #179

cozzbie · 2017-02-12T09:47:16Z

Is there anyway to redraw the rendering video onto a HTML5 canvas?

alexp25 · 2020-08-30T11:50:31Z

Is it possible to hide the video tag from view? This way I imagine it would be possible to show the video in canvas while the original video is playing in the background, hidden from the viewport

ibc closed this as completed Jun 14, 2016

Drawing HTML on top of a UIView is possible #38

Drawing HTML on top of a UIView is possible #38

Comments

telemakhos commented Aug 3, 2015

ibc commented Aug 3, 2015

ibc commented Aug 3, 2015

telemakhos commented Aug 3, 2015

murillo128 commented Aug 3, 2015

ibc commented Aug 4, 2015

murillo128 commented Aug 4, 2015

murillo128 commented Aug 4, 2015

telemakhos commented Aug 4, 2015

ibc commented Aug 5, 2015

ibc commented Aug 5, 2015

telemakhos commented Aug 5, 2015

1N50MN14 commented Aug 5, 2015

telemakhos commented Aug 5, 2015

1N50MN14 commented Aug 5, 2015

ibc commented Aug 6, 2015

telemakhos commented Aug 6, 2015

ibc commented Aug 6, 2015

RangerMauve commented Aug 14, 2015

ibc commented Aug 14, 2015

oNaiPs commented Aug 14, 2015

ibc commented Aug 15, 2015

ibc commented Aug 18, 2015

yocontra commented Aug 18, 2015

ibc commented Aug 18, 2015

derMani commented Sep 11, 2015

makc commented Sep 11, 2015

ibc commented Sep 12, 2015

1N50MN14 commented Sep 21, 2015

ibc commented Sep 21, 2015

acylum commented Oct 5, 2015

1N50MN14 commented Oct 26, 2015

acylum commented Oct 30, 2015

1N50MN14 commented Oct 30, 2015

jkuri commented Nov 12, 2015

ibc commented Nov 12, 2015

1N50MN14 commented Nov 26, 2015

ibc commented Nov 26, 2015

1N50MN14 commented Nov 26, 2015

derMani commented Dec 21, 2015

1N50MN14 commented Dec 23, 2015

ibc commented Dec 23, 2015

1N50MN14 commented Dec 23, 2015

ibc commented Dec 23, 2015

derMani commented Jan 11, 2016

1N50MN14 commented Jan 11, 2016

telemakhos commented Jan 11, 2016

1N50MN14 commented Jan 11, 2016

bassemBMI commented Feb 22, 2016

ibc commented Feb 24, 2016

xpluscal commented Apr 26, 2016

bitjson commented Apr 26, 2016

bassemBMI commented Apr 26, 2016

ibc commented Apr 26, 2016 • edited Loading

derMani commented May 18, 2016 • edited Loading

chris087 commented May 21, 2016 • edited Loading

ibc commented May 23, 2016

ibc commented Jun 14, 2016

cozzbie commented Feb 12, 2017 • edited Loading

alexp25 commented Aug 30, 2020 • edited Loading

ibc commented Apr 26, 2016 •

edited

Loading

derMani commented May 18, 2016 •

edited

Loading

chris087 commented May 21, 2016 •

edited

Loading

cozzbie commented Feb 12, 2017 •

edited

Loading

alexp25 commented Aug 30, 2020 •

edited

Loading