Black Bars around video stream and video stream is stuck when scrolling

[aspdev91]

• I have used Google with the error message or bug in association with the library and Cordova words to make sure the issue I'm reporting is only related to iOSRTC.
• I have provided steps to reproduce (e.g. sample code or updated extra/renderer-and-libwebrtc-tests.js file).
• I have provided third party library name and version, ios, Xcode and plugin version and adapter.js version if used.

Versions affected

• Cordova version (e.g 7.1.0): 10.0.0
• Cordova iOS version (e.g 5.1.0): 6.1.1
• Plugin version (e.g 6.0.12): 6.0.13
• iOS version (e.g 10.2): 13.7
• Xcode version (e.g 11.1 - 11A1027): 12.2 Beta
• WebRTC-adapter version (e.g. 7.4.0):
• WebRTC Framework version (e.g. JSSip 3.1.2):

Description

I'm not able to get a 4/3 video atm. It's also stuck to the screen as seen in the video below. I tested the same javascript code with on a hosted environment and accessed it with Safari. It worked perfectly fine. Below is a short video of what I encountered.

https://youtu.be/KiO-VVzpV-w

Steps to reproduce

If you run this code, you should be able to reproduce. I've tried numerous combinations of the constraints in getUserMedia using back and front cameras alongside different resolutions to no avail.

 cordova.plugins.iosrtc.registerGlobals();
 var appContainer = document.body;
  var localVideoEl;
  navigator.mediaDevices
    .getUserMedia({
      audio: false,
      video: {
        width: 640, 
        height: 480,
        aspectRatio: 4 / 3,
      },
    })
    .then(function (stream) {
      localVideoEl = document.createElement("video");
      localVideoEl.style.width = "640";
      localVideoEl.style.height = "480";
      localVideoEl.setAttribute("autoplay", "autoplay");
      localVideoEl.setAttribute("playsinline", "playsinline");
      localVideoEl.srcObject = stream;
      appContainer.appendChild(localVideoEl);

})

Expected results

A 4/3 video that is not stuck to the screen. I also fed the video in a image recognition model and the confidence score was super low. I think it's because it's feeding in those black bars as well since I assume its part of the video. I tested the same javascript code on a hosted environment and accessed it with Safari. It worked perfectly fine.

Actual results

See video below:
https://youtu.be/KiO-VVzpV-w

[hthetiot]

@aspdev91 The layout constraints warning have nothing to do with your issue see #422 that why i have hide your comment above.

If you want to update video location on scrool you need to call cordova.plugins.iosrtc.refreshVideos.

Here is an example to do that automaticly. We may add that by default later.

// Scan every 500ms for refreshing video tag position and on various user events.
var scanVideoTagTimer,
    scanVideoTagInterval = 500;

function scanVideoTag() {
    clearTimeout(scanVideoTagTimer);
    var shouldRefreshingVideos = $window.document.querySelectorAll('video').length > 0;
    if (shouldRefreshingVideos) {
        $window.cordova.plugins.iosrtc.refreshVideos();
    }
    scanVideoTagTimer = setTimeout(scanVideoTag, scanVideoTagInterval);
};

// Start scanVideoTag
scanVideoTagTimer = setTimeout(scanVideoTag, scanVideoTagInterval);

window.onorientationchange = scanVideoTag;
window.addEventListener('touchstart', scanVideoTag);
window.addEventListener('click', scanVideoTag);
window.addEventListener('touchmove', scanVideoTag);
window.addEventListener('touchend', scanVideoTag);

related: #478

[hthetiot]

@aspdev91 The black bar issue you have on capture can be related to this bug

• fix: use video view for correct screencap framing #570

I do recomand you try master until 6.0.14 is released and this but with it:

cordova plugin remove cordova-plugin-iosrtc --verbose
cordova plugin add https://github.com/cordova-rtc/cordova-plugin-iosrtc#master --verbose
cordova platform remove ios --no-save
cordova platform add ios --no-save

[hthetiot]

Let me kow if this 2 thing works for you and i will close the issue.

[hthetiot]

Notice that to get Image to canvas you need to use special iosrtc code see example here:

• https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/extra/renderer-and-libwebrtc-tests.js#L405

[aspdev91]

@hthetiot Thank you for the help on this.. This was a blocker for us.

The scrolling issue is resolved with the code you provided.

For the image canvas, I tried to run the test code and got this error in Safari debug tools:

Unhandled Promise Rejection: TypeError: undefined is not an object (evaluating 'localVideoEl.render.save')

For the black bar, I was on latest master when I made this post and I just pulled master again using your command. Unfortunately, the black bars are still there.

[hthetiot]

For the image canvas, I tried to run the test code and got this error in Safari debug tools:

localVideoEl need to be your element and you need to propably do that in a setTimeout or after the video canplay event.

For the black bar, I was on latest master when I made this post and I just pulled master again using your command. Unfortunately, the black bars are still there.

Please create separate issue that would help to keep issue scope.

[hthetiot]

Black Bars around video stream

Did you use object-fit also it will depend the size of your video tag

[aspdev91]

@hthetiot The object fix resolved the black bars! Thanks!

After some tinkering, the canvas is working as well. Thanks @hthetiot

I have a follow up question on alternatives for the canvas as this adds performance overhead. I'll post it a separate issue.

[hthetiot]

@aspdev91 glad it is solved for you, I plan to add SHIM to emulate Original Stream to Canvas in the future.

For object-fit CSS here is quick reference for people landing on this issue:
https://css-tricks.com/almanac/properties/o/object-fit/