-
-
Notifications
You must be signed in to change notification settings - Fork 338
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
canvas drawImage(videoElem...) cause memory leak via RequestFrameAnimation #582
Comments
From what I see the only method to get a frame into canvas right now is use render.save() to an image element and then load that image into canvas. Works but sound a very bad idea. Also if I do this repeatedly I get flooded with If anyone knows of a better way to continuosly get camera frames into canvas in iOS I would love to know about it. |
It's possible to capture the frame data from the existing buffer. But the image format is I420. It might be possible to send it to this canvas adapter.. https://github.com/brion/yuv-canvas |
Looks like this fork https://github.com/PeterXu/cordova-plugin-iosrtc has cordova.plugins.iosrtc.MediaStream() that can use a canvas. Did not had the oportunity to fully investigate this week, but will try ASAP. |
@namello-zmtp This can already be done using existing iosRTC API, we could implement a SHAM/SHIM for canvas.drawimage but currently its not planned. Example: TLDR: only once video started playing via after "canplay" event access render property of the video tag and use save method with callback.
It does and we may soon backport or colaborate with @PeterXu to put some of his work back in iosRTC, depending the time available from both contributors. Notice that this fork does not contains all fixes we made since May 2019: Also see related comment: |
Note: We can easily fix the warning "Plugin should use a background thread" for render.save. |
Its easy peasy in fact to add support for drawnimage if you understand how Javascript works. Example: // Apply CanvasRenderingContext2D.drawImage monkey patch
var drawImage = CanvasRenderingContext2D.prototype.drawImage;
CanvasRenderingContext2D.prototype.drawImage = function (arg) {
var args = Array.prototype.slice.call(arguments);
var context = this;
if (arg instanceof HTMLVideoElement && arg.render) {
arg.render.save(function (data) {
var img = new window.Image();
img.addEventListener("load", function () {
args.splice(0, 1, img);
drawImage.apply(context, args);
});
img.setAttribute("src", "data:image/jpg;base64," + data);
});
} else {
return drawImage.apply(context, args);
}
}; I will make a PR for testing. |
Please try this PR see testing instructions: |
Duplicate #116 |
Fixed on master a98e6ad |
Related #594 |
document.body.innerHTML = "";
var canvas = document.createElement('canvas');
document.body.appendChild(canvas);
var video = document.createElement('video');
video.autoplay = true;
document.body.appendChild(video);
const onFrame = () => {
// Resize canvas if video siwe change
if (
canvas.width !== video.videoWidth ||
canvas.height !== video.videoHeight
) {
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
}
const canvasContext = canvas.getContext('2d');
canvasContext.drawImage(video, 0, 0);
// setTimeout(onFrame, parseInt((1000/60)*25));
window.requestAnimationFrame(onFrame);
};
// Refresh video tags position
if (window.cordova && window.cordova.plugins.iosrtc) {
setInterval(function () {
cordova.plugins.iosrtc.refreshVideos();
}, 500);
}
navigator.mediaDevices.getUserMedia({
video: true,
audio: false
}).then((stream) => {
video.oncanplay = (event) => {
window.requestAnimationFrame(onFrame);
};
video.srcObject = stream;
}).catch((error) => {
console.log(error);
}); Tested and working on master. |
|
@hthetiot is this implementation performant? If this is the case, this plugin is amazing and solves all problems on iOS, as you can have the video in a canvas and use it as a "common" DOM element. |
@gabrielenosso I did not try this, but had tried before the
method. It did not seem performant at all. actually I could see the iphone killing the video for using too much resources. |
@fimdomeio If you can give me some more input, I could try to go deeper. |
Try this tell us: |
Yes but we have SHIM for that that what we use for now.
If new underlying implementation of the API change you should not care, the render.save may get more render methods to compare performance in future released |
@hthetiot, @gabrielenosso, Trying the example above appears to work quite well in terms of framerate. I see a continuous memory increase (until app crashes), that appear to be caused by |
See #594 I think there is a memory leaks... |
@fimdomeio can you try this PR (see testing instruction), this should solve memory leak: |
please confirm the fix #600 solve your issue @fimdomeio |
@hthetiot On it. Just stuck on xcode update and a not so fast internet connection. will give feedback ASAP. |
From my testing #600 does not fix the leak. |
My sample test project is at https://github.com/fimdomeio/iosrtc-example We got to the point where we realised that the memory leak disapears if in src/PluginMediaStreamRenderer.swift on the save function
we return "0"
We suspect the leak might be happening when the string is copied from swift to javascript cordova-plugin-iosrtc/src/iosrtcPlugin.swift Line 972 in 66acf1a
Are you able to reproduce the memory leak or need more detailed info? |
This help, keep you posted @fimdomeio |
I changed issue title cause 6.0.14 has drawImage(video) support, but its not usable via RequestFrameAnimation due memory leak. I need to release #605 first via 6.0.16 |
memory leaks for video to canvas wil be addressed in 6.1.1 |
Try master or wait for 6.0.17 see #609 |
@fimdomeio @namello-zmtp fixed in 6.0.21 and 8.0.0 via master |
Please read first!
Please use Public Google Group (mailing list) for general technical discussions and questions.
extra/renderer-and-libwebrtc-tests.js
file).Note: If the checkboxes above are not checked (which you do after the issue is posted), the issue will be closed, removing this checkbox will result in automatic closed issue.
Versions affected
Description
It appears that the plugin doesn't emit the necessary data for canvas drawImage(videoElem...) to function (capture the current frame basically) this prevents libraries like https://github.com/zxing-js from working.
Steps to reproduce
canvas: CanvasRenderingContext2D;
videoElem: HTMLVideoElement;
canvas.drawImage(videoElem, 0, 0); // Doesn't error but data isn't valid
Expected results
The frame should be copied over
Actual results
Nothing is copied over
The text was updated successfully, but these errors were encountered: