JS Heap OOM Exception on video download/remux #444
Comments
|
log capture from a different attempt, after manually removing a bunch of cached log files, mediainfo jsons, mp4s of failed transcodes - managed to get all the way through the transcode but crashed after mediainfo |
|
oh no, this is probably the worst thing one can encounter... really hoping this is just a node version error |
|
I appear to be having this issue with normal stream capture as well and not just the vod download. it seems to be something related to the remux specifically that is causing the problem. I can open the recorded ts file in cache/capture and it recorded the entire stream, but the remux process appeared to create a corrupt MP4 file in the vods folder and crash the entire server after the stream ends. attached are the logs with debug mode on, stream ended at 00:17 2023-06-10.log |
|
interestingly, running the ffmpeg command in a docker exec session crashes the LiveStreamDVR server process without my having interacted directly with the application at all, leads me to believe the culprit is something listening to file change events as the remuxed vod mp4 file is being written to stopping the server.js process with a SIGSTOP (to stop the container from immediately quitting when the process dies), running the ffmpeg command while it was stopped rendered the file out successfully, then when server.js was reawakened with SIGCONT it OOM'd. |
|
that could definitely make sense, i should make an option to disable file watching |
|
could you try the dev version with debug enabled and check how the memory rises? i also added a config option to disable the vod watching, maybe that helps |
|
on the develop tag docker image: test 1: disabled file watcher, downloaded a moistcr1tikal vod (previously had none of his vods saved), no crash, server.js memory stayed consistently below 200 megs seems apparent that this issue is dependent on the streamer, and indicates that the file watcher implementation doesn't scale to large vod collections:
now that I can see from your commit the specific function I may be able to take a shot at fixing it when I get the time, but leaving the watcher disabled works for now |
|
does the new log line show the memory grow with each file watch event? i can find some memory issues searching for chokidar so maybe that's just what it does |
|
I haven't seen any information about memory usage in the logs actually, I've been monitoring resource usage through htop. |
|
in the docker console when in debug mode, does it tell you about memory usage when file changes have been detected? |
|
no, I just checked my logs, made sure watcher was enabled an server was in debug mode, tried downloading a vod, but strangely I don't see any lines about memory usage at all |
|
also it seems that the file watcher disabler feature in 17ecfb8 fixes the issue for downloading a vod, but not for finishing a recorded stream |
|
that is quite odd considering it's disabled for both occasions, unless you didn't restart after changing it? maybe it's something other than the file watchers... |
|
disabling the watcher might not have taken the first time because I wasn't aware that that option required a restart to apply, but it has been disabled for about a week and the past 2 streams have still caused a crash - although interestingly these past two streams managed to finish remuxing before the crash, might be because they were shorter and finished remuxing before the server OOM'd, unsure. these past two crashes I wasn't actively watching the docker log output to confirm that the issue was the same OOM exception (the OOM exception isn't written to the docker logs for some reason) but I am pretty sure they were |
|
is this after downloading multiple vods in the same "session"? i am noticing raised memory usage in docker on windows wsl |
|
i tried to do some memory profiling on my windows machine but honestly i just don't have the experience to be able to get any useful information out of it, things that are as low level as this i really don't know where to start |
|
is this still happening? i wonder if it has to do with the amount of log entries or keyvalues in memory |
|
yes it's still happening to me, not on every stream but usually on the longer ones and only when a stream ends. I want to see if I can diagnose and potentially fix the issue myself, but haven't found the time yet |
|
since my last comment, I've taken a couple stabs at sitting down and getting to the root of this issue, which involved setting up an isolated development VM, copying my entire vod database, etc- today I sat down for another attempt, realized that my configuration had broken since subsequent upstream updates, and decided that this isn't worth the effort, for me anyway. the root of the issue is that I had over 100 vods totaling almost 500 gigabytes, which don't all need to be under the application's purview. I'm going to wipe everything and setup the server again from a clean slate, this time configured to use the file export functionality, and I recommend users with similar issues do the same. |
service was down for a stream so I missed a stream and I attempted to download using the video download function, but the server consistently crashes every single time when the remux process starts:
also I tried raising the node heap size with NODE_OPTIONS=--max-old-space-size=8192 and that didn't help.
The text was updated successfully, but these errors were encountered: