[Support]: Wifi camera causes "frigate.capture" process to be reaped by OOM-killer and is never restarted #10814
Replies: 3 comments
-
|
I don't think it makes sense to have a watchdog process for the capture process which itself is just running a watchdog for the ffmpeg process. Most likely what is happening here is the ffmpeg process itself is erorring out continuously but refused to end (we can see this as |
Beta Was this translation helpful? Give feedback.
-
|
Just to state I'm still experiencing this on 0.14.0. It's with one of my Wifi cameras which I think can occasionally drop connection or enough packets to confuse ffmpeg. I may try around the source code a bit for anything obvious I can test in a fork. It's a bit of an annoying issue because the huge spike in memory (I had one go to 14GB yesterday) triggers all kinds of alerts on my server monitoring, as then it starts writing to swap etc plus it looks like the server generally starts slowing down on detections |
Beta Was this translation helpful? Give feedback.
-
|
Just to bump this back up. I actually had it occur today on an internal wired camera - I disconnect the network cable to swap it out for a longer one, so it was disconnected for all of about a minute. Shortly after, noticed alerts on my Frigate box because the OOM-killer had killed the 2024-11-13 18:39:50 - ethernet connection dropped I'm assuming this must be something specific to my setup as it doesn't seem to be reported much, if something as simple as a camera going offline could cause it... |
Beta Was this translation helpful? Give feedback.
-
Describe the problem you are having
Occasionally, something seems to happen that causes some of all of my cameras to drop offline for a few seconds.
Mostly they all recover cleanly. However, one particular camera has a habit of not recovering. The
frigate.captureprocess for that camera then looks to very rapidly grow in size to 5GB+, before being killed by the Linux OOM killer. As far as I can tell, Frigate never attempts to restart the capture process, so that camera is then permanently stuck unable to detect.hikcam8 = default stream from camera for record
hikcam8_resized = stream reduced to 1080p @ fps for detect
It starts with an IO timeout streaming RTSP from the camera to go2rtc:
21:11:55 WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error="read tcp 172.31.6.2:48994->192.168.30.98:554: i/o timeout"5 seconds later the resized version of the stream throws an EOF
21:12:00 WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error=EOF url="ffmpeg:hikcam8#video=h264#width=1920#height=1080#raw=-fpsmax 5#hardware=vaapi"3 seconds after that the record process loses the connection.
21:12:03 ffmpeg.indoors_wifi2.record ERROR : rtsp://127.0.0.1:8554/hikcam8: Connection timed out~ 30 seconds later, the frigate.capture process has ballooned in memory usage and been killed.
21:12:32 Memory cgroup out of memory: Killed process 2264357 (frigate.capture) total-vm:5283008kB, anon-rss:3304660kB, file-rss:8976kB, shmem-rss:60kB, UID:0 pgtables:7028kB oom_score_adj:0Interested if a) the capture processes could be monitored and restarted if necessary b) there is any way to protect against a capture process consuming so much memory when the feed has presumably disappeared.
Version
0.13.2-6476F8A
Frigate config file
Relevant log output
FFprobe output from your camera
Frigate stats
No response
Operating system
Debian
Install method
Docker Compose
Coral version
USB
Network connection
Wired
Camera make and model
Hikvision DS-2CD2442FWD
Any other information that may be helpful
No response
Beta Was this translation helpful? Give feedback.
All reactions