Neolink Memory Leak / Errors

[KSti56]

Describe the bug
This software is really amazing, but unfortunately I've been having a lot of issues with it, specifically with memory leaks. Memory usage is at 3.3G after just an hour. I have read a couple of the issues pertaining to memory leaks (like this one) and it seems like the main recommendation was to update. I updated a couple weeks ago (currently on b37b3ee release). Unfortunately, memory leaks continue and the logs are full of errors. I'm no expert at this stuff, so any help would be greatly appreciated.

Additionally, I checked today and saw a bunch of these errors:

gst_poll_read_control: assertion 'set != NULL' failed
INFO  neolink::rtsp::stream] Buffer full on audsrc
INFO  neolink::rtsp::stream] Buffer full on vidsrc

To Reproduce
No specific steps to reproduce. I believe it has been happening since installation.

Expected behavior
No memory leaks (I would expect maybe 200-300MB of usage?).

Versions
NVR software: BlueIris v5.9.3.4
Neolink software: b37b3ee
Reolink camera model and firmware: 3 B800s -- Firmware on 2 of them is v3.0.0.82_20080600 and the other one is v3.0.0.183_21012800

[QuantumEntangledAndy]

Well aware of this issue, (multiple open issues too) I've used valgrind, massif, and all sorts of other programs to track it. Can't seem to squash this one

Also Buffer full on audsrc means that your client (VLC or whatever) is not pulling the frames and the gstreamer buffer is full, this is mostly harmless as it just means we stop sending frames to gstraemer for a while.

The program kind of works like this

• Neolink
  • Gets frames from camera
  • Buffers for reorganising packets (maximum of about 200 frames)
  • Buffers for paused playback (15s of frames)
  • Hand frames to gstreamer
• Gstreamer
  • Gets frames from neolink
  • Has its own buffer to hold the frames
  • Waits for right time/request from rtsp client to deliver frames

Most of the apps memory allocations happen in the gstreamer part which I don't have direct control over. I'll keep looking when I can but nothing seems to be obvious

[KSti56]

Thanks for the reply! I really appreciate your work on this project. Sorry about the duplicate issue, I didn't see any other ones relating to this exact issue (but now that I went through the older issues, I see what you are referring to). I'm glad to hear that you're aware of the issue and trying to fix it, I know it's probably frustrating that you can't find the root cause. I'll keep monitoring issues with hopes of an eventual fix.

As for the Buffer full on audsrc, I'm using BlueIris. Is there a certain configuration parameter that I can adjust to fix this? I have noticed that the video feed (and therefore recordings) cut out somewhat often, which I assume is related to this.

And as for the memory leak issue, I assume the only temporary fixes are just either allocating more memory and/or setting up automatic restarts every x hours?

Thanks again for the help and work on this project! Let me know if there is any information that would be helpful for the debugging/troubleshooting process.

[RutgerDiehard]

I've seen memory leaks when running with 2 x RLC-CX810. If it helps troubleshooting, I've reverted back to image: quantumentangledandy/neolink:v0.5.17 which has been running consistently at about 120MB for several days now. So it would suggest changes after this version has caused the leak. I didn't test every image between but this was the first one that worked consistently.

[Pytonballoon810]

For those using neolink in a docker configuration here is a docker compose work around for the memory leak:

...
  neolink:
    container_name: neolink
    restart: always
    deploy:
      resources:
        limits:
          memory: 512M
    image: quantumentangledandy/neolink
...

also i came up with this quick bash script for restarting the container when the process itself takes up too much memory:

#!/bin/bash

# The name of the Docker container
CONTAINER_NAME="neolink"

# Define the memory threshold (in percentage)
THRESHOLD=20

# Get the total system memory in kilobytes
TOTAL_MEM=$(grep MemTotal /proc/meminfo | awk '{print $2}')

# Function to calculate the total memory usage of processes with the word "neolink"
get_neolink_mem_usage() {
    ps aux | grep -i "neolink" | grep -v "grep" | awk '{mem+=$6} END {print mem}'
}

# Get the current memory usage of neolink processes
NEOLINK_MEM=$(get_neolink_mem_usage)

# Convert the memory usage to percentage
MEM_PERCENT=$(echo "$NEOLINK_MEM $TOTAL_MEM" | awk '{printf "%.2f", ($1/$2)*100}')

# Restart the Docker container if memory usage exceeds the threshold
if (( $(echo "$MEM_PERCENT > $THRESHOLD" | bc -l) )); then
    echo "Memory usage of neolink processes is $MEM_PERCENT%, which exceeds the threshold of $THRESHOLD%"
    echo "Restarting the Docker container $CONTAINER_NAME..."
    docker restart $CONTAINER_NAME
else
    echo "Memory usage of neolink processes is $MEM_PERCENT%, which is within the safe limit."
fi

You could probably use a chron job to run this script every minute and that should also work.

[KSti56]

I've seen memory leaks when running with 2 x RLC-CX810. If it helps troubleshooting, I've reverted back to image: quantumentangledandy/neolink:v0.5.17 which has been running consistently at about 120MB for several days now. So it would suggest changes after this version has caused the leak. I didn't test every image between but this was the first one that worked consistently.

I switched to this Docker version, and while it did seem to be running pretty smoothly, it eventually crashed after 7-8 hours (I think it was a memory leak issue, but I wasn't monitoring it). Interestingly, I also received these errors right when the recording stopped. I haven't seen these messages before, so if anyone has any ideas of what they mean, I would appreciate the help.

neolink_core::bc_protocol::connection::bcconn] Reaching limit of channel
neolink_core::bc_protocol::connection::bcconn] Remaining: 0 of 100 message space for 4 (ID: 3)
neolink::rtsp] cam19: Join Pause
neolink::rtsp] cam19: Retryable error: Timed out waiting to send Media Frame (Caused by: deadline has elapsed)

[RutgerDiehard]

I've seen memory leaks when running with 2 x RLC-CX810. If it helps troubleshooting, I've reverted back to image: quantumentangledandy/neolink:v0.5.17 which has been running consistently at about 120MB for several days now. So it would suggest changes after this version has caused the leak. I didn't test every image between but this was the first one that worked consistently.

I switched to this Docker version, and while it did seem to be running pretty smoothly, it eventually crashed after 7-8 hours (I think it was a memory leak issue, but I wasn't monitoring it). Interestingly, I also received these errors right when the recording stopped. I haven't seen these messages before, so if anyone has any ideas of what they mean, I would appreciate the help.

neolink_core::bc_protocol::connection::bcconn] Reaching limit of channel
neolink_core::bc_protocol::connection::bcconn] Remaining: 0 of 100 message space for 4 (ID: 3)
neolink::rtsp] cam19: Join Pause
neolink::rtsp] cam19: Retryable error: Timed out waiting to send Media Frame (Caused by: deadline has elapsed)

Although quantumentangledandy/neolink:0.5.17 ran with no memory issues and I didn't have to babysit it, I wasn't able to run Frigate with clean logs. I would get regular stream errors and FFMPEG crashes from multiple cameras. So, I set about testing new versions of neolink to see if it would solve the Frigate errors. quantumentangledandy/neolink:0.5.18 spammed the container logs with messages to do with encryption, so I tried quantumentangledandy/neolink:0.6.0. This, for the last 12 hours, has been perfectly stable, Frigate shows no stream errors and no FFMPEG crashes, and it runs with the resource shown below. This is with 4 x 4K cameras (2 x Reolink RLC-811a, 2 x RLC-CX810) and a RLC-410 via a Reolink NVR.

I have limited the resources available to neolink and Frigate in the Portainer stack (which runs on bare-metal Ubuntu Server 24.04) just so they don't impact other containers I use on the system. So far, I'm very happy how it's running.

[jmoney7823956789378]

Testing with 0.5.17, 0.6.0, 0.6.3rc2, using 15x D800 Reolink cameras:

Hosted on a generic ubuntu22.02 LXC on proxmox
Forced to use a watchdog process to terminate and restart the service at 75% total memory consumption