device/telemetry: enforce resource limits on collector process

[snormore]

We should enforce basic OS-level resource limits (CPU and memory) on the telemetry collector process running on devices. These are hardware switches with constrained environments, and the limits serve as a safeguard against unexpected behavior or bugs. The goal is to ensure the collector can't consume unbounded resources and risk impacting critical system functions.

Options

Set setrlimit with golang.org/x/sys/unix

You can set a soft and hard limit on address space using RLIMIT_AS (or RLIMIT_DATA for data segment size) on Unix-like systems:

import "golang.org/x/sys/unix"

func setMemoryLimit(bytes uint64) error {
    rlim := &unix.Rlimit{Cur: bytes, Max: bytes}
    return unix.Setrlimit(unix.RLIMIT_AS, rlim)
}

This only works if you're on a platform that respects RLIMIT_AS (e.g., Linux). It will cause the process to be killed or fail memory allocations if it tries to exceed this.

Use cgroups

This can be configured externally or if careful it can be done programmatically by writing to cgroup files inside the process (needs permission).

Heap monitoring + SetGCPercent

You can monitor heap usage using runtime.ReadMemStats, and combine that with aggressive GC tuning:

import (
    "runtime"
    "runtime/debug"
)

func enforceSoftLimit(limit uint64) {
    go func() {
        for {
            var m runtime.MemStats
            runtime.ReadMemStats(&m)
            if m.HeapAlloc > limit {
                debug.FreeOSMemory()
            }
            time.Sleep(time.Second)
        }
    }()
}

This does not enforce a limit but can help stay within bounds via GC pressure.