Slow and steady increase in memory usage for eventd

[DavidZagury]

Description

Steps to reproduce the issue:

1. Run test on a system with no call to config reload / reboot for a long period of time.

Describe the results you received:

The memory usage of the eventd is continuously rising
e.g.
Attached is a summary of the memory usage in one switch from for the eventd proccess from ~13mb on 2024-07-11 to 57mb two months later on 2024-09-16.

Average RAM consumption per process in GB (1).csv

Describe the results you expected:

Memory usage shouldn't continuously rise with time.

Output of show version:

Version running is 202311, based on hash 156b067

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

[bingwang-ms]

1. Test keeps running for 2 months, without any reboot/config reload.
2. The device is not completely idle. There are some tests running.
3. It's not clear if there is any link flap or BGP flap. @DavidZagury Please confirm and add more details if possible.

@qiluo-msft Can you please help take a look and assign to the right owner?

[zbud-msft]

Events are cached when there is no collector listening to xpub socket. Max cache size is 100MB

https://github.com/sonic-net/sonic-buildimage/blob/master/src/sonic-eventd/src/eventd.cpp#L33C1-L33C50

[zbud-msft]

If possible, can you verify that eventd size is reduced when events are collected: https://github.com/sonic-net/sonic-gnmi/tree/master?tab=readme-ov-file#streaming-events