NSMgr v1.12.1-rc.1 causes latency spikes at certain traffic level

[szvincze]

Question

Were there any modification in NSMgr between v1.11.2 and v1.12.1-rc.1 that can cause latency spikes?

Examples

In NSM v1.11.2 and before we managed to run traffic up to 30k TPS without spikes in latency, but with v1.12.1-rc.1 we observed significant latency spikes (up to 5 seconds) that caused disturbances and even disconnection of the SCTP peers.

To exclude the different NSM components we tried to run the same test with the previous release that worked well but added the components from the new RC one by one. When we used the forwarder-vpp from the RC and the other components from 1.11.2 the setup worked as expected, without latency spikes. The new forwarder-vpp and new registry-k8s combination also performed well, but when we added the NSMgr from the RC the latency spikes appeared.
So, it seems the NSMgr component has influence on this behavior.

Important to note that the mentioned latency spikes come periodically, in every two minutes, and this behavior starts when we reach 15k TPS traffic level.

Context

• Kubernetes Version: 1.27.1
• NSM v1.12.1-rc.1

[szvincze]

In the meantime we repeated the tests with increased CPU resource limit for NSMgr pod and the latency spikes disappeared. It proves that something changed in NSMgr that causes higher CPU need periodically.

[glazychev-art]

@szvincze
Thanks for the info!
Could you please attach your logs?

[szvincze]

@glazychev-art, please find the logs attached.

[denis-tingaikin]

cc @VitalyGushin