Slow performance when querying custom profiles

[mkoelewijn]

We have upgraded to a new server for NfSen with the following hardware specifications:

CPU: 2x Intel(R) Xeon(R) Gold 5420+ @ 2.0GHz
Memory: 128GB Micron DDR5 SDRAM
Storage: NVMe PE8010 RI U.2

We are running the following software versions:

• NfSen: 1.3.11
• Nfdump: 1.7.4-4f409e9

Our configuration is as follows:

NfSen:

• 6 sources with 1/100 sampling (including 1 source receiving flows from 300+ different devices).
• We are encountering a "Too many exporters (>256)" error during queries. We've modified the setup to prevent these errors from appearing in the NfSen UI.

Nfdump:

• maxworkers set to 64.

When querying the LIVE profile, performance is significantly improved—approximately 5 times faster than our previous NfSen setup. For example:

Summary: total flows: 280850300, total bytes: 84.4 T, total packets: 113.1 G, avg bps: 162.3 M, avg pps: 27190, avg bpp: 746
Time window: 2024-07-05 01:49:16 - 2024-08-22 05:05:12
Total flows processed: 280850300, passed: 280850300, Blocks skipped: 0, Bytes read: 41209733320
Sys: 16.9656s User: 192.8634s Wall: 159.2610s flows/second: 1763459.2 Runtime: 160.1039s

Note: This query does not trigger any "Too many exporters (>256)" errors.

The issue arises when we create a new profile with different channels; the query speed slows down significantly (taking as long as it did on the old server, or even longer). For example, a Stat TopN 10 query:

Summary: total flows: 6113, total bytes: 3.3 G, total packets: 7.0 M, avg bps: 655285, avg pps: 177, avg bpp: 461
Time window: 2024-08-22 01:03:23 - 2024-08-22 12:04:50
Total flows processed: 6113, passed: 6113, Blocks skipped: 0, Bytes read: 126371472
Sys: 0.5604s User: 81.1822s Wall: 81.7372s flows/second: 74.8 Runtime: 81.7373s

Note: This query does trigger the "Too many exporters (>256)" errors.

What could be causing the slow performance when querying profiles other than LIVE?

Could the "Too many exporters (>256)" errors be contributing to this issue?

Do you have any recommendations to improve the speed and fully leverage the hardware capabilities?

[phaag]

That is pretty difficult to tell. I cannot debug your installation. The "Too many exporters (>256)" does not really harm. You may try the latest master which does improve performance over #4f409e9. If possible, try to debug the performance bottleneck.
Furthermore, there are certain things which cannot be processed in parallel - at least not to my knowledge - such as building up the top N hash.
If you could try with the latest master, let me know the results.