Deduplication of NetFlow data

[2tefan]

Hi again!

As far as I know, ntopng currently can't deduplicate NetFlow data like in this example here:

flowchart LR
    nfd[NetFlow device] -->|NetFlow| np1
    nfd[NetFlow device] -->|NetFlow| np2
    np1[nprobe #1]
    np2[nprobe #2]
    nt[ntopng
    ❌ no dedup.]
    np1 -->|ZMQ| nt
    np2 -->|ZMQ| nt

Loading

What would you like to add or change?:
Could you please add support for deduplication of ntopng side?

Why do you and others need this?:
We would like to deploy multiple nprobes at the same time for redundancy proposes. So by default, ntopng receives the same NetFlow data from multiple nprobes. And if one nprobe fails, the other nprobes can still send NetFlow traffic.

Because as far as I know nprobe supports deduplication. This would allow us to setup an architecture like this:

flowchart LR
    nfd1[NetFlow device #1] 
    nfd1 -->|NetFlow| np1
    nfd1 -->|NetFlow| np2
    nfd2[NetFlow device #2] 
    nfd2 -->|NetFlow| np1
    nfd2 -->|NetFlow| np2
    nfd3[NetFlow device #3] 
    nfd3 -->|NetFlow| np1
    nfd3 -->|NetFlow| np2
    np1[nprobe #1
    ✅ dedup.]
    np2[nprobe #2
    ✅ dedup.]
    nt[ntopng
    ✅ dedup.]
    np1 -->|ZMQ| nt
    np2 -->|ZMQ| nt

Loading

without having duplicate NetFlow data.

Thank you!

[lucaderi]

As of today nprobe supports flow deduplication, but not ntopng. We will consider you request, but for the time being please use nprobe deduplication.