nProbe + ntopng not showing UDP DDOS fragments

[melicherm]

Hello dear community,
based on our latest DDOS attack- that we have encountered - we found out, that around 20Gbit of traffic was not visible in ntopng.

It seems that nProbe / ntopng ignores UDP fragments. We think nProbe or ntopng checks for valid flows. Because a DDOS does not need to use valid flow packets -> e.g. the flow does not have a start packet and it's just random data in UDP fragments -> they are invisible using nptobe+ntopng.

Could someone check if my assumption is OK?

Environment:

• OS name: Debian
• OS version: 12 (bookworm)
• Architecture: amd64
• ntopng version/revision: ntopng Enterprise XL v.6.3.241004 rev.24527
• 4Tbit router with sflow exports to nProbe nProbe Enterprise XL 10.7.241003

How did you reproduce it?

UDP packets, random length, random data inside packets send out over a router that has sFlow -> nProbe -> ntopng and look in flows to find nothing.

Thank you community!

[MatteoBiscosi]

Hi @melicherm is it possible to have a pcap / small portion of that attack (in case we will talk by email)?

[melicherm]

@MatteoBiscosi - we have just this dump:

Frame 15: 1490 bytes on wire (11920 bits), 1490 bytes captured (11920 bits) on interface team0.295, id 0
Section number: 1
Interface id: 0 (team0.295)
Interface name: team0.295
Encapsulation type: Ethernet (1)
Arrival Time: Sep 16, 2024 15:30:33.079046360 CEST
UTC Arrival Time: Sep 16, 2024 13:30:33.079046360 UTC
Epoch Arrival Time: 1726493433.079046360
[Time shift for this packet: 0.000000000 seconds]
[Time delta from previous captured frame: 0.000002343 seconds]
[Time delta from previous displayed frame: 0.000002343 seconds]
[Time since reference or first frame: 0.000012911 seconds]
Frame Number: 15
Frame Length: 1490 bytes (11920 bits)
Capture Length: 1490 bytes (11920 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: eth:ethertype:ip:data]
Ethernet II, Src: HuaweiTechno_c3:55:fa (80:e1:bf:c3:55:fa), Dst: 52:54:00:dd:34:65 (52:54:00:dd:34:65)
Destination: 52:54:00:dd:34:65 (52:54:00:dd:34:65)
Address: 52:54:00:dd:34:65 (52:54:00:dd:34:65)
.... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Source: HuaweiTechno_c3:55:fa (80:e1:bf:c3:55:fa)
Address: HuaweiTechno_c3:55:fa (80:e1:bf:c3:55:fa)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: node-5o0.pool-1-0.dynamic.totinternet.net (1.0.156.176), Dst: AAA.BBB.CCC.DDD (AAA.BBB.CCC.DDD)
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
0000 00.. = Differentiated Services Codepoint: Default (0)
.... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
Total Length: 1476
Identification: 0x9371 (37745)
001. .... = Flags: 0x1, More fragments
0... .... = Reserved bit: Not set
.0.. .... = Don't fragment: Not set
..1. .... = More fragments: Set
...0 0000 1011 0110 = Fragment Offset: 1456
Time to Live: 51
Protocol: UDP (17)
Header Checksum: 0x93a7 [validation disabled]
[Header checksum status: Unverified]
Source Address: node-5o0.pool-1-0.dynamic.totinternet.net (1.0.156.176)
Destination Address: AAA.BBB.CCC.DDD (AAA.BBB.CCC.DDD)
[Source GeoIP: Choeng Thale, TH]
[Source GeoIP City: Choeng Thale]
[Source or Destination GeoIP City: Choeng Thale]
[Source GeoIP Country: Thailand]
[Source or Destination GeoIP Country: Thailand]
[Source GeoIP ISO Two Letter Country Code: TH]
[Source or Destination GeoIP ISO Two Letter Country Code: TH]
[Source GeoIP Latitude: 7,9957]
[Source or Destination GeoIP Latitude: 7,9957]
[Source GeoIP Longitude: 98,3101]
[Source or Destination GeoIP Longitude: 98,3101]
[Destination GeoIP: SK]
[Destination GeoIP Country: Slovakia]
[Source or Destination GeoIP Country: Slovakia]
[Destination GeoIP ISO Two Letter Country Code: SK]
[Source or Destination GeoIP ISO Two Letter Country Code: SK]
[Destination GeoIP Latitude: 48,6667]
[Source or Destination GeoIP Latitude: 48,6667]
[Destination GeoIP Longitude: 19,5]
[Source or Destination GeoIP Longitude: 19,5]
Data (1456 bytes)
Data [truncated]: be988d746efde6de5f236f5870dbdb124193f9fa64ad71fb89ec296e88413b4a862d5974633088531b5ca92cd53bcbaee103382d3303b0510cc00c002e00010000344401160006070100005460670faad966e80fc938ab02736c00afdafc36425de03135358907485bb87d8e87f8b
[Length: 1456]

Hope this helps. Will ask for .pcap if it's available.

DST IP is hidden.

[melicherm]

@MatteoBiscosi - got the .pcap (around 1Mil packets available @1.2GB) i have extracted 200 packets. Would like to send it to you per email - 256 KB. Can you give me your address?

Thank you!

[lucaderi]

@melicherm Please send me the URL from which I can download the pcap. My email is deri@ntop.org

[lucaderi]

using this setup

nprobe --zmq tcp://127.0.0.1:1234 -i ~/Downloads/attack_export.pcapng -b 2 --zmq-probe-mode
ntopng -i tcp://127.0.0.1:1234c

and the pcap you sent us, we see the traffic. Can you please check how your sFlow exporter behaves in case of fragmented traffic? Theoretically, it should not matter.

[melicherm]

Hi, if i load the same attack .pcap (1.2G) there is some DNS, ICMP traffic, that i see in nprobe. Maybe that is what you are seeing? But if i use only the UDP fragmented packets extracted from the whole attack i don't see anything.

This is the nprobe+ntop setup:

cat /etc/nprobe/nprobe.conf

--collector-port=6343
--zmq=tcp://*:5556
–disable-cache
-n=none
-T=@NTOPNG@
-G=/var/run/nprobe.pid

cat /etc/ntopng/ntopng.conf | grep -v #
-G=/var/run/ntopng.pid
-i=tcp://127.0.0.1:5556
-w=80
-x=1024000
-X=1024000

I have put only the UDP fragmented packets in an .pcap and used tcpreplay with 2 routers to simulate the same thing that happened in production. The traffic is routed through this router with sFlow activated:

I see around 5Gbit/s of UDP - only fragments - of traffic on the interface. The router sends sflow packets to nProbe

Interface PHY Protocol InUti OutUti inErrors outErrors
100GE0/1/32 up up 0.01% 5.41% 0 0
100GE0/1/34 up up 5.41% 0.01% 0 0

here is the sFlow input to nprobe from the router:
sflow-udp-in.pcap.zip

So traffic from router is definitely coming in the server as sFlow. At an very high rate -> because sampling is 1024 and there is 500K pps going through the router as the attack

tcpdump -i eno8303 port 6343 -nvvv

13:05:10.966214 IP (tos 0x0, ttl 254, id 25696, offset 0, flags [none], proto UDP (17), length 1192)
XXX.XXX.XXX.XXX.40000 > YYY.YYY.YYY.YYY.6343: [no cksum] sFlowv5, IPv4 agent XXX.XXX.XXX.XXX, agent-id 1, seqnum 91233, uptime 1395569211, samples 4, length 1164
expanded flow sample (3), length 276, seqnum 356238, type 0, idx 34, rate 1024, pool 364787712, drops 0, records 6
enterprise 0 Raw packet (1) length 64
protocol Ethernet (1), length 701, stripped bytes 653, header_size 48
enterprise 0 Ethernet frame (2) length 24
frame len 701, type 2048
enterprise 0 IPv4 Data (3) length 32
enterprise 0 Extended Switch data (1001) length 16
src vlan 0, src pri 0, dst vlan 0, dst pri 0
enterprise 0 Extended Router data (1002) length 16
enterprise 0 Extended Gateway data (1003) length 32
expanded flow sample (3), length 276, seqnum 356239, type 0, idx 34, rate 1024, pool 364788736, drops 0, records 6
enterprise 0 Raw packet (1) length 64
protocol Ethernet (1), length 928, stripped bytes 880, header_size 48
enterprise 0 Ethernet frame (2) length 24
frame len 928, type 2048
enterprise 0 IPv4 Data (3) length 32
enterprise 0 Extended Switch data (1001) length 16
src vlan 0, src pri 0, dst vlan 0, dst pri 0
enterprise 0 Extended Router data (1002) length 16
enterprise 0 Extended Gateway data (1003) length 32
expanded flow sample (3), length 276, seqnum 356241, type 0, idx 34, rate 1024, pool 364790784, drops 0, records 6
enterprise 0 Raw packet (1) length 64
protocol Ethernet (1), length 714, stripped bytes 666, header_size 48
enterprise 0 Ethernet frame (2) length 24
frame len 714, type 2048
enterprise 0 IPv4 Data (3) length 32
enterprise 0 Extended Switch data (1001) length 16
src vlan 0, src pri 0, dst vlan 0, dst pri 0
enterprise 0 Extended Router data (1002) length 16
enterprise 0 Extended Gateway data (1003) length 32
expanded flow sample (3), length 276, seqnum 356242, type 0, idx 34, rate 1024, pool 364791808, drops 0, records 6
enterprise 0 Raw packet (1) length 64
protocol Ethernet (1), length 1064, stripped bytes 1016, header_size 48
enterprise 0 Ethernet frame (2) length 24
frame len 1064, type 2048
enterprise 0 IPv4 Data (3) length 32
enterprise 0 Extended Switch data (1001) length 16
src vlan 0, src pri 0, dst vlan 0, dst pri 0
enterprise 0 Extended Router data (1002) length 16
enterprise 0 Extended Gateway data (1003) length 32

If i check the Lo interface - i see only around 4-5 packets every 5 seconds:

lo-traffic.pcap.zip

tcpdump -i lo port 5556 -nvvve
13:11:49.357402 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 887: (tos 0x0, ttl 64, id 61592, offset 0, flags [DF], proto TCP (6), length 873)
127.0.0.1.5556 > 127.0.0.1.36362: Flags [P.], cksum 0x015e (incorrect -> 0xe1eb), seq 3337524860:3337525681, ack 331570497, win 256, options [nop,nop,TS val 2996399584 ecr 2996394584], length 821

23:11:23.030753 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 887: (tos 0x0, ttl 64, id 61592, offset 0, flags [DF], proto TCP (6), length 873)
127.0.0.1.5556 > 127.0.0.1.36362: Flags [P.], cksum 0x015e (incorrect -> 0xe1eb), seq 0:821, ack 1, win 256, options [nop,nop,TS val 2996399584 ecr 2996394584], length 821

13:11:49.357423 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 29060, offset 0, flags [DF], proto TCP (6), length 52)
127.0.0.1.36362 > 127.0.0.1.5556: Flags [.], cksum 0xfe28 (incorrect -> 0x4362), seq 1, ack 821, win 32513, options [nop,nop,TS val 2996399584 ecr 2996399584], length 0

23:11:23.030780 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 29060, offset 0, flags [DF], proto TCP (6), length 52)
127.0.0.1.36362 > 127.0.0.1.5556: Flags [.], cksum 0xfe28 (incorrect -> 0x4362), seq 1, ack 821, win 32513, options [nop,nop,TS val 2996399584 ecr 2996399584], length 0

In this setup ntop does not see any flows:

if i run the nprobe as:
nprobe /etc/nprobe/nprobe.conf

18/Oct/2024 13:53:51 [nprobe.c:12942] nProbe started successfully
18/Oct/2024 13:53:51 [collect.c:3510] Collecting flows from XXX.XXX.XXX.XXX [total: 1/256]
^C18/Oct/2024 13:54:06 [nprobe.c:724] Received shutdown request... [signal: 2]
18/Oct/2024 13:54:06 [nprobe.c:8671] Flushing active flows
18/Oct/2024 13:54:08 [nprobe.c:4380] Processed packets: 21827584 (max bucket search: 0)
18/Oct/2024 13:54:08 [nprobe.c:4361] Fragment queue length: 0
18/Oct/2024 13:54:08 [nprobe.c:4398] UDP collection stats: [collected pkts: 1833][UDP socket drops: 0]
18/Oct/2024 13:54:08 [nprobe.c:4405] Flow collection stats: [processed: 0][dropped (holes in collected flow sequence): 0]
18/Oct/2024 13:54:08 [nprobe.c:4411] Flow export stats: [0 bytes/0 pkts][0 flows/0 pkts sent]
18/Oct/2024 13:54:08 [nprobe.c:4423] Flow export drop stats: [0 bytes/0 pkts][0 flows/0.00 %]
18/Oct/2024 13:54:08 [nprobe.c:4429] Total flow stats: [0 bytes/0 pkts][0 flows/0 pkts sent]

Based on the first nprobe .cfg file i have in /etc/nprobe/nprobe.conf the nprobe sees the flows:

21M of packets, collected 1833 UDP stats, but nothing got exported.

[melicherm]

Based on the sflow data (which seems correct - -> sflow-udp-in.pcap.zip i think nprobe cannot create a flow from the data it has -> e.g. because it's just UDP frament only src IP and dst IP is known... so no ports, etc are knows.

Presumably nprobe should somehow push the data to ntopng as unknown / unknown-fragments type or something, so users can see the data and filter them.

this is nprobe in debug:

startSample ----------------------
sampleType_tag 0:3
sampleType FLOWSAMPLE
sampleSequenceNo 1293739
sourceId 0:34
meanSkipCount 1024
samplePool 1324788736
dropEvents 0
inputPort 34
outputPort 32
flowBlock_tag 0:1
flowSampleType HEADER
headerProtocol 1
sampledPacketSize 1490
strippedBytes 1442
headerLen 48
headerBytes 90-16-BA-4B-03-3F-1C-43-63-79-B6-50-08-00-45-A4-05-C4-D8-7B-20-B6-32-11-45-F9-05-91-A1-1F-C2-A0-DA-09-32-50-BA-87-89-A8-62-9D-3A-7B-01-55-CC-0D
flowBlock_tag 0:2
flowSampleType ETHERNET
ethernet_type 2048
ethernet_len 1490
ethernet_src 1c436379b650
ethernet_dst 9016ba4b033f
flowBlock_tag 0:3
flowSampleType IPV4
sampledPacketSize 1476
IPSize 1476
srcIP 5.145.161.31
dstIP YYY.YYY.YYY.YYY
IPProtocol 17
IPTOS 164
UDPSrcPort 0
UDPDstPort 0
flowBlock_tag 0:1001
extendedType SWITCH
in_vlan 0
in_priority 0
out_vlan 0
out_priority 0
flowBlock_tag 0:1002
extendedType ROUTER
nextHop 10.10.20.31
srcSubnetMask 0
dstSubnetMask 24
flowBlock_tag 0:1003
extendedType GATEWAY
bgp_nexthop 0.0.0.0
my_as 0
src_as 0
src_peer_as 0
dst_as 0
dst_peer_as 0
BGP_localpref 0
endSample ----------------------

but closing the nprobe with ^C:
18/Oct/2024 13:54:08 [nprobe.c:4405] Flow collection stats: [processed: 0][dropped (holes in collected flow sequence): 0]
18/Oct/2024 13:54:08 [nprobe.c:4411] Flow export stats: [0 bytes/0 pkts][0 flows/0 pkts sent]
18/Oct/2024 13:54:08 [nprobe.c:4423] Flow export drop stats: [0 bytes/0 pkts][0 flows/0.00 %]
18/Oct/2024 13:54:08 [nprobe.c:4429] Total flow stats: [0 bytes/0 pkts][0 flows/0 pkts sent]

@lucaderi can you please check if my understanding is correct?