UniFi Gateways firmware 4.1.x incompatible with udm-iptv (UXG Lite)

[dmutsaers]

What is your ISP? (if relevant)

No response

Diagnostic Information

=== Configuration ===
WAN Interface: eth1
WAN VLAN: 4 (dev iptv)
WAN DHCP: true (options "-O staticroutes -V IPTV_RG")
WAN Ranges: 10.0.0.0/8 10.252.0.0/16 10.253.0.0/16 217.166.0.0/16
LAN Interfaces: eth0.8
IGMP Proxy quickleave disabled: false
IGMP Proxy debug: false
=== IP Link and Route ===
17: iptv@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 10.252.214.207/16 scope global iptv
valid_lft forever preferred_lft forever
10.0.0.0/8 via 10.252.0.1 metric 217
10.252.0.0/16 proto kernel scope link src 10.252.214.207
=== Service Logs ===
Dec 10 10:01:49 Next-GenGatewayLite udm-iptvd[10937]: group time 208
Dec 10 10:01:49 Next-GenGatewayLite udm-iptvd[10937]: version: 2
Dec 10 10:01:49 Next-GenGatewayLite udm-iptvd[10937]: group 224.2.2.2 exclude mode
Dec 10 10:01:49 Next-GenGatewayLite udm-iptvd[10937]: group time 204
Dec 10 10:01:49 Next-GenGatewayLite udm-iptvd[10937]: version: 2
Dec 10 10:01:49 Next-GenGatewayLite udm-iptvd[10937]: interface eth0.8:192.168.8.1
Dec 10 10:01:49 Next-GenGatewayLite udm-iptvd[10937]: group 224.0.251.124 exclude mode
Dec 10 10:01:49 Next-GenGatewayLite udm-iptvd[10937]: group time 260
Dec 10 10:01:49 Next-GenGatewayLite udm-iptvd[10937]: version: 2
Dec 10 10:01:49 Next-GenGatewayLite udm-iptvd[10937]: group 224.0.251.109 exclude mode

Describe the Bug

If the latest Unifi firmware (4.1.2/4.1.3) has been installed on the UXG Lite/Max/Pro/Enterprise the CPU usage maxes out to 100% when an IPTV stream is active using udm-iptv. IPTV becomes unusable. It seems the ulogd process uses all CPU capacity.
Older firmware doesn't seem to suffer from this issue.

High CPU usage (Unifi forum)

Unifi Gateways firmware 4.1.2 + 4.1.3 incompatible with udm-iptv (UXG Lite/Max/Pro/Enterprise) #358

Expected Behavior

Installing udm-iptv shouldn't cause 100% CPU usage on UXG Lite/Max/Pro/Enterprise.

[dmutsaers]

Unifi's response: "We cannot assure stability on modified OS's, but I understand why you're adjusting the OS."

[dmutsaers]

UniFi Gateways 4.1.8 has been released, but IPTV issues persist.

[spiralshapeturtle]

@dmutsaers which process at the UI device is consuming the CPU? Could you post an "top" screenshot when the issues is present?

[dmutsaers]

I've added two screenshots, the 1st without an active IPTV stream, the 2nd with an active IPTV stream.

[dmutsaers]

I think disabling Intrusion Prevention "fixes" the issue:

[latereaction]

I think disabling Intrusion Prevention "fixes" the issue:

Not for me - I don't have IPS enabled.

[spiralshapeturtle]

@dmutsaers ksoftirqd (packetforwarder if I recall well) is consuming the CPU you mean in the second screenshot? In the UI topic thy talk about ulog as consumer, could you elaborate?

[spiralshapeturtle]

I think disabling Intrusion Prevention "fixes" the issue:

Not for me - I don't have IPS enabled.

Which proces is eating your CPU?

[dmutsaers]

Strange, just by enabling or disabling "Intrusion Prevention" I can trigger or resolve the issue.

[spiralshapeturtle]

Strange, just by enabling or disabling "Intrusion Prevention" I can create or solve the issue.

I dont think that is strange, what I think is the multicast traffic or the inspection of this content is pushed over the CPU instead of hardware offloading. So you just "use" the CPU when you enable the inspection it? I assume in every other speedtest.net test the CPU remains very low due to the hardware offloading?

[dmutsaers]

@dmutsaers ksoftirqd (packetforwarder if I recall well) is consuming the CPU you mean in the second screenshot? In the UI topic thy talk about ulog as consumer, could you elaborate?

I've never checked which process was consuming all CPU. The first time I looked at the processes with TOP was after upgrading to 4.1.8. today (at your request). I noticed the ksoftirq and ubnt-idsips-dae processes we're consuming all CPU on my UXG-Lite.

[dmutsaers]

I dont think that is strange, what I think is the multicast traffic or the inspection of this content is pushed over the CPU instead of hardware offloading. So you just "use" the CPU when you enable the inspection it? I assume in every other speedtest.net test the CPU remains very low due to the hardware offloading?

That may be true, it never happened before in the 4.0.x branch. A speedtest does not stress the CPU.

[latereaction]

@dmutsaers ksoftirqd (packetforwarder if I recall well) is consuming the CPU you mean in the second screenshot? In the UI topic thy talk about ulog as consumer, could you elaborate?

I've never checked which process was consuming all CPU. The first time I looked at the processes with TOP was after upgrading to 4.1.8. today (at your request). I noticed the ksoftirq and ubnt-idsips-dae processes we're consuming all CPU on my UXG-Lite.

I monitored it for quite some time on 4.1.2 and for me it was the ULOGD process that was consuming all CPU.

[spiralshapeturtle]

I dont think that is strange, what I think is the multicast traffic or the inspection of this content is pushed over the CPU instead of hardware offloading. So you just "use" the CPU when you enable the inspection it? I assume in every other speedtest.net test the CPU remains very low due to the hardware offloading?

That may be true, it never happened before in the 4.0.x branch. A speedtest does not stress the CPU.

Hardware offloading is introduced in 4.1, so yes the packetforwarding did change from 4.0 to 4.1

[spiralshapeturtle]

# Disable quickleave for igmpproxy
IPTV_IGMPPROXY_DISABLE_QUICKLEAVE="false"
# Enable debugging for igmpproxy
IPTV_IGMPPROXY_DEBUG="false"
IPTV_IGMPPROXY_PROGRAM="igmpproxy"

Does igmpproxy make a difference, instead of the improxy?

[dmutsaers]

That makes a lot of difference!
(IGMPPROXY isn't downloaded/installed by default, I had to "apt install")

TOP (IGMPPROXY, Intrusion Prevention disabled)

TOP (IGMPPROXY, Intrusion Prevention enabled)

[dmutsaers]

I've been experimenting with IGMPPROXY, IMPROXY and Intrusion Prevention. I have best results with Intrusion Prevention disabled. I could enable Intrusion Prevention with IGMPPROXY, the IPTV stream is stable, but it still stresses the CPU a lot. Enabling Intrusion Prevention with IMPROXY is not possible (without hiccups).

[rogierlommers]

I've been experimenting with IGMPPROXY, IMPROXY and Intrusion Prevention. I have best results with Intrusion Prevention disabled. I could enable Intrusion Prevention with IGMPPROX, the IPTV stream is stable, but it still stresses the CPU a lot. Enabling Intrusion Prevention with IMPROXY is not possible.

Nice learnings! Thanks for posting!

[spiralshapeturtle]

I've been experimenting with IGMPPROXY, IMPROXY and Intrusion Prevention. I have best results with Intrusion Prevention disabled. I could enable Intrusion Prevention with IGMPPROXY, the IPTV stream is stable, but it still stresses the CPU a lot. Enabling Intrusion Prevention with IMPROXY is not possible.

Which unit do you have, this is a UXG-Max 4.1.8 with the fabian script, inspection on, IGMPPROXY (IMPROXY stops after a few seconds), debug logs off, STB vlan IGMP snooping enabled.

I'm not seeing high CPU?

[dmutsaers]

Which unit do you have, this is a UXG-Max 4.1.8 with the fabian script, inspection on, IGMPPROXY (IMPROXY stops after a few seconds), debug logs off, STB vlan IGMP snooping enabled.

I'm not seeing high CPU?

I'm using an UXG-Lite. High CPU usage is only observed with an active multicast IGMP IPTV stream and Intrusion Prevention enabled.

[dmutsaers]

Not for me - I don't have IPS enabled.

Just for fun, could you enable Intrusion Prevention, and after enabling Intrusion Prevention disable Intrusion Prevention again?

[spiralshapeturtle]

Which unit do you have, this is a UXG-Max 4.1.8 with the fabian script, inspection on, IGMPPROXY (IMPROXY stops after a few seconds), debug logs off, STB vlan IGMP snooping enabled.
I'm not seeing high CPU?

I'm using an UXG-Lite. High CPU usage is only observed with an active multicast IGMP IPTV stream and Intrusion Prevention enabled.

Of course my test was with IPTV (KPN) enabled/streaming ;-) Could you change the first post/title that the issue might only occurs to the UXG-LITE hardware? It's a bit misleading now for others who search via google.

[latereaction]

Not for me - I don't have IPS enabled.

Just for fun, could you enable Intrusion Prevention, and after enabling Intrusion Prevention disable Intrusion Prevention again?

Ill include that test when i upgrade tonight.

[spiralshapeturtle]

@dmutsaers you might want to upgrade your hardware, the UXG-LITE got a different chipset then the UXG-Max.

Below UXG-Max

IPS off

IPS enabled

[spiralshapeturtle]

@dmutsaers ksoftirqd (packetforwarder if I recall well) is consuming the CPU you mean in the second screenshot? In the UI topic thy talk about ulog as consumer, could you elaborate?

I've never checked which process was consuming all CPU. The first time I looked at the processes with TOP was after upgrading to 4.1.8. today (at your request). I noticed the ksoftirq and ubnt-idsips-dae processes we're consuming all CPU on my UXG-Lite.

I monitored it for quite some time on 4.1.2 and for me it was the ULOGD process that was consuming all CPU.

Just checking, UXG-LITE hardware?

[dmutsaers]

@dmutsaers you might want to upgrade your hardware, the UXG-LITE got a different chipset then the UXG-Max.

Below UXG-Max

IPS off
IPS enabled

Your results make me wonder there is something really wrong with the UXG-Lite.
A dual-core ARM Cortex-A53 CPU running at 1 GHz and a quad-core ARM Cortex-A53 CPU running at 1.5 GHz shouldn't make such a big difference, should it?

[dmutsaers]

UXG-Lite, IPS off, active IPTV stream, IGMPPROXY

UXG-Lite, IPS on, active IPTV stream, IGMPPROXY

[spiralshapeturtle]

@dmutsaers you might want to upgrade your hardware, the UXG-LITE got a different chipset then the UXG-Max.
Below UXG-Max
IPS off
IPS enabled

Your results make me wonder there is something really wrong with the UXG-Lite. A dual-core ARM Cortex-A53 CPU running at 1 GHz and a quad-core ARM Cortex-A53 CPU running at 1.5 GHz shouldn't make such a big difference, should it?

The forwarding chip is another one which makes it a total different device. This is not related to the CPU capacity. In my screenshots there is no CPU load for forwarding. Somehow the UXG-Max does handle this traffic differently than the UXG-Lite.

More about chipsets overhere.
https://community.ui.com/questions/UniFi-Network-Product-Chipset-Matrix-and-Birth/feb69f12-321d-4102-a5dd-bbab81d08483#answer/6c22a9f9-dbdb-4523-9c5e-564f362af9f5

[latereaction]

OK so I have some interesting/confusing news - I re-upgraded to 4.1.8 last night, reinstalled udm-iptv and switched it in the config to igmpproxy(this is all currently with IPS off) and I am seeing completely normal CPU usage with active IPTV streams. Curiosity then got the better of me and i switched back to improxy, as igmpproxy seems to use a little more RAM. Still seeing completely normal CPU usage. here is my usage graph from UNA for last night when the IPTV would have been on, and overnight to this morning:

I haven't changed anything in my config other than keeping UNA early access up to date, but I don't see how that could have an effect (happy to be corrected).

I ran out of time to try it with IPS enabled, so that will be todays task, however I'm not expecting great things as the UXG Lite struggles really with it anyway.

Edit:
UXG Info just in case:

udm-iptv config:
https://pastebin.com/qXP5a8hi

[dmutsaers]

@latereaction A bit similar as my experiences, it all crumbles down if you enable IPS.

[latereaction]

@latereaction A bit similar as my experiences, it all crumbles down if you enable IPS.

Yeah previously it would have been obvious as the CPU would be pinned at 100% for long periods during IPTV streams.

I think even though Ubiquiti advertise this device as IPS capable - i really think its at its limit. I ended up disabling it pretty sharp after getting the device and seeing the poor performance. I do run Ad blocker on some of my VLANS and also use country blocking, but that's it.

Its a little frustrating that we don't know the root cause of this though, as until Ubiquiti fix their broken (for us) IPTV implementation we are at the behest of the performance of this add-on on all future firmwares.