Crowdsec ipset rules do not survive a firewalld reload

[stephdl]

Steps to reproduce

• Install crowdsec
• once installed check if the ipv4/ipv6 sets of ipset are in iptables

[root@R1-pve ~]# ip6tables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
DROP       all      anywhere             anywhere             match-set crowdsec6-blacklists src

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination 

[root@R1-pve ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists src

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination  

reload the firewall like agent.add_public_service after it has added opened services

firewall-cmd --reload

Expected behavior

I expect that my sets are loaded

Actual behavior

The sets are not loaded anymore and obviously crwdsec can ban, the drop is not honored

[root@R1-pve ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination 

[root@R1-pve ~]# ip6tables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination   

however if I restart crowdsec-firewall-bouncer.service the sets are back

systemctl restart crowdsec-firewall-bouncer.service

I tried to make permanent the set to the drop zone of firewalld or with also a rich-rule of the public zone but we face to a strange behavior with two kinds of sets, crowdset wants to use the set that you can see with ipset -L -n and since I want to make permanent the set to firewalld, it looks like there is a conflict

Components

crowdsec 1.0.6

[stephdl]

QA

Install crowdsec ghcr.io/nethserver/crowdsec:1.0.7-dev.3

Now the test is really fun, we run the bouncer inside a container, however the test must be done on debian and rocky9

As a side note we do not remove permanent rules of firewalld nor remove deb and rpm of crowdsec-firewall-bouncer-iptables, we just stop and disable it

test case 1

• direct install add-module ghcr.io/nethserver/crowdsec:1.0.7-dev.3
• Verify the service are up
  systemctl status crowdsec1 crowdsec1-firewall-bouncer
• verify that you can see the nftables sets (possibly you have ips inside, better is yes)

nft list set ip crowdsec crowdsec-blacklists
nft list set ip6 crowdsec6 crowdsec6-blacklists

• remove the whitelist and try to be banned on your LAN by SSH

cscli parsers remove  crowdsecurity/whitelists
systemctl restart crowdsec1

• you can see the BAN by
  cscli decisions list
• you can remove the ban by : cscli decisions delete -i xxx.xxx.xxx.xxx
• verify the nft rules are not removed by
  systemctl restart firewalld && firewall-cmd --reload
• stop the crowdsec1-firewall-bouncer, the rules in nft must be removed

test case 2

• install add-module ghcr.io/nethserver/crowdsec:1.0.6
• now upgrade api-cli run update-module --data '{"module_url":"ghcr.io/nethserver/crowdsec:1.0.7-dev.3","instances":["crowdsec1"],"force":true}'
• Verify the service are up
  systemctl status crowdsec1 crowdsec1-firewall-bouncer
• verify the former service of crowdsec-firewall-bouncer is down
  systemctl status crowdsec-firewall-bouncer
• verify the old set of ipsets are empty, it should but not an issue if they have a timeout (automatically removed once the timeout is ended)
  ipset -L
iptables -L
• verify that you can see the nftables sets (possibly you have ips inside, better is yes)

nft list set ip crowdsec crowdsec-blacklists
nft list set ip6 crowdsec6 crowdsec6-blacklists

• remove the whitelist and try to be banned on your LAN by SSH

cscli parsers remove  crowdsecurity/whitelists
systemctl restart crowdsec1

• you can see the BAN by
  cscli decisions list
• you can remove the ban by : cscli decisions delete -i xxx.xxx.xxx.xxx
• verify the nft rules are not removed by
  systemctl restart firewalld && firewall-cmd --reload
• stop the crowdsec1-firewall-bouncer, the rules in nft must be removed

[mrmarkuz]

Test case 1: OK

Test case 2: NOK

On a single Rocky node it works.
On a Debian cluster node after a firewall-cmd --reload the rules aren't there anymore:

# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

[DavidePrincipi]

QA note

In some Debian installations (i.e. Digital Ocean hosting) the nft command is not installed.

apt install nftables

[DavidePrincipi]

In testing crowdsec 1.0.7-dev.4

[DavidePrincipi]

Test case

Repeat test cases 1 and 2. Notice that

1. new testing version 1.0.7-dev.4

2. the cscli command was moved into the module environment. Invocation becomes for instance:

   runagent -m crowdsec1 cscli ...
   

3. the bouncer service name is now crowdsec1-firewall-bouncer.service (notice the MODULE_ID is used as service name prefix)

   

[nrauso]

on Rocky Linux:

• test case 1: VERIFIED
• test case 2: VERIFIED

On Debian:

• test case 1: VERIFIED
• test case 2: VERIFIED

Note for Debian: used DO VPS so I needed to install nftables tool.

[DavidePrincipi]

VERIFIED

Verified additional fix, and test case https://community.nethserver.org/t/ns8-crowdsec-limited-domain-levels-in-allow-list/23301/7

[DavidePrincipi]

Released in
https://github.com/NethServer/ns8-crowdsec/releases/tag/1.0.7