farmer local network security

[xmonader]

Ticket to address the protection of the local devices on the farmer's LAN

• Gist from Jan
  • https://gist.github.com/delandtj/b184502ea04f2dbbb71fceb1fe500363

Questions

• is a VM able to reach the farmer's lan devices?
• is it possible that other devices on the farmer's lan to affect the running node?
• is it possible for a node to reach other devices on the farmer's lan?
• how does it handle the farmerbot situation (all of the nodes in the farm send wol packets based on the power events)?

Technical Specs for the ticket

TBD by Jan and Ashraf

[mik-tf]

Any progress on this one? @delandtj

[delandtj]

So in a nutshell (tested on my machine of how it behaves:) you'd want to add these nft rules in the namespace that does NAT for vms.
In case of the 'light' version I assume you'd do it in the host namespace, otherwise in the ndmz namespace.

HERE are the tools and calls necessary to find the default gw mac address, as well the rules to be added to the forwarding chain for each networkd incantation (In errlang, of course)

[mik-tf]

Great. Looks straightforward. @delandtj

@xmonader perhaps you can have a look and see how to implement this solution.

Do you guys propose host namespace or ndmz namespace?

[delandtj]

Do you guys propose host namespace or ndmz namespace?
you'll have to put the rules in the one that is appropriate

[mik-tf]

Alright great. So all we need now is to implement + test it.

[xmonader]

@delandtj How does that work in case of a farmerbot managing the farm?

[delandtj]

we'll have to learn to live with it a bit during development and testing, adding nft rules as we go.
but indeed, talk a bit about what we need.
so :

• local network ports allowed:
  • mycelium IPv6 ND and multicast
  • yggdrasil discovery
  • WoL packets (which are basically allowed, as they are not ip/tcp or ip/udp (or are they))
  • some other thingies ? @LeeSmet ?

[LeeSmet]

For mycelium/ygg, discovery is 1 thing, but we also need to allow connections to the default ports of those services so something can be done with the discovered peers. Other than that I think that about sums it up

[mik-tf]

So as I understand, we are now ready to implement + test this.

@xmonader can you manage to test this with dev team? Maybe Lee and Jan can help.

[muhamadazmy]

You need to be careful also with power management. Make sure that nodes can still send wake on lan packets to wake up their zos neighbours.

[mik-tf]

@delandtj please add info if needed. Then we can proceed and implement + test.

[delandtj]

the rules are all in the forward chain, so anything originating from zos itself, 'should(TM)' 'just work(TM)'
the proof of the pudding is in the eating

[mik-tf]

Good! So let's cook that pudding now.

[mik-tf]

@xmonader as discussed, we can check to make a 3.15 patch. See gitea's 3.15 patch project: https://git.ourworld.tf/tfgrid/-/projects/122

[Omarabdul3ziz]

• tried the suggested solution on zos4 network by getting the default gateway mac address and applying daddr != %s drop in the forward chain. but couldn't make it work, this blocked all access to both the internet and the lan
  tracking the packets on the host while pinging 8.8.8.8 from the ndmz namespace and found that the destination address never matched the router ip resulting in all packets being dropped. am i missing something?

• explored other possible configurations and found we can block a range of ips, so we could block the lan's ip range zos interface runs on
  
  also excluding the nameserver if it is on the range
  

  chain forward {
    type filter hook forward priority 0; policy accept;
    ip daddr != 192.168.123.1 ip daddr 192.168.123.48/24 drop
  }

  applying this rule to the forward chain blocks packets that do not originate from the host so zos itself can still reach other neighbors. and also prevents the namespaces (ndmz/nr) from accessing the lan while allowing internet access
  

• but there are some limitations:

  • namespaces can still communicate with each other, as they operate within the range 100.127.0.0/16, which is not affected by this rule
  • if zos is nated behind another network, this network will not be secure (not sure if this is valid production setup)
    for example, in my setup with qemu nated behind a private network, i can’t access zos neighbors but I can still connect to peers on the network like my phone
    

so are those limitations acceptable?

[delandtj]

I was thinking about that too, as mac address filtering makes more problems than they solve, I noticed... there would be a lot more rules needed, so simplification is key.
Good thinking Omar

[Omarabdul3ziz]

some notes for future reference about what was happening here

• at first i overthought the requirement. thought we needed to protect all private networks behind zos, but then clarified it’s only about protecting the local lan zos runs on. so this is a limitation.

• after deploying to mainnet (zos4), vms lost internet access. found out zos was running on a public IP, so rules blocked all internet traffic. fixed by checking if the default gateway IP is in a private range, if not, skip applying rules.

• faced some inconsistency, rules sometimes worked as expected (internet accessible, lan blocked), other times blocked everything. solved this by explicitly adding both (accept traffic to not lan & drop traffic to lan) rules, ensuring proper order.

• what is excluded:

  • the default gateway (often acts as a nameserver)
  • ygg/myc peer discovery ports.

  imo, no need to exclude something special for WOL. rules only affect the forward chain (traffic not originating/destined for zos). so zos will still be able to communicate with other nodes.

• rules applied in the host namespace for zos4. and in the ndmz namespace for zos3.

Work is down and prs are pending review

[Omarabdul3ziz]

update

• with myc back again, i ran some tests and found that outgoing packets from workload work fine by allowing all traffic except to the LAN. However, incoming packets to the workload are unexpectedly dropped, forcing me to disable the feature for now.
• even we explicitly excluded myc dst/src port but it looks the raw expression @th,0,16 didn't work as expected

  zos/pkg/network/nft/nft.go

  Line 68 in d0d84d8

  buf.WriteString(" meta l4proto { tcp, udp } @th,0,16 { 9651, 9650 } accept;\n")

  
  i did it this way due to a limitation in nft v0.9.1 the current version on zos
• i think better to upgrade nft to 1.1.1 0-initramfs#90 and test again

[xmonader]

moved to 3.17