[workspacekit] Use veth instead of slirp4netns

[csweichel]

Is your feature request related to a problem? Please describe

Today we use slirp4netns to connect the network namespace of our workspaces. While convenient and easy to use, this comes at a runtime cost because we make networking a userland issue.

Describe the behaviour you'd like

Instead, we should defer to the ws-dameon IWS to create veth pairs between ring1 and the workspace pod.

Also, if there's an opportunity to improve the logging, that would be great.

Describe alternatives you've considered

Maybe IPVLAN or MACVLAN work nicer because we don't have to setup routing in the workspace pod - but the interaction with CNI is unclear.

[Furisto]

Did some research into how a veth pair can be provided. The assumption is that you have some shell running with

unshare -m -n -U -r --propagation unchanged bash
echo $$

The propagation unchanged part is import as unshare by default will mount the rootfs as private in the new mount namespace, so mount events in the host mount namespace will not propagate to the new mount namespace. This means that after moving one end of the veth pair in the net namespace it will show "peer reference not found" when listing the network interfaces with ip a. The reason for this is that attaching to the net namespace is done by creating a shared bind-mount in /run/netns and if this is not visible from the mount namespace of the child the reference cannot be resolved.

I recommend saving your iptables rules with iptables-save or using a virtual machine to avoid messing up your system.

# Create veth pair
ip link add veth-<instanceId> type veth peer name ceth-<instanceId>

# Turn anonymous network namespace into named namespace and 
# attach one end of veth pair to network namespace 
ip netns attach netns-<instanceId> <pid>
ip link set ceth-<instanceId> netns netns-<instanceId>

# Assign IP addresses to both ends of veth pair
ip addr add 10.0.3.1/24 dev veth-<instanceId>
ip netns exec netns-<instanceId> ip addr add 10.0.3.2/24 dev ceth0-<instanceId>

# Bring up network interfaces
ip link set veth-<instanceId> up
ip netns exec netns-<instanceId> ip link set ceth-0<instanceId> up
ip netns exec netns-<instanceId> ip link set lo up

# Set default gateway
ip netns exec netns-<instanceId> ip route add default via 10.0.3.1

# Enable forwarding
echo 1 > /proc/sys/net/ip4/ip_forward 

# Add iptables rules
iptables -A FORWARD -o eth0 -i veth-<instanceId> -j ACCEPT
iptables -A FORWARD -i eth0 -o veth-<instanceId> -j ACCEPT
iptables -t nat -A POSTROUTING -s 10.0.3.2/24 -o eth0 -j MASQUERADE

# Setup DNS
mkdir -p /etc/netns/netns-<instanceId>
ln -s /etc/resolv.conf /etc/netns/netns-<instanceId>/resolv.conf

[utam0k]

I feel that perhaps @Furisto's method would work with IWS and nsinsider. (I will make a sequence diagram at a later date for more details)

@csweichel @Furisto
Apart from this, there is bypass4nets available from kernel 5.9, which is worth considering. This is a technique that uses seccomp notify to rewrite the fd of the network communication to the host. What do you think?

[csweichel]

Bypass4netns was a hot contender. We can't just drop it in and use it though because we already use seccomp-notify. We'd need to integrate it with workspacekit.

It does solve a different problem though. We do have a privileged process we can externalise operations to, unlike what bypass4netns was built for.

[utam0k]

I had not come up with a good way to go about this either. I was thinking that if there is a way to do it, it would be the best option.

Bypass4netns was a hot contender. We can't just drop it in and use it though because we already use seccomp-notify. We'd need to integrate it with workspacekit.

[utam0k]

Manually created a pair of veth and successfully communicated with the outside world.

##### IN HOST ######

# Enable forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward

# enter into the target pod pid
nsenter -t 212137 -n -m -p bash

##### IN POD'S NAMESPACE ######

# Set the values
INSTANCE_ID=w2
WORKSPACE_PID=57

# Create veth pair
ip link add veth-$INSTANCE_ID type veth peer name ceth-$INSTANCE_ID

# Turn anonymous network namespace into named namespace and
# attach one end of veth pair to network namespace
# ip netns attach netns-<instanceId> <pid>
mkdir -p /var/run/netns
ln -s /proc/$WORKSPACE_PID/ns/net /var/run/netns/netns-$INSTANCE_ID
ip link set ceth-$INSTANCE_ID netns netns-$INSTANCE_ID

# Assign IP addresses to both ends of veth pair
ip addr add 10.0.5.1/24 dev veth-$INSTANCE_ID
ip netns exec netns-$INSTANCE_ID ip addr add 10.0.5.2/24 dev ceth-$INSTANCE_ID

# Bring up network interfaces
ip link set veth-$INSTANCE_ID up
ip netns exec netns-$INSTANCE_ID ip link set ceth-$INSTANCE_ID up
ip netns exec netns-$INSTANCE_ID ip link set lo up

# Add iptables rules
iptables -A FORWARD -o eth0 -i veth-$INSTANCE_ID -j ACCEPT
iptables -A FORWARD -i eth0 -o veth-$INSTANCE_ID -j ACCEPT
iptables -t nat -A POSTROUTING -s 10.0.5.0/24 -o eth0 -j MASQUERADE

# Setup DNS
mkdir -p /etc/netns/netns-$INSTANCE_ID
ln -s /etc/resolv.conf /etc/netns/netns-$INSTANCE_ID/resolv.conf

# Set default gateway
ip netns exec netns-$INSTANCE_ID ip route replace default via 10.0.5.1

### in workspace's namespace ###
root@gfe3de62b9ab20671cf9d9f:/# traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  10.0.5.1 (10.0.5.1)  0.036 ms  0.005 ms  0.004 ms
 2  10-132-0-67.kubernetes.default.svc.cluster.local (10.132.0.67)  0.091 ms  0.063 ms  0.040 ms
 3  * * *
...
12  * * *
13  dns.google (8.8.8.8)  1.023 ms  0.786 ms  0.338 ms

However, when I changed the default gateway, the workspace kept reconnecting. I am not sure if this is because I manually changed the settings of a workspace that has finished starting, or if the original default gateway (10.0.0.2) has a special meaning.
@csweichel @Furisto Do you know anything about this?