BPF Masquerading vs IPv4/IPv6 Masquerading

[veith4f]

How to categorize this issue?
/area networking
/kind bug

What happened:
In our gardener landscape, we observed problems with communication between pods and targets outside the cluster. I have never actually seen these problems but inherited a modified/patched version of gardener-extension-networking-cilium that sets:

enableIpv4Masquerade: true
enableIpv6Masquerade: false
enableBPFMasquerade: false

on values.yaml of the internal section of the extension. https://github.com/gardener/gardener-extension-networking-cilium/blob/master/charts/internal/cilium/values.yaml#L148

whereas the default is:

enableIpv4Masquerade: true
enableIpv6Masquerade: false
enableBPFMasquerade: true

As far as I am aware, external communication is working with our configuration whereas I have been told it does not work with the default.

What you expected to happen:
After reading https://docs.cilium.io/en/stable/network/concepts/masquerading/
I understand that

1. bpf masquerading depends on the bpf nodeport feature
2. v4/v6 masquerading and bpf masquerading at the same time are bound to cause interference and should not be used together.

Correct me, if I am wrong but as I understand bpf masquerading is a full replacement for both ipv4 and ipv6 masquerading. Correct configurations are thus exclusively any of the following:

• !v4 && !v6 & !bpf
• v4 && !v6 && !bpf
• !v4 && v6 && !bpf
• !v4 && !v6 && bpf

How to reproduce it (as minimally and precisely as possible):
In a test scenario, a pod would try to communicate via nodeport service with an external location. The expectation is that configurations

• (v4 || v6) && bpf
  cause problems while the previously stated ones work as intended.

Anything else we need to know?:
I understand that the extension is not meant to offer full cilium configuration but rather instead offer configuration "profiles" that ensure sensible settings and avoid misconfiguration. While this can certainly work well and provide ease-of-use, I find the choice to disable bgp control peculiar. Isn't that a major feature of cilium?

Also, performance features like bpf.preallocateMaps are disabled by default. I get that both can be turned on via Shoot configuration. My question is if the provided configuration "profile" has been optimized for performance as this is the main reason to prefer cilium over callico.

Environment:
all gardener/extension versions