Connecting a remote interface without creating a VLAN on top

[tedlean]

In order to facilitate a wider range of virtual environments hosting the Kubernetes cluster it could be feasible to allow the injection an interface into the NSC without creating a vlan on top. Some environments uses vlan in the underlay to provide interfaces for the worker-nodes. Here it might not be tolerated that an additional vlan is added on top ending up with a "Q-in-Q" setup.
Therefore it should be possible to have a plain Ethernet interface injected without adding any additional vlan tagging.

[ljkiraly]

Instead of injection, we agreed offline to go for a cluster break-out option for not adding the extra VLAN tag just connecting the given interface with the virtual interface (which belongs to local mechanism) created in NSC pod.

The general naming convention is used by vpp for vlan sub-interfaces so the base interface name can not follow that naming convention. The base interface name is set in NSM by device-selector file to map to 'via' label.
So for example the base interface for cluster breakout 'eth2.100' must be renamed to 'eth2_vlan100' (or something like that).

[tedlean]

Will this naming convention have impact on the user-provided configuration.
Currently we have this in the Attractor(/FE) CR:

               name:
                  description: name of the interface
                  type: string
                nsm-vlan:
                  description: if the type is "nsm-vlan", this information must
                    be specified
                  properties:
                    base-interface:
                      description: (immutable) master interface of the vlan interface
                        to be used for external connectivity
                      type: string
                    vlan-id:
                      description: (immutable) vlan ID of the vlan interface to
                        be used for external connectivity
                      format: int32
                      type: integer
                  type: object

The "name" is meant to be the name of interface exposed inside the Pod hosting the nVIP FE.
If a "nsm-vlan" type is to be used for breakout the base-interface and and vlan-id have to be provided.

For an ignorant user it would be natural set the name to "eth1.220" if the base-interface is "eth1" and the vlan-id is "220".

But if I understand you correctly this is not possible/allowed?
If this is correct we for sure need to make a strong note about this in the user-guide and maybe even enforce a check in the nVIP operator to prevent this.

One thing I do not understand is why this is an issue. The pod has it own netns, where I would guess interfaces can be renamed to anything we like.

[ljkiraly]

@tedlean I reopen this issue - supposing closed by fault.

The "name" is meant to be the name of interface exposed inside the Pod.
For an ignorant user it would be natural set the name to "eth1.220" if the base-interface is "eth1" and the vlan-id is "220".

But if I understand you correctly this is not possible/allowed?

No, the base-interface can not be eth1.220, the name of interface exposed inside the Pod can be anything.

[tedlean]

Okay, so it actually is the naming of the interface on the VM worker-node, which cannot be "eth1.220".
I do not really know how wide-spread a problem this will be. Normally I do not think that the interface provided into to a VM will have any "vlan extension" even though the L2 connectivity is provided by a VLAN in the underlay network.
Still this naming convention probably should be mentioned somewhere. Is it only the vlan setup that has this issue?

[ljkiraly]

Okay, so it actually is the naming of the interface on the VM worker-node, which cannot be "eth1.220".

Confirmed.

Is it only the vlan setup that has this issue?

Not yet checked but I think any other setup is affected.
When I created a host interface inside vpp using an AF_PACKET interface created by default naming scheme (e.g eth1.220) the cross-connection does not worked. Based on vpp documentation this AF_PACKET is expected to be one side of a veth pair. Maybe vpp handle the "host-eth1.220" as a sub-interface of a host interface, not as a host interface. I have to double check in documentation, but maybe @edwarnicke can correct me.

[edwarnicke]

@tedlean What is the nature of the issue you see when the interface is named eth1.220 ?

@ljkiraly Please note, AF_PACKET can be used to bind to any interface, not just one end of a veth pair.

[ljkiraly]

@edwarnicke

Please note, AF_PACKET can be used to bind to any interface, not just one end of a veth pair.

Thanks, that's fine.
I will try to detail the problem with interface name also since it is an issue on technical level.

What is the nature of the issue you see when the interface is named eth1.220 ?

Currently the interface created in NSC (by local mechanism type:kernel) is cross-connected with the sub-interface in vpp (created by remote-mechanism type:vlan) - also vlan-tag-rewrite was set on the sub-interface. In the change made by commit in case of vlan_id=0 the sub-interface is not created and no vtr is set, the host interface is directly connected to the tap/veth interface of NSC.

1. The host interface not getting nor forwarding packets as expected if

• AF_PACKET interface name is eth1.220
• the host interface in vpp will be host-eth1.220

2. The host interface getting and transmitting packets as expected if

• AF_PACKET interface name is eth1_220
• the host interface in vpp will be host-eth1_220

Meantime I figured out that the host interface can be renamed when it is created, but this will cause problem in the code where the interfaces are dumped and filtered by name. If you do not have better suggestion I will handle this in NSM code and eliminate this deviation renaming host-eth1.220 to host-eth1_220 by SwInterfaceSetInterfaceName

As I see now, with this change we move from a specific (vlan) remote break-out to a generic cluster breakout.

[edwarnicke]

@ljkiraly Is the interface name the issue, or is it some other characteristic of the kernel interface?

[ljkiraly]

@edwarnicke Seems that somehow the MTU setting failed (api call HwInterfaceSetMtu) for this renamed interface and the ACL creation was skipped. I double checked and I can confirm that the defined ACLs disables the traffic.
Also observed problems related to MTU settings - I heard from Zoltan, that he already reported that.

What would be the impact of skipping the ACL creation for the interfaces created for cluster breakout?