Implement the reconciler for SubnetConnectionBindingMap

[wenyingd]

1. Implement the reconciler for SubnetConnectionBindingMap, it may update the SubnetConnectionBindingMap status with condition ready is false if its dependent Subnet or SubnetSet is not ready (or realized) or it hits errors when realizing NSX SubnetConnectionBindingMaps. It updates the status with ready condition as true if it is successfully realized on NSX. The reconciler also watches the Subnet/SubnetSet CR events to sync the connection binding maps.
2. The change also modifies the Subnet/SubnetSet reconciler to watch SubnetConnectionBindingMap CR events. If a Subnet/SubnetSet is used by a SubnetConnectionBindingMap, a finalizer is added on the corresponding Subnet/SubnetSet CR, and the finalizer is removed automatically if the CR is not used by any SubnetConnectionBindingMaps.

Test Done:
A child Subnet CR and a SubnetConnectionBindingMap CR are prepared,

# cat test.yaml 
apiVersion: crd.nsx.vmware.com/v1alpha1
kind: Subnet
metadata:
  name: subnet-child
  namespace: svc-tkg-domain-c10
spec:
  accessMode: Public
  subnetDHCPConfig:
    mode: DHCPDeactivated
  ipAddresses:
  - 192.168.200.0/24
---
apiVersion: crd.nsx.vmware.com/v1alpha1
kind: SubnetConnectionBindingMap
metadata:
  name: binding1
  namespace: svc-tkg-domain-c10
spec:
  subnetName: subnet-child
  targetSubnetSetName: pod-default
  vlanTrafficTag: 201
---

1. When the dependent child Subnet is not ready

# kubectl get subnetconnectionbindingmaps -n svc-tkg-domain-c10 -oyaml
apiVersion: v1
items:
- apiVersion: crd.nsx.vmware.com/v1alpha1
  kind: SubnetConnectionBindingMap
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"crd.nsx.vmware.com/v1alpha1","kind":"SubnetConnectionBindingMap","metadata":{"annotations":{},"name":"binding1","namespace":"svc-tkg-domain-c10"},"spec":{"subnetName":"subnet-child","targetSubnetSetName":"pod-default","vlanTrafficTag":201}}
    creationTimestamp: "2024-11-18T10:18:50Z"
    generation: 1
    name: binding1
    namespace: svc-tkg-domain-c10
    resourceVersion: "5527560"
    uid: 8abcf1c8-956b-45cc-bf18-5b4a33120a7a
  spec:
    subnetName: subnet-child
    targetSubnetSetName: pod-default
    vlanTrafficTag: 201
  status:
    conditions:
    - lastTransitionTime: "2024-11-18T10:19:35Z"
      message: Subnet CR subnet-child is not realized on NSX
      reason: DependencyNotReady
      status: "False"
      type: Ready
kind: List
metadata:
  resourceVersion: ""

After the SubnetConnectionBindingMap is successfully realized, its ready condition is updated as true

# kubectl get subnetbindings -n svc-tkg-domain-c10 -oyaml
apiVersion: v1
items:
- apiVersion: crd.nsx.vmware.com/v1alpha1
  kind: SubnetConnectionBindingMap
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"crd.nsx.vmware.com/v1alpha1","kind":"SubnetConnectionBindingMap","metadata":{"annotations":{},"name":"binding1","namespace":"svc-tkg-domain-c10"},"spec":{"subnetName":"subnet-child","targetSubnetSetName":"pod-default","vlanTrafficTag":201}}
    creationTimestamp: "2024-11-19T03:02:48Z"
    generation: 1
    name: binding1
    namespace: svc-tkg-domain-c10
    resourceVersion: "669501"
    uid: dcfbf90e-f20a-4b9e-b980-41f04eda1f4f
  spec:
    subnetName: subnet-child
    targetSubnetSetName: pod-default
    vlanTrafficTag: 201
  status:
    conditions:
    - lastTransitionTime: "2024-11-19T03:20:16Z"
      status: "True"
      type: Ready
kind: List
metadata:
  resourceVersion: ""

2. The finalizer is added on the dependent subnet and/or subnetset,

# kubectl get subnets -n svc-tkg-domain-c10 -oyaml subnet-child
apiVersion: crd.nsx.vmware.com/v1alpha1
kind: Subnet
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"crd.nsx.vmware.com/v1alpha1","kind":"Subnet","metadata":{"annotations":{},"name":"subnet-child","namespace":"svc-tkg-domain-c10"},"spec":{"accessMode":"Public","ipAddresses":["192.168.200.0/24"],"subnetDHCPConfig":{"mode":"DHCPDeactivated"}}}
  creationTimestamp: "2024-11-19T04:23:28Z"
  finalizers:
  - subnet.nsx.vmware.com/finalizer
  generation: 2
  name: subnet-child
  namespace: svc-tkg-domain-c10
  resourceVersion: "759588"
  uid: 6cf18f78-5363-4e0c-8765-f039b20623e0
spec:
  accessMode: Public
  ipAddresses:
  - 192.168.200.0/24
  ipv4SubnetSize: 16
  subnetDHCPConfig:
    mode: DHCPDeactivated
status:
 ....


# kubectl get subnetsets -n svc-tkg-domain-c10 -oyaml pod-default
apiVersion: crd.nsx.vmware.com/v1alpha1
kind: SubnetSet
metadata:
  creationTimestamp: "2024-11-18T10:53:14Z"
  finalizers:
  - subnetset.nsx.vmware.com/finalizer
  generation: 2
  labels:
    nsxoperator.vmware.com/default-subnetset-for: Pod
  name: pod-default
  namespace: svc-tkg-domain-c10
  resourceVersion: "759589"
  uid: 8e223cfd-fd47-4011-9a14-11debe823d5d
spec:
  accessMode: PrivateTGW
  ipv4SubnetSize: 16
  subnetDHCPConfig: {}
status:
  ...

3. If deleting the dependent subnet (child subnet in the example), it is blocking and adding a condition deletionFailed with reason SubnetInUse

# kubectl delete subnets -n svc-tkg-domain-c10 subnet-child
subnet.crd.nsx.vmware.com "subnet-child" deleted
^C
# kubectl get subnets -n svc-tkg-domain-c10 -oyaml subnet-child
apiVersion: crd.nsx.vmware.com/v1alpha1
kind: Subnet
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"crd.nsx.vmware.com/v1alpha1","kind":"Subnet","metadata":{"annotations":{},"name":"subnet-child","namespace":"svc-tkg-domain-c10"},"spec":{"accessMode":"Public","ipAddresses":["192.168.200.0/24"],"subnetDHCPConfig":{"mode":"DHCPDeactivated"}}}
  creationTimestamp: "2024-11-19T04:23:28Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2024-11-19T05:35:13Z"
  finalizers:
  - subnet.nsx.vmware.com/finalizer
  generation: 3
  name: subnet-child
  namespace: svc-tkg-domain-c10
  resourceVersion: "761558"
  uid: 6cf18f78-5363-4e0c-8765-f039b20623e0
spec:
  accessMode: Public
  ipAddresses:
  - 192.168.200.0/24
  ipv4SubnetSize: 16
  subnetDHCPConfig:
    mode: DHCPDeactivated
status:
  conditions:
  - lastTransitionTime: "2024-11-19T04:23:29Z"
    message: NSX Subnet with DHCPDeactivated has been successfully created/updated
    reason: SubnetReady
    status: "True"
    type: Ready
  - lastTransitionTime: "2024-11-19T05:35:13Z"
    message: Subnet is used by SubnetConnectionBindingMap binding1 and not able to
      delete
    reason: SubnetInUse
    status: "True"
    type: DeletionFailed
  gatewayAddresses:
  - 192.168.200.1/24
  networkAddresses:
  - 192.168.200.0/24

4. After the SubnetConnectionBindingMap is deleted, the Subnet is deleted, and the finalizers are removed automatically.

# kubectl delete subnetbindings binding1 -n svc-tkg-domain-c10
subnetconnectionbindingmap.crd.nsx.vmware.com "binding1" deleted

# kubectl get subnets -n svc-tkg-domain-c10 -oyaml subnet-child
Error from server (NotFound): subnets.crd.nsx.vmware.com "subnet-child" not found

# kubectl get subnetsets -n svc-tkg-domain-c10 -oyaml pod-default
apiVersion: crd.nsx.vmware.com/v1alpha1
kind: SubnetSet
metadata:
  creationTimestamp: "2024-11-18T10:53:14Z"
  generation: 2
  labels:
    nsxoperator.vmware.com/default-subnetset-for: Pod
  name: pod-default
  namespace: svc-tkg-domain-c10
  resourceVersion: "763022"
  uid: 8e223cfd-fd47-4011-9a14-11debe823d5d
spec:
  accessMode: PrivateTGW
  ipv4SubnetSize: 16
  subnetDHCPConfig: {}
status:
  ...

5. SubnetSet scale-out and scale-in.
   After scale-out the target SubnetSet, the number of generated NSX SubnetConnectionBindingMaps is increased from 1 to 2.
   After scale-in the target SubnetSet, the number of generated NSX SubnetConnectionBindingMaps is decreased from 2 to 1,
   No changes was updated to SubnetConnectionBindingMap CR unless errors occurred.

6. If the dependent SubnetSet has no NSX SubnetPort, and all NSX VpcSubnets are gc, the SubnetConnectionBindingMap CR status is changed to not-ready.

# kubectl describe subnetbindings -n svc-tkg-domain-c10 binding1 
Name:         binding1
Namespace:    svc-tkg-domain-c10
Labels:       <none>
Annotations:  <none>
API Version:  crd.nsx.vmware.com/v1alpha1
Kind:         SubnetConnectionBindingMap
Metadata:
  Creation Timestamp:  2024-11-19T06:57:29Z
  Generation:          1
  Resource Version:    866270
  UID:                 54dce36d-871e-4325-85fa-4ab523ad9f3b
Spec:
  Subnet Name:             subnet-child
  Target Subnet Set Name:  pod-default
  Vlan Traffic Tag:        201
Status:
  Conditions:
    Last Transition Time:  2024-11-19T08:11:28Z
    Message:               SubnetSet CR pod-default is not realized on NSX
    Reason:                DependencyNotReady
    Status:                False
    Type:                  Ready
Events:
  Type     Reason            Age                From                                   Message
  ----     ------            ----               ----                                   -------
  Normal   SuccessfulUpdate  38m                subnetconnectionbindingmap-controller  SubnetConnectionBindingMap CR has been successfully updated
  Warning  FailUpdate        2s (x13 over 22s)  subnetconnectionbindingmap-controller  no existing NSX VpcSubnet created by SubnetSet CR 'svc-tkg-domain-c10/pod-default'

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 73.37884% with 312 lines in your changes missing coverage. Please review.

Project coverage is 73.43%. Comparing base (7230790) to head (219db8c).

Files with missing lines	Patch %	Lines
pkg/controllers/subnetset/subnetset_controller.go	35.00%	58 Missing and 7 partials ⚠️
pkg/controllers/subnet/subnet_controller.go	9.67%	51 Missing and 5 partials ⚠️
pkg/controllers/subnetset/subnetbinding_handler.go	45.45%	34 Missing and 2 partials ⚠️
...trollers/subnetbinding/subnetbinding_controller.go	85.23%	34 Missing and 1 partial ⚠️
pkg/controllers/subnet/subnetbinding_handler.go	50.72%	32 Missing and 2 partials ⚠️
pkg/nsx/services/subnetbinding/tree.go	81.65%	21 Missing and 10 partials ⚠️
pkg/nsx/services/subnetbinding/store.go	79.04%	19 Missing and 3 partials ⚠️
pkg/controllers/subnetbinding/subnets_handler.go	80.82%	10 Missing and 4 partials ⚠️
cmd/main.go	0.00%	9 Missing ⚠️
pkg/nsx/services/subnetbinding/subnetbinding.go	94.40%	6 Missing and 2 partials ⚠️
... and 1 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #901      +/-   ##
==========================================
- Coverage   73.50%   73.43%   -0.08%     
==========================================
  Files         108      118      +10     
  Lines       15274    16391    +1117     
==========================================
+ Hits        11227    12036     +809     
- Misses       3297     3570     +273     
- Partials      750      785      +35     

Flag	Coverage Δ
unit-tests	73.43% <73.37%> (-0.08%)	⬇️

Files with missing lines	Coverage Δ
pkg/clean/clean.go	77.58% <100.00%> (+1.00%)	⬆️
pkg/controllers/common/dependency_watcher.go	100.00% <100.00%> (ø)
pkg/nsx/client.go	83.78% <100.00%> (+0.26%)	⬆️
pkg/nsx/services/common/types.go	100.00% <ø> (ø)
pkg/nsx/services/subnetbinding/builder.go	100.00% <100.00%> (ø)
pkg/util/utils.go	85.71% <100.00%> (+0.17%)	⬆️
pkg/nsx/services/subnetbinding/compare.go	87.50% <87.50%> (ø)
pkg/nsx/services/subnetbinding/subnetbinding.go	94.40% <94.40%> (ø)
cmd/main.go	0.00% <0.00%> (ø)
pkg/controllers/subnetbinding/subnets_handler.go	80.82% <80.82%> (ø)
... and 7 more

[luolanzone]

not finished.

[luolanzone]

LGTM overall, leave it to nsx operator owner to approve it.

[yanjunz97]

LGTM

[edwardbadboy]

Looks good. One minor comment.

[lxiaopei]

some minor comments, LGTM.

[wenyingd]

/e2e

[wenyingd]

/e2e

[wenyingd]

/e2e