Merge branch eni-cleanup into master

* Call DisassociateTrunkInterface before deleting branch ENI (#372) * Call DisassociateTrunkInterface before deleting branch ENI * feat: Centralize leaked ENI cleanup (#374) * feat: centralized eni cleanup * Merge master into eni-cleanup (#385) * fix: paginate DescribeNetworkInterfaces with deep filters (#375) * fix: paginate DescribeNetworkInterfaces with deep filters * update metrics and address review comments * minor updates to address comments * Bump github.com/aws/aws-sdk-go from 1.49.13 to 1.50.29 (#380) Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.49.13 to 1.50.29. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Commits](aws/aws-sdk-go@v1.49.13...v1.50.29) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump k8s.io/client-go from 0.29.1 to 0.29.2 (#377) Bumps [k8s.io/client-go](https://github.com/kubernetes/client-go) from 0.29.1 to 0.29.2. - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](kubernetes/client-go@v0.29.1...v0.29.2) --- updated-dependencies: - dependency-name: k8s.io/client-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/prometheus/common from 0.46.0 to 0.49.0 (#378) Bumps [github.com/prometheus/common](https://github.com/prometheus/common) from 0.46.0 to 0.49.0. - [Release notes](https://github.com/prometheus/common/releases) - [Commits](prometheus/common@v0.46.0...v0.49.0) --- updated-dependencies: - dependency-name: github.com/prometheus/common dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Repo controlled build go version (#381) * update golang version (#383) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jason Du <jasonxdu@amazon.com> * fix:update cluster tag name in CNINode (#386) * fix:add node OS label in CNINode, retry get CNINode with backoff * update protobuf to 1.33.0 (#387) * add CNINode integration tests (#391) * use DescribeNetworkInterfaces with deep filters * add integration test to validate ec2 permissions * remove DisassociateAllBranchENIs as it is not useful (#400) * remove DisassociateAllBranchENIs as it is not useful * skip deletion success log for NotFound ENI * fix govulncheck * Merge master branch into eni-cleanup (#416) * fix: paginate DescribeNetworkInterfaces with deep filters (#375) * fix: paginate DescribeNetworkInterfaces with deep filters * update metrics and address review comments * minor updates to address comments * Bump github.com/aws/aws-sdk-go from 1.49.13 to 1.50.29 (#380) Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.49.13 to 1.50.29. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Commits](aws/aws-sdk-go@v1.49.13...v1.50.29) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump k8s.io/client-go from 0.29.1 to 0.29.2 (#377) Bumps [k8s.io/client-go](https://github.com/kubernetes/client-go) from 0.29.1 to 0.29.2. - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](kubernetes/client-go@v0.29.1...v0.29.2) --- updated-dependencies: - dependency-name: k8s.io/client-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/prometheus/common from 0.46.0 to 0.49.0 (#378) Bumps [github.com/prometheus/common](https://github.com/prometheus/common) from 0.46.0 to 0.49.0. - [Release notes](https://github.com/prometheus/common/releases) - [Commits](prometheus/common@v0.46.0...v0.49.0) --- updated-dependencies: - dependency-name: github.com/prometheus/common dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Repo controlled build go version (#381) * update golang version (#383) * update protobuf to 1.33.0 (#387) * pin envtest version due to an upstream bug (#390) * Bump k8s.io/client-go from 0.29.2 to 0.29.3 (#392) Bumps [k8s.io/client-go](https://github.com/kubernetes/client-go) from 0.29.2 to 0.29.3. - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](kubernetes/client-go@v0.29.2...v0.29.3) --- updated-dependencies: - dependency-name: k8s.io/client-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/aws/amazon-vpc-cni-k8s from 1.16.0 to 1.17.1 (#393) Bumps [github.com/aws/amazon-vpc-cni-k8s](https://github.com/aws/amazon-vpc-cni-k8s) from 1.16.0 to 1.17.1. - [Release notes](https://github.com/aws/amazon-vpc-cni-k8s/releases) - [Changelog](https://github.com/aws/amazon-vpc-cni-k8s/blob/master/CHANGELOG.md) - [Commits](aws/amazon-vpc-cni-k8s@v1.16.0...v1.17.1) --- updated-dependencies: - dependency-name: github.com/aws/amazon-vpc-cni-k8s dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/prometheus/common from 0.49.0 to 0.51.1 (#395) Bumps [github.com/prometheus/common](https://github.com/prometheus/common) from 0.49.0 to 0.51.1. - [Release notes](https://github.com/prometheus/common/releases) - [Commits](prometheus/common@v0.49.0...v0.51.1) --- updated-dependencies: - dependency-name: github.com/prometheus/common dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/aws/aws-sdk-go from 1.50.29 to 1.51.12 (#397) Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.50.29 to 1.51.12. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Commits](aws/aws-sdk-go@v1.50.29...v1.51.12) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * add github action to run gosec static analysis (#398) * add github action to run gosec static analysis * install gosec * update golang and dependency to fix CVE (#401) * revert pagination and call DescribeNetworkInterfaces with vpcID or subnetID filter * Revert "fix: paginate DescribeNetworkInterfaces with deep filters (#375)" This reverts commit b5699de. * call DescribeNetworkInterfaces with vpcID or subnetID filter * update EC2 supported instance types (#402) * remove global exclusion for G108,G114 and add nosec in code (#404) * Update controller_auth_proxy_patch.yaml (#405) Update the reference from gcr.io to registry.k8s.io > kube-rbac-proxy is moving to registry.k8s.io/kubebuilder/kube-rbac-proxy (from gcr.io/kubebuilder/kube-rbac-proxy) because GCR is being sunset. We need to update these references. * Fix log which causes panic (#407) * Fix log which causes panic * Consistent key name * consistent naming * run go mod tidy --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jason Du <jasonxdu@amazon.com> Co-authored-by: Hao Zhou <haouc@users.noreply.github.com> Co-authored-by: Senthil Kumaran <senthilx@amazon.com> Co-authored-by: Garvin Pang <garvinpang@protonmail.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jason Du <jasonxdu@amazon.com> Co-authored-by: Hao Zhou <haouc@users.noreply.github.com> Co-authored-by: Senthil Kumaran <senthilx@amazon.com> Co-authored-by: Garvin Pang <garvinpang@protonmail.com>
aws · May 1, 2024 · b5a054e · b5a054e
1 parent c80fd41
commit b5a054e
Show file tree

Hide file tree

Showing 49 changed files with 2,097 additions and 610 deletions.
diff --git a/PROJECT b/PROJECT
@@ -19,6 +19,7 @@ resources:
   version: v1beta1
 - api:
     crdVersion: v1
+  controller: true
   domain: k8s.aws
   group: vpcresources
   kind: CNINode

diff --git a/apis/vpcresources/v1alpha1/cninode_types.go b/apis/vpcresources/v1alpha1/cninode_types.go
@@ -35,6 +35,8 @@ type Feature struct {
 // CNINodeSpec defines the desired state of CNINode
 type CNINodeSpec struct {
 	Features []Feature `json:"features,omitempty"`
+	// Additional tag key/value added to all network interfaces provisioned by the vpc-resource-controller and VPC-CNI
+	Tags map[string]string `json:"tags,omitempty"`
 }
 
 // CNINodeStatus defines the managed VPC resources.

diff --git a/apis/vpcresources/v1alpha1/zz_generated.deepcopy.go b/apis/vpcresources/v1alpha1/zz_generated.deepcopy.go
diff --git a/config/crd/bases/vpcresources.k8s.aws_cninodes.yaml b/config/crd/bases/vpcresources.k8s.aws_cninodes.yaml
@@ -56,6 +56,12 @@ spec:
                       type: string
                   type: object
                 type: array
+              tags:
+                additionalProperties:
+                  type: string
+                description: Additional tag key/value added to all network interfaces
+                  provisioned by the vpc-resource-controller and VPC-CNI
+                type: object
             type: object
           status:
             description: CNINodeStatus defines the managed VPC resources.

diff --git a/config/rbac/role.yaml b/config/rbac/role.yaml
@@ -61,6 +61,8 @@ rules:
   - create
   - get
   - list
+  - patch
+  - update
   - watch
 - apiGroups:
   - vpcresources.k8s.aws

diff --git a/controllers/core/node_controller.go b/controllers/core/node_controller.go
@@ -20,6 +20,7 @@ import (
 
 	"github.com/aws/amazon-vpc-resource-controller-k8s/apis/vpcresources/v1alpha1"
 	"github.com/aws/amazon-vpc-resource-controller-k8s/pkg/condition"
+	"github.com/aws/amazon-vpc-resource-controller-k8s/pkg/config"
 	rcHealthz "github.com/aws/amazon-vpc-resource-controller-k8s/pkg/healthz"
 	"github.com/aws/amazon-vpc-resource-controller-k8s/pkg/k8s"
 	"github.com/aws/amazon-vpc-resource-controller-k8s/pkg/node/manager"
@@ -36,15 +37,6 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/healthz"
 )
 
-// MaxNodeConcurrentReconciles is the number of go routines that can invoke
-// Reconcile in parallel. Since Node Reconciler, performs local operation
-// on cache only a single go routine should be sufficient. Using more than
-// one routines to help high rate churn and larger nodes groups restarting
-// when the controller has to be restarted for various reasons.
-const (
-	MaxNodeConcurrentReconciles = 10
-)
-
 // NodeReconciler reconciles a Node object
 type NodeReconciler struct {
 	client.Client
@@ -117,7 +109,7 @@ func (r *NodeReconciler) SetupWithManager(mgr ctrl.Manager, healthzHandler *rcHe
 
 	return ctrl.NewControllerManagedBy(mgr).
 		For(&corev1.Node{}).
-		WithOptions(controller.Options{MaxConcurrentReconciles: MaxNodeConcurrentReconciles}).
+		WithOptions(controller.Options{MaxConcurrentReconciles: config.MaxNodeConcurrentReconciles}).
 		Owns(&v1alpha1.CNINode{}).
 		Complete(r)
 }

diff --git a/controllers/crds/cninode_controller.go b/controllers/crds/cninode_controller.go
@@ -0,0 +1,252 @@
+// Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License"). You may
+// not use this file except in compliance with the License. A copy of the
+// License is located at
+//
+//     http://aws.amazon.com/apache2.0/
+//
+// or in the "license" file accompanying this file. This file is distributed
+// on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+// express or implied. See the License for the specific language governing
+// permissions and limitations under the License.
+
+package crds
+
+import (
+	"context"
+	"time"
+
+	"github.com/aws/amazon-vpc-resource-controller-k8s/apis/vpcresources/v1alpha1"
+	ec2API "github.com/aws/amazon-vpc-resource-controller-k8s/pkg/aws/ec2/api"
+	"github.com/aws/amazon-vpc-resource-controller-k8s/pkg/aws/ec2/api/cleanup"
+	"github.com/aws/amazon-vpc-resource-controller-k8s/pkg/config"
+	"github.com/aws/amazon-vpc-resource-controller-k8s/pkg/k8s"
+	"github.com/aws/amazon-vpc-resource-controller-k8s/pkg/utils"
+	"github.com/go-logr/logr"
+	"github.com/prometheus/client_golang/prometheus"
+	v1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/apimachinery/pkg/util/wait"
+	"k8s.io/client-go/util/retry"
+	ctrl "sigs.k8s.io/controller-runtime"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/controller"
+	"sigs.k8s.io/controller-runtime/pkg/metrics"
+)
+
+var (
+	prometheusRegistered     = false
+	recreateCNINodeCallCount = prometheus.NewCounter(
+		prometheus.CounterOpts{
+			Name: "recreate_cniNode_call_count",
+			Help: "The number of requests made by controller to recreate CNINode when node exists",
+		},
+	)
+	recreateCNINodeErrCount = prometheus.NewCounter(
+		prometheus.CounterOpts{
+			Name: "recreate_cniNode_err_count",
+			Help: "The number of requests that failed when controller tried to recreate the CNINode",
+		},
+	)
+)
+
+func prometheusRegister() {
+	prometheusRegistered = true
+
+	metrics.Registry.MustRegister(
+		recreateCNINodeCallCount,
+		recreateCNINodeErrCount)
+
+	prometheusRegistered = true
+}
+
+// CNINodeReconciler reconciles a CNINode object
+type CNINodeReconciler struct {
+	client.Client
+	Scheme           *runtime.Scheme
+	Context          context.Context
+	Log              logr.Logger
+	EC2Wrapper       ec2API.EC2Wrapper
+	K8sAPI           k8s.K8sWrapper
+	ClusterName      string
+	VPCID            string
+	FinalizerManager k8s.FinalizerManager
+}
+
+//+kubebuilder:rbac:groups=vpcresources.k8s.aws,resources=cninodes,verbs=get;list;watch;create;update;patch;
+
+// Reconcile handles CNINode create/update/delete events
+// Reconciler will add the finalizer and cluster name tag if it does not exist and finalize on CNINode on deletion to clean up leaked resource on node
+func (r *CNINodeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
+	cniNode := &v1alpha1.CNINode{}
+	if err := r.Client.Get(ctx, req.NamespacedName, cniNode); err != nil {
+		if errors.IsNotFound(err) {
+			r.Log.Info("CNINode is deleted", "CNINode", req.NamespacedName)
+		}
+		// Ignore not found error
+		return ctrl.Result{}, client.IgnoreNotFound(err)
+	}
+
+	nodeFound := true
+	node := &v1.Node{}
+	if err := r.Client.Get(ctx, req.NamespacedName, node); err != nil {
+		if errors.IsNotFound(err) {
+			nodeFound = false
+		} else {
+			r.Log.Error(err, "failed to get the node object in CNINode reconciliation, will retry")
+			// Requeue request so it can be retried
+			return ctrl.Result{}, err
+		}
+	}
+
+	if cniNode.GetDeletionTimestamp().IsZero() {
+		shouldPatch := false
+		cniNodeCopy := cniNode.DeepCopy()
+		// Add cluster name tag if it does not exist
+		val, ok := cniNode.Spec.Tags[config.CNINodeClusterNameKey]
+		if !ok || val != r.ClusterName {
+			if len(cniNodeCopy.Spec.Tags) != 0 {
+				cniNodeCopy.Spec.Tags[config.CNINodeClusterNameKey] = r.ClusterName
+			} else {
+				cniNodeCopy.Spec.Tags = map[string]string{
+					config.CNINodeClusterNameKey: r.ClusterName,
+				}
+			}
+			shouldPatch = true
+		}
+		// if node exists, get & add OS label if it does not exist on CNINode
+		if nodeFound {
+			nodeLabelOS := node.ObjectMeta.Labels[config.NodeLabelOS]
+			val, ok = cniNode.ObjectMeta.Labels[config.NodeLabelOS]
+			if !ok || val != nodeLabelOS {
+				if len(cniNodeCopy.ObjectMeta.Labels) != 0 {
+					cniNodeCopy.ObjectMeta.Labels[config.NodeLabelOS] = nodeLabelOS
+				} else {
+					cniNodeCopy.ObjectMeta.Labels = map[string]string{
+						config.NodeLabelOS: nodeLabelOS,
+					}
+				}
+				shouldPatch = true
+			}
+		}
+
+		if shouldPatch {
+			r.Log.Info("patching CNINode to add required fields Tags and Labels", "cninode", cniNode.Name)
+			return ctrl.Result{}, r.Client.Patch(ctx, cniNodeCopy, client.MergeFromWithOptions(cniNode, client.MergeFromWithOptimisticLock{}))
+		}
+
+		// Add finalizer if it does not exist
+		if err := r.FinalizerManager.AddFinalizers(ctx, cniNode, config.NodeTerminationFinalizer); err != nil {
+			r.Log.Error(err, "failed to add finalizer on CNINode, will retry", "cniNode", cniNode.Name, "finalizer", config.NodeTerminationFinalizer)
+			return ctrl.Result{}, err
+		}
+		return ctrl.Result{}, nil
+
+	} else { // CNINode is marked for deletion
+		if !nodeFound {
+			//  node is also deleted, proceed with running the cleanup routine and remove the finalizer
+
+			// run cleanup for Linux nodes only
+			if val, ok := cniNode.ObjectMeta.Labels[config.NodeLabelOS]; ok && val == config.OSLinux {
+				r.Log.Info("running the finalizer routine on cniNode", "cniNode", cniNode.Name)
+				cleaner := &cleanup.NodeTerminationCleaner{
+					NodeName: cniNode.Name,
+				}
+				cleaner.ENICleaner = &cleanup.ENICleaner{
+					EC2Wrapper: r.EC2Wrapper,
+					Manager:    cleaner,
+					VPCID:      r.VPCID,
+					Log:        ctrl.Log.WithName("eniCleaner").WithName("node"),
+				}
+
+				if err := cleaner.DeleteLeakedResources(); err != nil {
+					r.Log.Error(err, "failed to cleanup resources during node termination, request will be requeued")
+					// Return err if failed to delete leaked ENIs on node so it can be retried
+					return ctrl.Result{}, err
+				}
+			}
+
+			if err := r.FinalizerManager.RemoveFinalizers(ctx, cniNode, config.NodeTerminationFinalizer); err != nil {
+				r.Log.Error(err, "failed to remove finalizer on CNINode, will retry", "cniNode", cniNode.Name, "finalizer", config.NodeTerminationFinalizer)
+				return ctrl.Result{}, err
+			}
+			return ctrl.Result{}, nil
+		} else {
+			// node exists, do not run the cleanup routine(periodic cleanup routine will anyway delete leaked ENIs), remove the finalizer
+			// to proceed with object deletion, and recreate similar object
+
+			// Create a copy without deletion timestamp for creation
+			newCNINode := &v1alpha1.CNINode{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:            cniNode.Name,
+					Namespace:       "",
+					OwnerReferences: cniNode.OwnerReferences,
+					// TODO: should we include finalizers at object creation or let controller patch it on Create/Update event?
+					Finalizers: cniNode.Finalizers,
+				},
+				Spec: cniNode.Spec,
+			}
+
+			if err := r.FinalizerManager.RemoveFinalizers(ctx, cniNode, config.NodeTerminationFinalizer); err != nil {
+				r.Log.Error(err, "failed to remove finalizer on CNINode on node deletion, will retry")
+				return ctrl.Result{}, err
+			}
+			// wait till CNINode is deleted before recreation as the new object will be created with same name to avoid "object already exists" error
+			if err := r.waitTillCNINodeDeleted(k8s.NamespacedName(newCNINode)); err != nil {
+				// raise event if CNINode could not be deleted after removing the finalizer
+				r.K8sAPI.BroadcastEvent(cniNode, utils.CNINodeDeleteFailed, "CNINode deletion failed and object could not be recreated by the vpc-resource-controller, will retry",
+					v1.EventTypeWarning)
+				// requeue here to check if CNINode deletion is successful and retry CNINode deletion if node exists
+				return ctrl.Result{}, err
+			}
+
+			r.Log.Info("creating CNINode after it has been deleted as node still exists", "cniNode", newCNINode.Name)
+			recreateCNINodeCallCount.Inc()
+			if err := r.createCNINodeFromObj(ctx, newCNINode); err != nil {
+				recreateCNINodeErrCount.Inc()
+				// raise event on node publish warning that CNINode is deleted and could not be recreated by controller
+				utils.SendNodeEventWithNodeName(r.K8sAPI, node.Name, utils.CNINodeCreateFailed,
+					"CNINode was deleted and failed to be recreated by the vpc-resource-controller", v1.EventTypeWarning, r.Log)
+				// return nil as deleted and we cannot recreate the object now
+				return ctrl.Result{}, nil
+			}
+			r.Log.Info("successfully recreated CNINode", "cniNode", newCNINode.Name)
+		}
+	}
+	return ctrl.Result{}, nil
+}
+
+// SetupWithManager sets up the controller with the Manager.
+func (r *CNINodeReconciler) SetupWithManager(mgr ctrl.Manager) error {
+	if !prometheusRegistered {
+		prometheusRegister()
+	}
+	return ctrl.NewControllerManagedBy(mgr).
+		For(&v1alpha1.CNINode{}).
+		WithOptions(controller.Options{MaxConcurrentReconciles: config.MaxNodeConcurrentReconciles}).
+		Complete(r)
+}
+
+// waitTillCNINodeDeleted waits for CNINode to be deleted with timeout and returns error
+func (r *CNINodeReconciler) waitTillCNINodeDeleted(nameSpacedCNINode types.NamespacedName) error {
+	oldCNINode := &v1alpha1.CNINode{}
+
+	return wait.PollUntilContextTimeout(context.TODO(), 500*time.Millisecond, time.Second*3, true, func(ctx context.Context) (bool, error) {
+		if err := r.Client.Get(ctx, nameSpacedCNINode, oldCNINode); err != nil && errors.IsNotFound(err) {
+			return true, nil
+		}
+		return false, nil
+	})
+}
+
+// createCNINodeFromObj will create CNINode with backoff and returns error if CNINode is not recreated
+func (r *CNINodeReconciler) createCNINodeFromObj(ctx context.Context, newCNINode client.Object) error {
+	return retry.OnError(retry.DefaultBackoff, func(error) bool { return true },
+		func() error {
+			return r.Client.Create(ctx, newCNINode)
+		})
+}
-Original file line number
+Diff line change
@@ Expand Up / @@ -61,6 +61,8 @@ rules: @@
       - create
       - get
       - list
+      - patch
+      - update
       - watch
     - apiGroups:
       - vpcresources.k8s.aws
@@ Expand Down @@