all controllers - races with vpcRefs, subnetRefs and other references.

[gecube]

Good day!

I am playing with FluxCD GitOps approach and EKS controller.

I tried to create the cluster with the following manifest:

apiVersion: eks.services.k8s.aws/v1alpha1
kind: Cluster
metadata:
  name: production
  namespace: infra-production
spec:
  name: production
  roleARN: arn:aws:iam::966321756598:role/eks-production-role
  logging:
    clusterLogging:
      - enabled: true
        types:
          - api
          - audit
          - authenticator
          - controllerManager
          - scheduler
  resourcesVPCConfig:
    endpointPrivateAccess: true
    endpointPublicAccess: false
    subnetRefs:
      - from:
          name: production-private-eu-west-2a
      - from:
          name: production-private-eu-west-2b
  tags:
    Name: production

The cluster was created successfully, but now I am observing the next state of the object:

apiVersion: eks.services.k8s.aws/v1alpha1
kind: Cluster
metadata:
  resourceVersion: '73958425'
  name: production
  uid: 8290c91e-ae11-4ebb-b068-fef1d21f85a8
  creationTimestamp: '2023-07-25T06:08:08Z'
  generation: 3
  managedFields:
    - apiVersion: eks.services.k8s.aws/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:labels':
            'f:kustomize.toolkit.fluxcd.io/name': {}
            'f:kustomize.toolkit.fluxcd.io/namespace': {}
        'f:spec':
          'f:logging':
            'f:clusterLogging': {}
          'f:name': {}
          'f:resourcesVPCConfig':
            'f:endpointPrivateAccess': {}
            'f:endpointPublicAccess': {}
            'f:subnetRefs': {}
          'f:roleARN': {}
          'f:tags':
            'f:Name': {}
      manager: kustomize-controller
      operation: Apply
      time: '2023-07-25T06:13:09Z'
    - apiVersion: eks.services.k8s.aws/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:finalizers':
            .: {}
            'v:"finalizers.eks.services.k8s.aws/Cluster"': {}
        'f:spec':
          'f:kubernetesNetworkConfig':
            .: {}
            'f:ipFamily': {}
            'f:serviceIPv4CIDR': {}
          'f:resourcesVPCConfig':
            'f:subnetIDs': {}
          'f:version': {}
      manager: controller
      operation: Update
      time: '2023-07-25T06:08:09Z'
    - apiVersion: eks.services.k8s.aws/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          .: {}
          'f:ackResourceMetadata':
            .: {}
            'f:arn': {}
            'f:ownerAccountID': {}
            'f:region': {}
          'f:certificateAuthority': {}
          'f:conditions': {}
          'f:createdAt': {}
          'f:status': {}
      manager: controller
      operation: Update
      subresource: status
      time: '2023-09-14T08:38:40Z'
  namespace: infra-production
  finalizers:
    - finalizers.eks.services.k8s.aws/Cluster
  labels:
    kustomize.toolkit.fluxcd.io/name: infra-management
    kustomize.toolkit.fluxcd.io/namespace: flux-system
spec:
  kubernetesNetworkConfig:
    ipFamily: ipv4
    serviceIPv4CIDR: 172.20.0.0/16
  logging:
    clusterLogging:
      - enabled: true
        types:
          - api
          - audit
          - authenticator
          - controllerManager
          - scheduler
  name: production
  resourcesVPCConfig:
    endpointPrivateAccess: true
    endpointPublicAccess: false
    subnetIDs:
      - subnet-0f06902b47c880118
      - subnet-0c72af713be937dcc
    subnetRefs:
      - from:
          name: production-private-eu-west-2a
      - from:
          name: production-private-eu-west-2b
  roleARN: 'arn:aws:iam::966321756598:role/eks-production-role'
  tags:
    Name: production
  version: '1.27'
status:
  ackResourceMetadata:
    arn: 'arn:aws:eks:eu-west-2:966321756598:cluster/production'
    ownerAccountID: '966321756598'
    region: eu-west-2
  certificateAuthority: {}
  conditions:
    - lastTransitionTime: '2023-09-14T08:38:40Z'
      message: Reference resolution failed
      reason: >-
        both resource reference wrapper and ID cannot be used together:
        ResourcesVPCConfig.SubnetIDs,ResourcesVPCConfig.SubnetRefs
      status: Unknown
      type: ACK.ReferencesResolved
  createdAt: '2023-07-25T06:08:09Z'
  status: CREATING

So my idea is that EKS controller (or EC2 - it does not matter at this moment) takes

    subnetRefs:
      - from:
          name: production-private-eu-west-2a
      - from:
          name: production-private-eu-west-2b

block from YAML, removes it and add this one:

    subnetIDs:
      - subnet-0f06902b47c880118
      - subnet-0c72af713be937dcc

resolving which subnet by name has which ID.

But on the next reconciliation cycle FluxCD sees that the fields were removed and adds them back.

So finally we are getting in the next status:

    - lastTransitionTime: '2023-09-14T08:38:40Z'
      message: Reference resolution failed
      reason: >-
        both resource reference wrapper and ID cannot be used together:
        ResourcesVPCConfig.SubnetIDs,ResourcesVPCConfig.SubnetRefs
      status: Unknown
      type: ACK.ReferencesResolved

It looks like that no controller from ACK library must change the source spec of object, but rather make all such a changes in status field and maintain some internal state of the controller - no idea. What is interesting - the same issue is not observer with VPC, Subnets and other objects. No idea why.

Also once I observed the next yaml:

I wondered too much how I could get it. But possibly it could be linked to incorrect paste of IDs instead of references as well.

[ack-bot]

Issues go stale after 180d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 60d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/aws-controllers-k8s/community.
/lifecycle stale

[gecube]

/remove-lifecycle stale

[a-hilaly]

@gecube we have a PR attempting to fix this issue aws-controllers-k8s/eks-controller#128

[a-hilaly]

@gecube This is now fixed, in 1.5.0~

[a-hilaly]

Please reopen the Github issue if you still see this :)