EC2 controller does not seem to remediate instance

[Katnopic]

Describe the bug

We started a POC of EC2 Controller for internal use of managing EC2's from our EKS Clusters.

We have deployed the EC2 controller through helm (latest version 1.0.1), and gave the service account of the controller relevant permissions to manage EC2 (currently we gave ec2:* permissions on all resources), so far so good.

We have created an 'Instance' k8s object and we can see that it was created in our EC2 Console.

However, we expected that when we do manual changes on the created EC2 (for example, shutting it down or changing the security group), the controller would try to remediate the instance, but seems like this doesn't happen.

We also tried to terminate the instance manually to see what happens (we expected the controller to spin up a new instance to replace the terminated one), but again, controller reports it detects no changes

We changed the defaultResyncPeriod from the default 10h to 90 seconds so the controller does sync with the remote state more often to check this, but as you can see in the logs below, that doesn't fix our issue.

Steps to reproduce

We deployed ec2-controller through helm with the following values:

aws:
  region: {{ aws_region }}
installScope: namespace
# controller reconciliation configurations
reconcile:
  defaultResyncPeriod: 90
serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # The name of the service account to use.
  name: ack-ec2-controller
  annotations:
    # TODO: get this from eks terraform outputs
    eks.amazonaws.com/role-arn: {{ role_arn }} 
# log level for the controller
log:
  enable_development_logging: true
  level: debug

startup logs:

2023-02-27T12:24:32.363Z	INFO	setup	initializing service controller	{"aws.service": "ec2"}
2023-02-27T12:24:33.274Z	DEBUG	ackrt	Initiating reconciler	{"reconciler kind": "RouteTable", "resync period seconds": 90}
2023-02-27T12:24:33.274Z	DEBUG	ackrt	Initiating reconciler	{"reconciler kind": "TransitGateway", "resync period seconds": 90}
2023-02-27T12:24:33.274Z	DEBUG	ackrt	Initiating reconciler	{"reconciler kind": "ElasticIPAddress", "resync period seconds": 90}
2023-02-27T12:24:33.274Z	DEBUG	ackrt	Initiating reconciler	{"reconciler kind": "NATGateway", "resync period seconds": 90}
2023-02-27T12:24:33.274Z	DEBUG	ackrt	Initiating reconciler	{"reconciler kind": "VPC", "resync period seconds": 90}
2023-02-27T12:24:33.274Z	DEBUG	ackrt	Initiating reconciler	{"reconciler kind": "InternetGateway", "resync period seconds": 90}
2023-02-27T12:24:33.274Z	DEBUG	ackrt	Initiating reconciler	{"reconciler kind": "SecurityGroup", "resync period seconds": 90}
2023-02-27T12:24:33.274Z	DEBUG	ackrt	Initiating reconciler	{"reconciler kind": "DHCPOptions", "resync period seconds": 90}
2023-02-27T12:24:33.274Z	DEBUG	ackrt	Initiating reconciler	{"reconciler kind": "Subnet", "resync period seconds": 90}
2023-02-27T12:24:33.275Z	DEBUG	ackrt	Initiating reconciler	{"reconciler kind": "VPCEndpoint", "resync period seconds": 90}
2023-02-27T12:24:33.275Z	DEBUG	ackrt	Initiating reconciler	{"reconciler kind": "Instance", "resync period seconds": 90}
2023-02-27T12:24:33.275Z	INFO	setup	starting manager	{"aws.service": "ec2"}
2023-02-27T12:24:33.275Z	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
2023-02-27T12:24:33.275Z	INFO	controller.adoptedresource	Starting EventSource	{"reconciler group": "services.k8s.aws", "reconciler kind": "AdoptedResource", "source": "kind source: *v1alpha1.AdoptedResource"}
2023-02-27T12:24:33.275Z	INFO	controller.fieldexport	Starting EventSource	{"reconciler group": "services.k8s.aws", "reconciler kind": "FieldExport", "source": "kind source: *v1alpha1.FieldExport"}
2023-02-27T12:24:33.275Z	INFO	controller.adoptedresource	Starting Controller	{"reconciler group": "services.k8s.aws", "reconciler kind": "AdoptedResource"}
2023-02-27T12:24:33.275Z	INFO	controller.fieldexport	Starting Controller	{"reconciler group": "services.k8s.aws", "reconciler kind": "FieldExport"}
2023-02-27T12:24:33.275Z	INFO	controller.routetable	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "RouteTable", "source": "kind source: *v1alpha1.RouteTable"}
2023-02-27T12:24:33.275Z	INFO	controller.routetable	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "RouteTable"}
2023-02-27T12:24:33.275Z	INFO	controller.routetable	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "RouteTable", "source": "kind source: *v1alpha1.RouteTable"}
2023-02-27T12:24:33.275Z	INFO	controller.routetable	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "RouteTable"}
2023-02-27T12:24:33.275Z	INFO	controller.transitgateway	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "TransitGateway", "source": "kind source: *v1alpha1.TransitGateway"}
2023-02-27T12:24:33.275Z	INFO	controller.elasticipaddress	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "ElasticIPAddress", "source": "kind source: *v1alpha1.ElasticIPAddress"}
2023-02-27T12:24:33.275Z	INFO	controller.elasticipaddress	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "ElasticIPAddress"}
2023-02-27T12:24:33.275Z	INFO	controller.natgateway	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "NATGateway", "source": "kind source: *v1alpha1.NATGateway"}
2023-02-27T12:24:33.275Z	INFO	controller.natgateway	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "NATGateway"}
2023-02-27T12:24:33.275Z	INFO	controller.transitgateway	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "TransitGateway"}
2023-02-27T12:24:33.275Z	INFO	controller.vpc	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "VPC", "source": "kind source: *v1alpha1.VPC"}
2023-02-27T12:24:33.275Z	INFO	controller.securitygroup	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "SecurityGroup", "source": "kind source: *v1alpha1.SecurityGroup"}
2023-02-27T12:24:33.275Z	INFO	controller.dhcpoptions	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "DHCPOptions", "source": "kind source: *v1alpha1.DHCPOptions"}
2023-02-27T12:24:33.275Z	INFO	controller.dhcpoptions	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "DHCPOptions"}
2023-02-27T12:24:33.275Z	INFO	controller.transitgateway	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "TransitGateway", "source": "kind source: *v1alpha1.TransitGateway"}
2023-02-27T12:24:33.276Z	INFO	controller.transitgateway	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "TransitGateway"}
2023-02-27T12:24:33.275Z	INFO	controller.vpc	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "VPC"}
2023-02-27T12:24:33.275Z	INFO	controller.vpc	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "VPC", "source": "kind source: *v1alpha1.VPC"}
2023-02-27T12:24:33.275Z	INFO	controller.natgateway	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "NATGateway", "source": "kind source: *v1alpha1.NATGateway"}
2023-02-27T12:24:33.276Z	INFO	controller.natgateway	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "NATGateway"}
2023-02-27T12:24:33.276Z	INFO	controller.subnet	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "Subnet", "source": "kind source: *v1alpha1.Subnet"}
2023-02-27T12:24:33.276Z	INFO	controller.subnet	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "Subnet"}
2023-02-27T12:24:33.275Z	INFO	controller.internetgateway	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "InternetGateway", "source": "kind source: *v1alpha1.InternetGateway"}
2023-02-27T12:24:33.276Z	INFO	controller.internetgateway	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "InternetGateway"}
2023-02-27T12:24:33.276Z	INFO	controller.vpc	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "VPC"}
2023-02-27T12:24:33.275Z	INFO	controller.elasticipaddress	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "ElasticIPAddress", "source": "kind source: *v1alpha1.ElasticIPAddress"}
2023-02-27T12:24:33.276Z	INFO	controller.instance	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "Instance", "source": "kind source: *v1alpha1.Instance"}
2023-02-27T12:24:33.276Z	INFO	controller.instance	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "Instance"}
2023-02-27T12:24:33.276Z	INFO	controller.elasticipaddress	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "ElasticIPAddress"}
2023-02-27T12:24:33.275Z	INFO	controller.securitygroup	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "SecurityGroup", "source": "kind source: *v1alpha1.SecurityGroup"}
2023-02-27T12:24:33.276Z	INFO	controller.securitygroup	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "SecurityGroup"}
2023-02-27T12:24:33.276Z	INFO	controller.vpcendpoint	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "VPCEndpoint", "source": "kind source: *v1alpha1.VPCEndpoint"}
2023-02-27T12:24:33.276Z	INFO	controller.vpcendpoint	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "VPCEndpoint"}
2023-02-27T12:24:33.276Z	INFO	controller.vpcendpoint	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "VPCEndpoint", "source": "kind source: *v1alpha1.VPCEndpoint"}
2023-02-27T12:24:33.276Z	INFO	controller.vpcendpoint	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "VPCEndpoint"}
2023-02-27T12:24:33.275Z	INFO	controller.internetgateway	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "InternetGateway", "source": "kind source: *v1alpha1.InternetGateway"}
2023-02-27T12:24:33.277Z	INFO	controller.internetgateway	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "InternetGateway"}
2023-02-27T12:24:33.277Z	INFO	controller.instance	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "Instance", "source": "kind source: *v1alpha1.Instance"}
2023-02-27T12:24:33.277Z	INFO	controller.instance	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "Instance"}
2023-02-27T12:24:33.277Z	INFO	controller.securitygroup	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "SecurityGroup"}
2023-02-27T12:24:33.277Z	INFO	controller.subnet	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "Subnet", "source": "kind source: *v1alpha1.Subnet"}
2023-02-27T12:24:33.364Z	INFO	controller.subnet	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "Subnet"}
2023-02-27T12:24:33.464Z	INFO	controller.dhcpoptions	Starting EventSource	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "DHCPOptions", "source": "kind source: *v1alpha1.DHCPOptions"}
2023-02-27T12:24:33.464Z	INFO	controller.dhcpoptions	Starting Controller	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "DHCPOptions"}
2023-02-27T12:24:33.475Z	INFO	controller.adoptedresource	Starting workers	{"reconciler group": "services.k8s.aws", "reconciler kind": "AdoptedResource", "worker count": 1}
2023-02-27T12:24:33.476Z	INFO	controller.natgateway	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "NATGateway", "worker count": 1}
2023-02-27T12:24:33.477Z	INFO	controller.vpc	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "VPC", "worker count": 1}
2023-02-27T12:24:33.477Z	INFO	controller.securitygroup	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "SecurityGroup", "worker count": 1}
2023-02-27T12:24:33.477Z	INFO	controller.vpc	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "VPC", "worker count": 1}
2023-02-27T12:24:33.477Z	INFO	controller.securitygroup	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "SecurityGroup", "worker count": 1}
2023-02-27T12:24:33.477Z	INFO	controller.internetgateway	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "InternetGateway", "worker count": 1}
2023-02-27T12:24:33.565Z	INFO	controller.elasticipaddress	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "ElasticIPAddress", "worker count": 1}
2023-02-27T12:24:33.565Z	INFO	controller.transitgateway	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "TransitGateway", "worker count": 1}
2023-02-27T12:24:33.565Z	INFO	controller.vpcendpoint	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "VPCEndpoint", "worker count": 1}
2023-02-27T12:24:33.565Z	INFO	controller.dhcpoptions	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "DHCPOptions", "worker count": 1}
2023-02-27T12:24:33.565Z	INFO	controller.routetable	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "RouteTable", "worker count": 1}
2023-02-27T12:24:33.565Z	INFO	controller.elasticipaddress	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "ElasticIPAddress", "worker count": 1}
2023-02-27T12:24:33.565Z	INFO	controller.fieldexport	Starting workers	{"reconciler group": "services.k8s.aws", "reconciler kind": "FieldExport", "worker count": 1}
2023-02-27T12:24:33.565Z	INFO	controller.internetgateway	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "InternetGateway", "worker count": 1}
2023-02-27T12:24:33.565Z	INFO	controller.instance	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "Instance", "worker count": 1}
2023-02-27T12:24:33.565Z	INFO	controller.subnet	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "Subnet", "worker count": 1}
2023-02-27T12:24:33.565Z	INFO	controller.routetable	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "RouteTable", "worker count": 1}
2023-02-27T12:24:33.565Z	INFO	controller.transitgateway	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "TransitGateway", "worker count": 1}
2023-02-27T12:24:33.565Z	INFO	controller.instance	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "Instance", "worker count": 1}
2023-02-27T12:24:33.565Z	INFO	controller.natgateway	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "NATGateway", "worker count": 1}
2023-02-27T12:24:33.565Z	INFO	controller.dhcpoptions	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "DHCPOptions", "worker count": 1}
2023-02-27T12:24:33.565Z	INFO	controller.vpcendpoint	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "VPCEndpoint", "worker count": 1}
2023-02-27T12:24:33.565Z	INFO	controller.subnet	Starting workers	{"reconciler group": "ec2.services.k8s.aws", "reconciler kind": "Subnet", "worker count": 1}

After, we deployed the Instance object:

---
apiVersion: ec2.services.k8s.aws/v1alpha1
kind: Instance
metadata:
  name: {{ env_name }}-tower
  namespace: {{ kubernetes_namespace }}
spec:
  imageID: {{ redhat_ami.image_id }}
  instanceType: {{ ec2_type }}
  subnetID: {{ ec2_subnet }}
  keyName: {{ ec2_keyname }}
  securityGroupIDs:
    - {{ ec2_sg }}
  tags:
    - key: Name
      value: {{ env_name }}-tower

At this point, we see the instance being created in the ec2 console. controller logs:

2023-02-27T12:24:33.571Z	DEBUG	ackrt	> r.Sync	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "generation": 1}
2023-02-27T12:24:33.571Z	DEBUG	ackrt	>> r.resetConditions	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "generation": 1}
2023-02-27T12:24:33.571Z	DEBUG	ackrt	<< r.resetConditions	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "generation": 1}
2023-02-27T12:24:33.571Z	DEBUG	ackrt	>> rm.ResolveReferences	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.571Z	DEBUG	ackrt	<< rm.ResolveReferences	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.571Z	DEBUG	ackrt	>> rm.EnsureTags	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.571Z	DEBUG	ackrt	<< rm.EnsureTags	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.571Z	DEBUG	ackrt	>> rm.ReadOne	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.571Z	DEBUG	ackrt	>>> rm.sdkFind	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.571Z	DEBUG	ackrt	<<< rm.sdkFind	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1, "error": "resource not found"}
2023-02-27T12:24:33.571Z	DEBUG	ackrt	<< rm.ReadOne	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1, "error": "resource not found"}
2023-02-27T12:24:33.571Z	DEBUG	ackrt	>> r.createResource	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.571Z	DEBUG	ackrt	>>> r.setResourceManaged	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.571Z	DEBUG	ackrt	>>>> r.patchResourceMetadataAndSpec	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.571Z	DEBUG	ackrt	>>>>> kc.Patch (metadata + spec)	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.579Z	DEBUG	ackrt	patched resource metadata + spec	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1, "json": "{\"metadata\":{\"finalizers\":[\"finalizers.ec2.services.k8s.aws/Instance\"]}}"}
2023-02-27T12:24:33.579Z	DEBUG	ackrt	<<<<< kc.Patch (metadata + spec)	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.579Z	DEBUG	ackrt	<<<< r.patchResourceMetadataAndSpec	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.579Z	DEBUG	ackrt	marked resource as managed	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.579Z	DEBUG	ackrt	<<< r.setResourceManaged	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.579Z	DEBUG	ackrt	>>> rm.ResolveReferences	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.579Z	DEBUG	ackrt	<<< rm.ResolveReferences	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.579Z	DEBUG	ackrt	>>> rm.EnsureTags	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.579Z	DEBUG	ackrt	<<< rm.EnsureTags	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.579Z	DEBUG	ackrt	>>> rm.Create	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.579Z	DEBUG	ackrt	>>>> rm.sdkCreate	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:33.586Z	DEBUG	exporter.field-export-reconciler	error did not need requeue	{"error": "the source resource is not synced yet"}
2023-02-27T12:24:34.760Z	DEBUG	ackrt	<<<< rm.sdkCreate	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.760Z	DEBUG	ackrt	<<< rm.Create	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.760Z	DEBUG	ackrt	>>> rm.ReadOne	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.760Z	DEBUG	ackrt	>>>> rm.sdkFind	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.857Z	DEBUG	ackrt	<<<< rm.sdkFind	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.857Z	DEBUG	ackrt	<<< rm.ReadOne	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.857Z	DEBUG	ackrt	>>> r.patchResourceMetadataAndSpec	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.857Z	DEBUG	ackrt	>>>> kc.Patch (metadata + spec)	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.867Z	DEBUG	ackrt	patched resource metadata + spec	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1, "json": "{\"spec\":{\"capacityReservationSpecification\":{\"capacityReservationPreference\":\"open\"},\"cpuOptions\":{\"coreCount\":4,\"threadsPerCore\":2},\"ebsOptimized\":false,\"enclaveOptions\":{\"enabled\":false},\"maintenanceOptions\":{\"autoRecovery\":\"default\"},\"metadataOptions\":{\"httpEndpoint\":\"enabled\",\"httpProtocolIPv6\":\"disabled\",\"httpPutResponseHopLimit\":1,\"httpTokens\":\"optional\",\"instanceMetadataTags\":\"disabled\"},\"monitoring\":{},\"networkInterfaces\":[{\"description\":\"\",\"interfaceType\":\"interface\",\"networkInterfaceID\":\"eni-1111\",\"privateIPAddress\":\"REDACTED\",\"privateIPAddresses\":[{\"primary\":true,\"privateIPAddress\":\"REDACTED\"}],\"subnetID\":\"subnet-1111\"}],\"placement\":{\"availabilityZone\":\"REDACTEDa\",\"groupName\":\"\",\"tenancy\":\"default\"},\"privateDNSNameOptions\":{\"enableResourceNameDNSAAAARecord\":false,\"enableResourceNameDNSARecord\":false,\"hostnameType\":\"ip-name\"},\"privateIPAddress\":\"REDACTED\",\"securityGroups\":[\"sg-1111\"]},\"status\":{\"ackResourceMetadata\":{\"ownerAccountID\":\"REDACTED\",\"region\":\"REDACTED\"},\"amiLaunchIndex\":0,\"architecture\":\"x86_64\",\"enaSupport\":true,\"hypervisor\":\"xen\",\"instanceID\":\"REDACTED\",\"launchTime\":\"2023-02-27T12:24:34Z\",\"platformDetails\":\"Red Hat Enterprise Linux\",\"privateDNSName\":\"REDACTED\",\"publicDNSName\":\"\",\"rootDeviceName\":\"/dev/sda1\",\"rootDeviceType\":\"ebs\",\"sourceDestCheck\":true,\"state\":{\"code\":0,\"name\":\"pending\"},\"stateTransitionReason\":\"\",\"usageOperation\":\"RunInstances:0010\",\"usageOperationUpdateTime\":\"2023-02-27T12:24:34Z\",\"virtualizationType\":\"hvm\",\"vpcID\":\"vpc-1111\"}}"}
2023-02-27T12:24:34.867Z	DEBUG	ackrt	<<<< kc.Patch (metadata + spec)	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.867Z	DEBUG	ackrt	<<< r.patchResourceMetadataAndSpec	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.867Z	INFO	ackrt	created new resource	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.867Z	DEBUG	ackrt	<< r.createResource	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.867Z	DEBUG	ackrt	>> r.lateInitializeResource	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.867Z	DEBUG	ackrt	>>> rm.LateInitialize	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.867Z	DEBUG	ackrt	>>>> rm.sdkFind	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.873Z	DEBUG	exporter.field-export-reconciler	error did not need requeue	{"error": "the source resource is not synced yet"}
2023-02-27T12:24:34.904Z	DEBUG	ackrt	<<<< rm.sdkFind	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.904Z	DEBUG	ackrt	<<< rm.LateInitialize	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.904Z	DEBUG	ackrt	>>> r.patchResourceMetadataAndSpec	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.904Z	DEBUG	ackrt	>>>> kc.Patch (metadata + spec)	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.916Z	DEBUG	ackrt	patched resource metadata + spec	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1, "json": "{\"spec\":{\"hibernationOptions\":{\"configured\":false}},\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2023-02-27T12:24:34Z\",\"message\":\"Late initialization successful\",\"reason\":\"Late initialization successful\",\"status\":\"True\",\"type\":\"ACK.LateInitialized\"}]}}"}
2023-02-27T12:24:34.916Z	DEBUG	ackrt	<<<< kc.Patch (metadata + spec)	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.916Z	DEBUG	ackrt	<<< r.patchResourceMetadataAndSpec	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.916Z	DEBUG	ackrt	<< r.lateInitializeResource	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.916Z	DEBUG	ackrt	>> r.ensureConditions	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.916Z	DEBUG	ackrt	>>> rm.IsSynced	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.916Z	DEBUG	ackrt	<<< rm.IsSynced	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.916Z	DEBUG	ackrt	<< r.ensureConditions	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.916Z	DEBUG	ackrt	< r.Sync	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.916Z	DEBUG	ackrt	requeuing	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1, "after": "1m30s"}
2023-02-27T12:24:34.916Z	DEBUG	ackrt	> r.patchResourceStatus	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.916Z	DEBUG	ackrt	>> kc.Patch (status)	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.924Z	DEBUG	exporter.field-export-reconciler	error did not need requeue	{"error": "the source resource is not synced yet"}
2023-02-27T12:24:34.932Z	DEBUG	ackrt	patched resource status	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1, "json": "{\"metadata\":{\"generation\":3,\"resourceVersion\":\"81758069\"},\"spec\":{\"capacityReservationSpecification\":{\"capacityReservationPreference\":\"open\"},\"cpuOptions\":{\"coreCount\":4,\"threadsPerCore\":2},\"ebsOptimized\":false,\"enclaveOptions\":{\"enabled\":false},\"hibernationOptions\":{\"configured\":false},\"maintenanceOptions\":{\"autoRecovery\":\"default\"},\"metadataOptions\":{\"httpEndpoint\":\"enabled\",\"httpProtocolIPv6\":\"disabled\",\"httpPutResponseHopLimit\":1,\"httpTokens\":\"optional\",\"instanceMetadataTags\":\"disabled\"},\"monitoring\":{},\"networkInterfaces\":[{\"description\":\"\",\"interfaceType\":\"interface\",\"networkInterfaceID\":\"eni-1111\",\"privateIPAddress\":\"REDACTED\",\"privateIPAddresses\":[{\"primary\":true,\"privateIPAddress\":\"REDACTED\"}],\"subnetID\":\"subnet-1111\"}],\"placement\":{\"availabilityZone\":\"REDACTEDa\",\"groupName\":\"\",\"tenancy\":\"default\"},\"privateDNSNameOptions\":{\"enableResourceNameDNSAAAARecord\":false,\"enableResourceNameDNSARecord\":false,\"hostnameType\":\"ip-name\"},\"privateIPAddress\":\"REDACTED\",\"securityGroups\":[\"sg-1111\"],\"tags\":[{\"key\":\"Name\",\"value\":\"REDACTED-tower\"}]},\"status\":{\"ackResourceMetadata\":{\"ownerAccountID\":\"REDACTED\",\"region\":\"REDACTED\"},\"amiLaunchIndex\":0,\"architecture\":\"x86_64\",\"conditions\":[{\"lastTransitionTime\":\"2023-02-27T12:24:34Z\",\"message\":\"Late initialization successful\",\"reason\":\"Late initialization successful\",\"status\":\"True\",\"type\":\"ACK.LateInitialized\"},{\"lastTransitionTime\":\"2023-02-27T12:24:34Z\",\"message\":\"Resource synced successfully\",\"reason\":\"\",\"status\":\"True\",\"type\":\"ACK.ResourceSynced\"}],\"enaSupport\":true,\"hypervisor\":\"xen\",\"instanceID\":\"REDACTED\",\"launchTime\":\"2023-02-27T12:24:34Z\",\"platformDetails\":\"Red Hat Enterprise Linux\",\"privateDNSName\":\"REDACTED\",\"publicDNSName\":\"\",\"rootDeviceName\":\"/dev/sda1\",\"rootDeviceType\":\"ebs\",\"sourceDestCheck\":true,\"state\":{\"code\":0,\"name\":\"pending\"},\"stateTransitionReason\":\"\",\"usageOperation\":\"RunInstances:0010\",\"usageOperationUpdateTime\":\"2023-02-27T12:24:34Z\",\"virtualizationType\":\"hvm\",\"vpcID\":\"vpc-1111\"}}"}
2023-02-27T12:24:34.932Z	DEBUG	ackrt	<< kc.Patch (status)	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.932Z	DEBUG	ackrt	< r.patchResourceStatus	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1}
2023-02-27T12:24:34.932Z	DEBUG	ackrt	requeueing	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 1, "after": "1m30s"}
2023-02-27T12:24:34.938Z	DEBUG	ackrt	> r.Sync	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "generation": 3}
2023-02-27T12:24:34.938Z	DEBUG	ackrt	>> r.resetConditions	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "generation": 3}
2023-02-27T12:24:34.938Z	DEBUG	ackrt	<< r.resetConditions	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "generation": 3}
2023-02-27T12:24:34.938Z	DEBUG	ackrt	>> rm.ResolveReferences	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:34.938Z	DEBUG	ackrt	<< rm.ResolveReferences	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:34.938Z	DEBUG	ackrt	>> rm.EnsureTags	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:34.938Z	DEBUG	ackrt	<< rm.EnsureTags	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:34.938Z	DEBUG	ackrt	>> rm.ReadOne	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:34.938Z	DEBUG	ackrt	>>> rm.sdkFind	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:34.978Z	DEBUG	ackrt	<<< rm.sdkFind	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:34.978Z	DEBUG	ackrt	<< rm.ReadOne	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:34.978Z	DEBUG	ackrt	>> r.updateResource	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:34.978Z	DEBUG	ackrt	<< r.updateResource	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:34.978Z	DEBUG	ackrt	>> r.lateInitializeResource	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:34.978Z	DEBUG	ackrt	>>> rm.LateInitialize	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:34.978Z	DEBUG	ackrt	>>>> rm.sdkFind	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:35.016Z	DEBUG	ackrt	<<<< rm.sdkFind	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:35.016Z	DEBUG	ackrt	<<< rm.LateInitialize	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:35.016Z	DEBUG	ackrt	>>> r.patchResourceMetadataAndSpec	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:35.017Z	DEBUG	ackrt	no difference found between metadata and spec for desired and latest object.	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:35.017Z	DEBUG	ackrt	<<< r.patchResourceMetadataAndSpec	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:35.017Z	DEBUG	ackrt	<< r.lateInitializeResource	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:35.017Z	DEBUG	ackrt	>> r.ensureConditions	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:35.017Z	DEBUG	ackrt	>>> rm.IsSynced	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:35.017Z	DEBUG	ackrt	<<< rm.IsSynced	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:35.017Z	DEBUG	ackrt	<< r.ensureConditions	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:35.017Z	DEBUG	ackrt	< r.Sync	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:35.017Z	DEBUG	ackrt	requeuing	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3, "after": "1m30s"}
2023-02-27T12:24:35.017Z	DEBUG	ackrt	> r.patchResourceStatus	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:35.017Z	DEBUG	ackrt	>> kc.Patch (status)	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:35.030Z	DEBUG	ackrt	patched resource status	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3, "json": "{\"metadata\":{\"resourceVersion\":\"81758076\"},\"spec\":{\"tags\":[{\"key\":\"Name\",\"value\":\"REDACTED-tower\"}]},\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2023-02-27T12:24:35Z\",\"message\":\"Late initialization successful\",\"reason\":\"Late initialization successful\",\"status\":\"True\",\"type\":\"ACK.LateInitialized\"},{\"lastTransitionTime\":\"2023-02-27T12:24:35Z\",\"message\":\"Resource synced successfully\",\"reason\":\"\",\"status\":\"True\",\"type\":\"ACK.ResourceSynced\"}]}}"}
2023-02-27T12:24:35.030Z	DEBUG	ackrt	<< kc.Patch (status)	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:35.030Z	DEBUG	ackrt	< r.patchResourceStatus	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:24:35.030Z	DEBUG	ackrt	requeueing	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3, "after": "1m30s"}

when we run describe instance, we get the following in the 'Status' section:

Status:
  Ack Resource Metadata:
    Owner Account ID:  <REDACTED>
    Region:            <REDACTED>
  Ami Launch Index:    0
  Architecture:        x86_64
  Conditions:
    Last Transition Time:  2023-02-27T12:27:35Z
    Message:               Late initialization successful
    Reason:                Late initialization successful
    Status:                True
    Type:                  ACK.LateInitialized
    Last Transition Time:  2023-02-27T12:27:35Z
    Message:               Resource synced successfully
    Reason:
    Status:                True
    Type:                  ACK.ResourceSynced
  Ena Support:             true
  Hypervisor:              xen
  Instance ID:             <REDACTED>
  Launch Time:             2023-02-27T12:24:34Z
  Platform Details:        Red Hat Enterprise Linux
  Private DNS Name:        <REDACTED>
  Public DNS Name:
  Root Device Name:        /dev/sda1
  Root Device Type:        ebs
  Source Dest Check:       true
  State:
    Code:                       0
    Name:                       pending
  State Transition Reason:
  Usage Operation:              RunInstances:0010
  Usage Operation Update Time:  2023-02-27T12:24:34Z
  Virtualization Type:          hvm
  Vpc ID:                       <REDACTED>
Events:                         <none>

(notice the state.name value - is it normal at this state? it stays like this forever, long after the instance is up and running)

At this point, every manual change we do to the instance state, the controller does not revert it to the desired state, but does not report an error in the logs either, just seems to ignore it. Here's an example of controller logs reconcile after changing security group of the instance through AWS UI:

2023-02-27T12:36:36.396Z	DEBUG	ackrt	> r.Sync	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "generation": 3}
2023-02-27T12:36:36.396Z	DEBUG	ackrt	>> r.resetConditions	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "generation": 3}
2023-02-27T12:36:36.396Z	DEBUG	ackrt	<< r.resetConditions	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "generation": 3}
2023-02-27T12:36:36.396Z	DEBUG	ackrt	>> rm.ResolveReferences	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.396Z	DEBUG	ackrt	<< rm.ResolveReferences	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.396Z	DEBUG	ackrt	>> rm.EnsureTags	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.396Z	DEBUG	ackrt	<< rm.EnsureTags	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.396Z	DEBUG	ackrt	>> rm.ReadOne	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.396Z	DEBUG	ackrt	>>> rm.sdkFind	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.526Z	DEBUG	ackrt	<<< rm.sdkFind	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.526Z	DEBUG	ackrt	<< rm.ReadOne	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.526Z	DEBUG	ackrt	>> r.updateResource	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.526Z	INFO	ackrt	desired resource state has changed	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3, "diff": [{"Path":{"Parts":["Spec","BlockDeviceMappings"]},"A":null,"B":[{"deviceName":"/dev/sda1","ebs":{"deleteOnTermination":true}}]},{"Path":{"Parts":["Spec","SecurityGroups"]},"A":["sg-1111"],"B":["sg-2222"]}]}
2023-02-27T12:36:36.526Z	DEBUG	ackrt	>>> rm.Update	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.526Z	DEBUG	ackrt	>>>> rm.customUpdateInstance	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.526Z	DEBUG	ackrt	<<<< rm.customUpdateInstance	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.526Z	DEBUG	ackrt	<<< rm.Update	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3, "latest": {}}
2023-02-27T12:36:36.526Z	DEBUG	ackrt	>>> r.patchResourceMetadataAndSpec	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.526Z	DEBUG	ackrt	no difference found between metadata and spec for desired and latest object.	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.526Z	DEBUG	ackrt	<<< r.patchResourceMetadataAndSpec	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.526Z	INFO	ackrt	updated resource	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.526Z	DEBUG	ackrt	<< r.updateResource	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.526Z	DEBUG	ackrt	>> r.lateInitializeResource	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.526Z	DEBUG	ackrt	>>> rm.LateInitialize	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.526Z	DEBUG	ackrt	>>>> rm.sdkFind	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.581Z	DEBUG	ackrt	<<<< rm.sdkFind	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.581Z	DEBUG	ackrt	<<< rm.LateInitialize	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.581Z	DEBUG	ackrt	>>> r.patchResourceMetadataAndSpec	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.581Z	DEBUG	ackrt	no difference found between metadata and spec for desired and latest object.	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.581Z	DEBUG	ackrt	<<< r.patchResourceMetadataAndSpec	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.581Z	DEBUG	ackrt	<< r.lateInitializeResource	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.581Z	DEBUG	ackrt	>> r.ensureConditions	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.581Z	DEBUG	ackrt	>>> rm.IsSynced	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.581Z	DEBUG	ackrt	<<< rm.IsSynced	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.581Z	DEBUG	ackrt	<< r.ensureConditions	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.581Z	DEBUG	ackrt	< r.Sync	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.581Z	DEBUG	ackrt	requeuing	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3, "after": "1m30s"}
2023-02-27T12:36:36.581Z	DEBUG	ackrt	> r.patchResourceStatus	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.581Z	DEBUG	ackrt	>> kc.Patch (status)	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.600Z	DEBUG	ackrt	patched resource status	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3, "json": "{\"metadata\":{\"resourceVersion\":\"81771462\"},\"spec\":{\"tags\":[{\"key\":\"Name\",\"value\":\"REDACTED-tower\"}]},\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2023-02-27T12:36:36Z\",\"message\":\"Late initialization successful\",\"reason\":\"Late initialization successful\",\"status\":\"True\",\"type\":\"ACK.LateInitialized\"},{\"lastTransitionTime\":\"2023-02-27T12:36:36Z\",\"message\":\"Resource synced successfully\",\"reason\":\"\",\"status\":\"True\",\"type\":\"ACK.ResourceSynced\"}]}}"}
2023-02-27T12:36:36.600Z	DEBUG	ackrt	<< kc.Patch (status)	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.600Z	DEBUG	ackrt	< r.patchResourceStatus	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3}
2023-02-27T12:36:36.600Z	DEBUG	ackrt	requeueing	{"account": "REDACTED", "role": "", "region": "REDACTED", "kind": "Instance", "namespace": "REDACTED", "name": "REDACTED-tower", "is_adopted": false, "generation": 3, "after": "1m30s"}

In the controller log we can see that the security group changed (from sg-1111 to sg-2222), but in the instance itself, the security group is not reverted back to what is stated in the instance config, and stays sg-2222.

Expected outcome
Expecting the instance state to remediate from manual changes based on the Kubernetes Instance object (in the above case, we expecting the controller to set security group on the instance from sg-2222 to sg-1111)

Environment

• Kubernetes version
  Client Version: v1.24.2
  Kustomize Version: v4.5.4
  Server Version: v1.24.8-eks-ffeb93d
  Nodes version v1.24.10-eks-48e63af

• Using EKS (yes/no), if so version?
  Yes. versions stated above

• AWS service targeted (S3, RDS, etc.)
  EC2 controller

[LikithaVemulapalli]

Hello @Katnopic, currently we do not support Update codepath for Instances in ec2-controller. Could you please let us know how important is this feature for you, so that we can set our priorities! Thanks for opening this issue 🙂

[sergeipri]

Hello @Katnopic, currently we do not support Update codepath for Instances in ec2-controller. Could you please let us know how important is this feature for you, so that we can set our priorities! Thanks for opening this issue 🙂

Hi @LikithaVemulapalli,
Could you please elaborate on Update codepath for Instances in ec2-controller?
What is it and how it can be implemented.

I've seen similar issues 1288, 1355 where expected behaviour for reconciliation is to restore the aws object to its definition in case of drift.
According to @Katnopic findings the controller is not detecting drift, nor fixing it.

[jaypipes]

Hello @Katnopic, currently we do not support Update codepath for Instances in ec2-controller. Could you please let us know how important is this feature for you, so that we can set our priorities! Thanks for opening this issue slightly_smiling_face

Hi @LikithaVemulapalli, Could you please elaborate on Update codepath for Instances in ec2-controller? What is it and how it can be implemented.

I've seen similar issues 1288, 1355 where expected behaviour for reconciliation is to restore the aws object to its definition in case of drift. According to @Katnopic findings the controller is not detecting drift, nor fixing it.

@sergeipri @Katnopic What @LikithaVemulapalli was saying is that the update code paths for the Instance resource manager have not yet been implemented due to the complexity of the update code path and lack of resources on AWS side to complete the work. There is full intention to complete the update code path, but not sure when it will happen. I've changed this GH issue label to priority/important-soon to reflect your desire to see this implemented.

[eightseventhreethree]

The controller should probably include this issue in it's README because in it's current state it's basically unusable for actual EC2 instance use cases.

[michaelhtm]

Hello @eightseventhreethree, We are looking into this and we'll have a PR solving updates for at least SecurityGroupID updates, and drift detection for terminated instances. If there are any remaining, we can create a github issue to track them and prioritize them. Thanks for bringing this up!

[eightseventhreethree]

@michaelhtm The biggest ones that would make the controller useful would be; lifecycle recreation if
ami ID or User-data changes. The use case is migrating tons of TF state of AWS Outpost nodes in cluster. The creation of the AWS Outpost node works, but the Status of the instance stays stuck in Pending and changes to AMI ID or User-Data do nothing. If I can rely on the status and those other two primitives I can build on top of that.