Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Constant reconcile cycles for ECS Services #1519

Open
1 task done
daniel-maganto opened this issue Oct 10, 2024 · 2 comments
Open
1 task done

[Bug]: Constant reconcile cycles for ECS Services #1519

daniel-maganto opened this issue Oct 10, 2024 · 2 comments
Labels
bug Something isn't working needs:triage

Comments

@daniel-maganto
Copy link

daniel-maganto commented Oct 10, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Affected Resource(s)

service.ecs.aws.upbound.io/v1beta1
taskdefinition.ecs.aws.upbound.io/v1beta1

Resource MRs required to reproduce the bug

apiVersion: v1
items:
- apiVersion: ecs.aws.upbound.io/v1beta2
  kind: Service
  metadata:
    annotations:
      crossplane.io/composition-resource-name: service
      crossplane.io/external-create-pending: "2024-08-02T09:58:18Z"
      crossplane.io/external-create-succeeded: "2024-08-02T09:58:18Z"
      crossplane.io/external-name: dmaganto-service-test
    creationTimestamp: "2024-08-02T09:56:17Z"
    finalizers:
    - finalizer.managedresource.crossplane.io
    generateName: dmaganto-service-test-lmrk8-
    generation: 4
    labels:
      crossplane.io/claim-name: dmaganto-service-test
      crossplane.io/claim-namespace: default
      crossplane.io/composite: dmaganto-service-test-lmrk8
      tenant: dmaganto
    name: dmaganto-service-test
    ownerReferences:
    - apiVersion: microservices.dmaganto.infra/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: XECSService
      name: dmaganto-service-test-lmrk8
      uid: a59fbe11-d489-4baa-9cdc-6a5c0df47fca
    resourceVersion: "834777017"
    uid: 4e28d99a-3488-42d1-9326-231b26de9a55
  spec:
    deletionPolicy: Delete
    forProvider:
      cluster: dmaganto-test
      launchType: FARGATE
      loadBalancer:
      - containerName: dmaganto-service
        containerPort: 8080
        targetGroupArn: arn:aws:elasticloadbalancing:eu-central-1:YYYYYYYY:targetgroup/dmaganto-service/48ad6d4edfccf5cc
      networkConfiguration:
        securityGroups:
        - sg-05868ec477984d824
        subnets:
        - subnet-01dfaac2e8d274251
        - subnet-098df828ecf5271a2
      propagateTags: TASK_DEFINITION
      region: eu-central-1
      tags:
        Name: dmaganto-test
        Project: dmaganto
        crossplane-kind: service.ecs.aws.upbound.io
        crossplane-name: dmaganto-service-test
        crossplane-providerconfig: aws-ekstestdmaganto
      taskDefinition: dmaganto-service
      taskDefinitionRef:
        name: dmaganto-service-test-12345678913
    initProvider:
      desiredCount: 1
    managementPolicies:
    - Observe
    - Create
    - Update
    - Delete
    providerConfigRef:
      name: aws-ekstestdmaganto
  status:
    atProvider:
      cluster: dmaganto-test
      deploymentCircuitBreaker:
        enable: false
        rollback: false
      deploymentController:
        type: ECS
      deploymentMaximumPercent: 200
      deploymentMinimumHealthyPercent: 100
      desiredCount: 1
      enableEcsManagedTags: false
      enableExecuteCommand: false
      healthCheckGracePeriodSeconds: 0
      iamRole: /aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS
      id: arn:aws:ecs:eu-central-1:YYYYYYYY:service/dmaganto-test/dmaganto-service-test
      launchType: FARGATE
      loadBalancer:
      - containerName: dmaganto-service
        containerPort: 8080
        elbName: ""
        targetGroupArn: arn:aws:elasticloadbalancing:eu-central-1:YYYYYYYY:targetgroup/dmaganto-service/48ad6d4edfccf5cc
      networkConfiguration:
        assignPublicIp: false
        securityGroups:
        - sg-05868ec477984d824
        subnets:
        - subnet-01dfaac2e8d274251
        - subnet-098df828ecf5271a2
      platformVersion: LATEST
      propagateTags: TASK_DEFINITION
      schedulingStrategy: REPLICA
      tags:
        Name: dmaganto-test
        Project: dmaganto
        crossplane-kind: service.ecs.aws.upbound.io
        crossplane-name: dmaganto-service-test
        crossplane-providerconfig: aws-ekstestdmaganto
      tagsAll:
        Name: dmaganto-test
        Project: dmaganto
        crossplane-kind: service.ecs.aws.upbound.io
        crossplane-name: dmaganto-service-test
        crossplane-providerconfig: aws-ekstestdmaganto
      taskDefinition: dmaganto-service:72
      waitForSteadyState: false
    conditions:
    - lastTransitionTime: "2024-10-09T04:21:00Z"
      reason: ReconcileSuccess
      status: "True"
      type: Synced
    - lastTransitionTime: "2024-08-02T09:58:20Z"
      reason: Available
      status: "True"
      type: Ready
    - lastTransitionTime: "2024-10-06T06:06:48Z"
      reason: Success
      status: "True"
      type: LastAsyncOperation
- apiVersion: ecs.aws.upbound.io/v1beta2
  kind: TaskDefinition
  metadata:
    annotations:
      crossplane.io/composition-resource-name: taskdefinition
      crossplane.io/external-create-pending: "2024-08-02T09:56:17Z"
      crossplane.io/external-create-succeeded: "2024-08-02T09:56:17Z"
      crossplane.io/external-name: dmaganto-service
    creationTimestamp: "2024-08-02T09:56:02Z"
    finalizers:
    - finalizer.managedresource.crossplane.io
    generateName: dmaganto-service-12345678913-5wnb8-
    generation: 4
    labels:
      crossplane.io/claim-name: dmaganto-service-12345678913
      crossplane.io/claim-namespace: default
      crossplane.io/composite: dmaganto-service-12345678913-5wnb8
      taskdefinition: dmaganto-dmaganto-service-12345678913
      tenant: dmaganto
    name: dmaganto-service-test-12345678913
    ownerReferences:
    - apiVersion: microservices.dmaganto.infra/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: XECSTaskdef
      name: dmaganto-service-12345678913-5wnb8
      uid: 4fb769a5-8704-4920-9961-f442a1ef1678
    resourceVersion: "832341752"
    uid: 7d7d5c95-a02a-40eb-9fb9-a071a0bd817c
  spec:
    deletionPolicy: Delete
    forProvider:
      containerDefinitions: [  \n  {\n    \"command\": [\n      \"/opt/java/openjdk/bin/java\",\n
        \     \"-jar\",\n      \"/jar/devops-service.jar\"\n    ],\n    \"cpu\": 0,\n
        \   \"environment\": [\n      {\n        \"name\": \"STAGE\",\n        \"value\":
        \"test\"\n      },\n      {\n        \"name\": \"TENANT\",\n        \"value\":
        \"dmaganto\"\n      },\n      {\n        \"name\": \"OTEL_SERVICE_NAME\",\n
        \       \"value\": \"dmaganto-service\"\n      },\n      {\n        \"name\":
        \"PROJECT\",\n        \"value\": \"mytest\"\n      }\n    ],\n    \"essential\":
        true,\n    \"image\": \"JJJJJJJJ.dkr.ecr.eu-central-1.amazonaws.com/blueprint/devops-service:v0.0.116\",\n
        \   \"logConfiguration\": {\n      \"logDriver\": \"awsfirelens\"\n    },\n
        \   \"name\": \"dmaganto-service\",\n    \"portMappings\": [\n      {\n        \"containerPort\":
        8080,\n        \"hostPort\": 8080,\n        \"protocol\": \"tcp\"\n      }\n
        \   ]\n  },\n  {\n      \"name\": \"log_router\",\n      \"image\": \"JJJJJJJJ.dkr.ecr.eu-central-1.amazonaws.com/dmaganto/ecs-logging:3.0.0_2\",\n
        \     \"cpu\": 0,\n      \"memoryReservation\": 50,\n      \"essential\":
        true,\n      \"environment\": [\n          {\n              \"name\": \"TENANT\",\n
        \             \"value\": \"dmaganto\"\n          },\n          {\n              \"name\":
        \"STAGE\",\n              \"value\": \"test\"\n          },\n          {\n
        \             \"name\": \"OPENSEARCH_HOST\",\n              \"value\": \"vpc-dmaganto-nonprod-YYYYYYYY.eu-central-1.es.amazonaws.com\"\n
        \         }\n      ],\n      \"logConfiguration\": {\n          \"logDriver\":
        \"awslogs\",\n          \"options\": {\n              \"awslogs-group\": \"dmaganto-test\",\n
        \             \"awslogs-region\": \"eu-central-1\", \n              \"awslogs-stream-prefix\":
        \"firelens\"\n          }\n      },\n      \"firelensConfiguration\": {\n
        \         \"type\": \"fluentbit\",\n          \"options\": {\n              \"config-file-type\":
        \"file\",\n              \"config-file-value\": \"/extra.conf\"\n          }\n
        \     }\n  }\n]
      cpu: "256"
      executionRoleArn: arn:aws:iam::YYYYYYYY:role/dmaganto-service-test-executionrole
      executionRoleArnRef:
        name: dmaganto-service-test-executionrole
      executionRoleArnSelector:
        matchLabels:
          role: dmaganto-service-test-executionrole
      family: dmaganto-service
      memory: "512"
      networkMode: awsvpc
      region: eu-central-1
      requiresCompatibilities:
      - FARGATE
      tags:
        Project: dmaganto
        crossplane-kind: taskdefinition.ecs.aws.upbound.io
        crossplane-name: dmaganto-service-test-12345678913
        crossplane-providerconfig: aws-ekstestdmaganto
      taskRoleArn: arn:aws:iam::YYYYYYYY:role/dmaganto-service-test-taskrole
    initProvider: {}
    managementPolicies:
    - Create
    - LateInitialize
    - Observe
    - Delete
    providerConfigRef:
      name: aws-ekstestdmaganto
  status:
    atProvider:
      arn: arn:aws:ecs:eu-central-1:YYYYYYYY:task-definition/dmaganto-service:72
      arnWithoutRevision: arn:aws:ecs:eu-central-1:YYYYYYYY:task-definition/dmaganto-service
      containerDefinitions: '[{"command":["/opt/java/openjdk/bin/java","-jar","/jar/devops-service.jar"],"cpu":0,"environment":[{"name":"OTEL_SERVICE_NAME","value":"dmaganto-service"},{"name":"PROJECT","value":"mytest"},{"name":"STAGE","value":"test"},{"name":"TENANT","value":"dmaganto"}],"essential":true,"image":"JJJJJJJJ.dkr.ecr.eu-central-1.amazonaws.com/blueprint/devops-service:v0.0.116","logConfiguration":{"logDriver":"awsfirelens"},"mountPoints":[],"name":"dmaganto-service","portMappings":[{"containerPort":8080,"hostPort":8080,"protocol":"tcp"}],"systemControls":[],"volumesFrom":[]},{"cpu":0,"environment":[{"name":"OPENSEARCH_HOST","value":"vpc-dmaganto-nonprod-YYYYYYYY.eu-central-1.es.amazonaws.com"},{"name":"STAGE","value":"test"},{"name":"TENANT","value":"dmaganto"}],"essential":true,"firelensConfiguration":{"options":{"config-file-type":"file","config-file-value":"/extra.conf"},"type":"fluentbit"},"image":"JJJJJJJJ.dkr.ecr.eu-central-1.amazonaws.com/dmaganto/ecs-logging:3.0.0_2","logConfiguration":{"logDriver":"awslogs","options":{"awslogs-group":"dmaganto-test","awslogs-region":"eu-central-1","awslogs-stream-prefix":"firelens"}},"memoryReservation":50,"mountPoints":[],"name":"log_router","portMappings":[],"systemControls":[],"user":"0","volumesFrom":[]}]'
      cpu: "256"
      executionRoleArn: arn:aws:iam::YYYYYYYY:role/dmaganto-service-test-executionrole
      family: dmaganto-service
      id: dmaganto-service
      ipcMode: ""
      memory: "512"
      networkMode: awsvpc
      pidMode: ""
      requiresCompatibilities:
      - FARGATE
      revision: 72
      skipDestroy: false
      tags:
        crossplane-kind: taskdefinition.ecs.aws.upbound.io
        crossplane-name: dmaganto-service-test-12345678913
        crossplane-providerconfig: aws-ekstestdmaganto
      tagsAll:
        crossplane-kind: taskdefinition.ecs.aws.upbound.io
        crossplane-name: dmaganto-service-test-12345678913
        crossplane-providerconfig: aws-ekstestdmaganto
      taskRoleArn: arn:aws:iam::YYYYYYYY:role/dmaganto-service-test-taskrole
      trackLatest: false
    conditions:
    - lastTransitionTime: "2024-10-07T08:04:35Z"
      reason: ReconcileSuccess
      status: "True"
      type: Synced
    - lastTransitionTime: "2024-08-02T09:56:18Z"
      reason: Available
      status: "True"
      type: Ready
    - lastTransitionTime: "2024-08-02T09:56:17Z"
      reason: Success
      status: "True"
      type: LastAsyncOperation
- apiVersion: ecs.aws.upbound.io/v1beta2
  kind: Cluster
  metadata:
    annotations:
      crossplane.io/composition-resource-name: cluster
      crossplane.io/external-create-pending: "2024-08-02T09:50:24Z"
      crossplane.io/external-create-succeeded: "2024-08-02T09:50:25Z"
      crossplane.io/external-name: dmaganto-test
    creationTimestamp: "2024-08-02T09:50:23Z"
    finalizers:
    - finalizer.managedresource.crossplane.io
    generateName: dmaganto-test-b8d6c-
    generation: 3
    labels:
      cluster: dmaganto-test
      crossplane.io/claim-name: dmaganto-test
      crossplane.io/claim-namespace: default
      crossplane.io/composite: dmaganto-test-b8d6c
      name: dmaganto-test
      tenant: dmaganto
    name: dmaganto-test
    ownerReferences:
    - apiVersion: microservices.dmaganto.infra/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: XECSCluster
      name: dmaganto-test-b8d6c
      uid: 22d50c7d-92d7-4a89-a7ad-08a0eda88855
    resourceVersion: "769021746"
    uid: f8b0e9d8-ca40-48b5-813c-7b9ba7f512e1
  spec:
    deletionPolicy: Delete
    forProvider:
      region: eu-central-1
      setting:
      - name: containerInsights
        value: enabled
      tags:
        Name: dmaganto-test
        Project: dmaganto
        crossplane-kind: cluster.ecs.aws.upbound.io
        crossplane-name: dmaganto-test
        crossplane-providerconfig: aws-ekstestdmaganto
    initProvider: {}
    managementPolicies:
    - '*'
    providerConfigRef:
      name: aws-ekstestdmaganto
  status:
    atProvider:
      arn: arn:aws:ecs:eu-central-1:YYYYYYYY:cluster/dmaganto-test
      id: arn:aws:ecs:eu-central-1:YYYYYYYY:cluster/dmaganto-test
      setting:
      - name: containerInsights
        value: enabled
      tags:
        Name: dmaganto-test
        Project: dmaganto
        crossplane-kind: cluster.ecs.aws.upbound.io
        crossplane-name: dmaganto-test
        crossplane-providerconfig: aws-ekstestdmaganto
      tagsAll:
        Name: dmaganto-test
        Project: dmaganto
        crossplane-kind: cluster.ecs.aws.upbound.io
        crossplane-name: dmaganto-test
        crossplane-providerconfig: aws-ekstestdmaganto
    conditions:
    - lastTransitionTime: "2024-08-02T09:50:35Z"
      reason: Available
      status: "True"
      type: Ready
    - lastTransitionTime: "2024-08-02T09:50:25Z"
      reason: ReconcileSuccess
      status: "True"
      type: Synced
    - lastTransitionTime: "2024-08-02T09:50:35Z"
      reason: Success
      status: "True"
      type: LastAsyncOperation
kind: List
metadata:
  resourceVersion: ""

Steps to Reproduce

Seems that ECS Service if you are using a Taskdefinition without version (because you want to use always the latest) it always is trying to update reconcile the resources creating a infinite bucle.

What happened?

It creates a tons of calls to AWS Resources and even generate many reboots in ECS Provider pod.

Relevant Error Output Snippet

crossplane provider-aws-ecs-11475561cee7-9d6c4f57-wkktv package-runtime 2024-10-10T05:52:51Z	DEBUG	provider-aws	Diff detected	{"uid": "13bc9db8-33ff-461d-a0fe-1bc24c8145d0", "name": "service1-policy-dev", "gvk": "ecs.aws.upbound.io/v1beta1, Kind=Service", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"task_definition\":*terraform.ResourceAttrDiff{Old:\"service1-policy:3889\", New:\"service1-policy\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}
crossplane provider-aws-ecs-11475561cee7-9d6c4f57-wkktv package-runtime 2024-10-10T05:52:51Z	DEBUG	provider-aws	Diff detected	{"uid": "81b0a42b-5d86-49e6-9e02-9cf90fcb6013", "name": "service2-dev", "gvk": "ecs.aws.upbound.io/v1beta1, Kind=Service", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"task_definition\":*terraform.ResourceAttrDiff{Old:\"service2:312\", New:\"service2\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}
crossplane provider-aws-ecs-11475561cee7-9d6c4f57-wkktv package-runtime 2024-10-10T05:52:51Z	DEBUG	provider-aws	Diff detected	{"uid": "1b94a1e8-fd25-4e51-80eb-b5225cd5f37c", "name": "service3-preprod", "gvk": "ecs.aws.upbound.io/v1beta1, Kind=Service", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"task_definition\":*terraform.ResourceAttrDiff{Old:\"service3:8\", New:\"service3\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}

Crossplane Version

1.15.1

Provider Version

1.4.0

Kubernetes Version

v1.30.4-eks-a737599

Kubernetes Distribution

EKS

Additional Info

No response
@daniel-maganto daniel-maganto added bug Something isn't working needs:triage labels Oct 10, 2024
@daniel-maganto
Copy link
Author

I also tested with the latest versions of Crossplane (1.17.1) and Providers (1.14.0) and the same behaviour is happening

@daniel-maganto
Copy link
Author

The problem might be resolved when you reference a TaskDefinitionRef it resolves the TaskDefinition using: status.atProvider.family + status.atProvider.revision instead of only status.atProvider.family

This is how I reference TaskDefinitionRef case:

        - combine:
            strategy: string
            string:
              fmt: '%s-%s-%s'
            variables:
            - fromFieldPath: spec.parameters.family
            - fromFieldPath: spec.resourceConfig.env
            - fromFieldPath: spec.parameters.task_id
          toFieldPath: spec.forProvider.taskDefinitionRef.name
          type: CombineFromComposite

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working needs:triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@daniel-maganto