FargateService trying to update ECS service when CODE_DEPLOY used

[adrialucena-seat]

The FargateService class is trying to update the ECS Service when a new task definition is provided to the service when DeploymentControllerType.CODE_DEPLOY is specified. When using Blue/Green deployment strategy powered by CodeDeploy in a Fargate Service, only the desired count, deployment configuration, and health check grace period can be updated using the update-service. Otherwise a new CodeDeploy deployment should be created.
So I haven't been able to do any deploy on production environment with CDK since I use CODE_DEPLOY.

Reproduction Steps

new FargateService(
            this,
            "FargateService",
            FargateServiceProps
                .builder()
                .cluster(this.props.cluster)
                .serviceName(this.getPrefixResourceName())
                .assignPublicIp(false)
                .desiredCount(this.props.desiredCount)
                .healthCheckGracePeriod(Duration.seconds(60))
                .minHealthyPercent(100)
                .maxHealthyPercent(200)
                .taskDefinition(this.taskDefinition)
                .securityGroup(this.securityGroup)
                .vpcSubnets(this.privateSubnets)
                .deploymentController(
                    DeploymentController
                        .builder()
                        .type(DeploymentControllerType.CODE_DEPLOY)
                        .build()
                )
                .cloudMapOptions(
                    CloudMapOptions
                        .builder()
                        .name(this.getPrefixResourceName())
                        .cloudMapNamespace(this.props.serviceDiscoveryNamespace)
                        .dnsRecordType(DnsRecordType.A)
                        .dnsTtl(Duration.seconds(60))
                        .build()
                )
                .build()
        );

Error Log

 Unable to update task definition on services with a CODE_DEPLOY deployment controller. Use AWS CodeDeploy to trigger a new deployment. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException; Request ID: 83f32eb8-4ed5-4263-b944-9df4d7fa3a62)

Environment

• **CLI Version :**1.30.0
• **Framework Version:**1.30.0
• **OS :**MacOS Mojave
• **Language :**Java

---

This is 🐛 Bug Report

[sonofachamp]

Hey @adrialucena-seat it appears CodeDeploy Blue/Green deployments are not yet supported in CloudFormation for ECS/Fargate.

Related:
aws-cloudformation/cloudformation-coverage-roadmap#37
aws/containers-roadmap#130

[adrialucena-seat]

Thank you @sonofachamp , I was already aware of that. That's why I created CodeDeploy application and CodeDeploy deployment group manually in the aws console associating it with the ECS/Fargate service created by CDK. The bug I opened was more related with ECS/Fargate using Blue/Green deployment. When creating a new task definition revision it should trigger the CodeDeploy deployment group associated with that service and not updating the service, as it will fail.
I guess this issue could be solved in parallel with the current working progress trying to support CodeDeploy Blue/Green deployments for Ecs/Fargate with CloudFormation.

[adrialucena-seat]

@sonofachamp as in my previous comment I explained, I think there is no need to wait for the CloudFormation support.

What CDK should do when CODE_DEPLOY is specified in Deployment Controller Type in the FargateService class is to trigger the CodeDeploy Deployment Group associated with the ECS Service and not the "update service" that traditionally did.

I created the CodeDeploy Deployment Group manually in AWS Console so no need to wait for CloudFormation support.
I still believe this is a bug and not a feature request.

[konstantinj]

Hi,

having the same issue and it's not limited to Fargate Services.

Unable to update task definition on services with a CODE_DEPLOY deployment controller. Use AWS CodeDeploy to trigger a new deployment.

But I agree it should be possible to manage the infrastructure, meaning the service config + creation of new taskDefinition revisions with CDK. I haven't investigated what CDK is doing and what's part of Cloudformation. But it seems that one of it sets the newly created taskDefinition revision as active on the service and this is of course not ok when using CodeDeploy for ECS.

Is ECS supposed to trigger a CodeDeploy Deployment when a new taskDefinition revision is set as active on the service? I can't find any documentation about it.

[konstantinj]

Found a workaround to be able to maintain new task definition revisions with CDK and use Code Deploy with an external CLI command.

        const taskDefinitionProps = {
            cpu: 1024,
            memoryLimitMiB: 2048,
        }

        const taskDefinition = new FargateTaskDefinition(this, 'TaskDefinition', taskDefinitionProps)

        const containerDefinitionOptions = {
            image: ContainerImage.fromEcrRepository(Repository.fromRepositoryName(this, 'Repository', 'docker'), 'latest'),
            environment: {
                NAME: 'latest',
            },
            logging: LogDriver.awsLogs({
                logGroup: logGroup,
                streamPrefix: 'docker',
            }),
        }
        const container = taskDefinition.addContainer('docker', containerDefinitionOptions)

        container.addPortMappings({
            containerPort: 80,
        })

        container.addUlimits({
            name: UlimitName.NOFILE,
            softLimit: 16384,
            hardLimit: 16384,
        })

        const service = new FargateService(this, 'Service', {
            cluster: cluster,
            healthCheckGracePeriod: Duration.seconds(60),
            minHealthyPercent: 50,
            maxHealthyPercent: 200,
            deploymentController: {
                type: DeploymentControllerType.CODE_DEPLOY,
            },
            taskDefinition: taskDefinition,
        })

        const taskDefinitionRev = new FargateTaskDefinition(this, 'TaskDefinitionNew', {
            ...taskDefinitionProps,
            family: taskDefinition.family,
        })

        const cfnTaskDefinition = taskDefinitionRev.node.defaultChild as CfnTaskDefinition
        cfnTaskDefinition.cfnOptions.updateReplacePolicy = CfnDeletionPolicy.RETAIN

        const containerRev = taskDefinitionRev.addContainer('docker', {
            ...containerDefinitionOptions,
            environment: {
                ...containerDefinitionOptions.environment,
                NAME: 'docker-new',
            },
            image: ContainerImage.fromAsset('docker'),
        })

        containerRev.addPortMappings(...container.portMappings)
        containerRev.addUlimits(...container.ulimits)

Explanation: Cloudformation would always try to update the current task definition which would lead to an immediate update of the ecs service. This is not possible (currently) when using DeploymentControllerType.CODE_DEPLOY. By adding a new task definition logical resource in the stack but giving it the same task family, Cloudformation will create a new task definition revision. The revision that is active in the ecs service must not be changed anymore. Unfortunately Cloudformation will remove old revisions. There the updateReplacePolicy needs to be set to retain.

For automating it the newly created task definition revision can be retrieved from the updated stack which can then be used in a cli call like:

aws ecs deploy ...

[konstantinj]

Let's wait for #1559 and everything will be fine.

[MrArnoldPalmer]

Hey there, I'm consolidating the issues around ecs blue/green deployment support into #1559. If you think this is a separate issue feel free to reopen.