Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(ApplicationLoadBalancedFargateService): Container size won't update #13127

Open
tai-acall opened this issue Feb 18, 2021 · 6 comments
Open

(ApplicationLoadBalancedFargateService): Container size won't update #13127

tai-acall opened this issue Feb 18, 2021 · 6 comments
Labels
@aws-cdk/aws-ecs-patterns Related to ecs-patterns library bug This issue is a bug. effort/small Small work item – less than a day of effort p2

Comments

@tai-acall
Copy link

Once the ECS stack first time deployment done, even I change the container size setting in CDK and run deployment command it won't update the ECS task size.

Reproduction Steps

  1. Build a ECS Stack with ApplicationLoadBalancedFargateService construct class with memoryLimitMiB and cpu option
  2. run deploy command
  3. change the value in memoryLimitMiB and cpu option
  4. run deploy command again

What did you expect to happen?

the task size should change to the latest value as define in CDK.

What actually happened?

the task size still as the first time deployment setting.

Environment

  • CDK CLI Version : @aws-cdk/aws-ecs-patterns "1.88.0
  • Framework Version:
  • Node.js Version:
  • OS :
  • Language (Version):

Other


This is 🐛 Bug Report

@tai-acall tai-acall added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Feb 18, 2021
@peterwoodworth peterwoodworth added the @aws-cdk/aws-ecs-patterns Related to ecs-patterns library label Feb 18, 2021
@SoManyHs SoManyHs added needs-reproduction This issue needs reproduction. p2 effort/small Small work item – less than a day of effort and removed needs-triage This issue or PR still needs to be triaged. labels Feb 25, 2021
@JacobTheEvans
Copy link

I am experiencing the same issue did you ever find a work around @tai-acall?

@tai-acall
Copy link
Author

I have side containers in my work, so I let the memory limitation of ECS service as variable then setting each container limitation separately. that will trigger the deploy process because the task definition will be changed while the ECS service limit be changed and limitation of each container will be changed too.

ex:
ECS Service memory = 1024
Container A memory usage limit = 512
Container B memory = ECS Service memory - Container A memory usage limit

ECS Service memory = 2048
Container A memory usage limit = 512
Container B memory = ECS Service memory - Container A memory usage limit
-> Task definition changed !

I know this is not the best way to do that but might the easiest way in my work.

I hope this will help. @JacobTheEvans

@JacobTheEvans
Copy link

Thanks @tai-acall ! I found a different solution.

Solution
As I dug deeper what I found was that memoryLimitMiB and cpu in ApplicationLoadBalancedFargateService is not used as the default for the containers. I went into the documentation here and found the following under the memoryLimitMiB section: This default is set in the underlying FargateTaskDefinition construct Meaning that if you want the default to be set each of the containers without having to manually set it you must add the memoryLimitMiB and cpu keys to the FargateTaskDefinition

TLDR
Set memoryLimitMiB and cpu in FargateTaskDefinition not in ApplicationLoadBalancedFargateService

@gnagy
Copy link
Contributor

gnagy commented Feb 4, 2022

I found a related issue, sorry for piggybacking on this one, let me know if I should open a separate ticket.

The latest docs for setting memory limits recommends to set it both at the task and container level, so apps like Java can observe the limits set:
https://aws.amazon.com/blogs/containers/how-amazon-ecs-manages-cpu-and-memory-resources/

[Update 12/11/2020] Some applications are container-aware and can configure themselves to take full advantage of the resources available inside the container. The latest versions of Java are a good example of this pattern. For this reason, some of these applications work best when CPU and memory resources are explicitly configured at the container level, in addition to the configuration at the task level. In other words, while containers without specific resource configurations can nominally access all task resources, some applications will interpret this configuration differently and may artificially limit the amount of resources they think are available to the container.

Would it be feasible to also set memoryLimitMiB also in ContainerDefinitionOptions in ApplicationLoadBalancedServiceBase ?

mergify bot pushed a commit that referenced this issue Jul 25, 2022
…1201)

I ran into an issue when deploying a Java application on Fargate, where the container kept getting killed because of out-of-memory condition. Setting  memoryLimitMiB also on container fixed the problem, based on:

https://aws.amazon.com/blogs/containers/how-amazon-ecs-manages-cpu-and-memory-resources/

> [Update 12/11/2020] Some applications are container-aware and can configure themselves to take full advantage of the resources available inside the container. The latest versions of Java are a good example of this pattern. For this reason, some of these applications work best when CPU and memory resources are explicitly configured at the container level, in addition to the configuration at the task level. In other words, while containers without specific resource configurations can nominally access all task resources, aws/amazon-ecs-agent#1735.

I initially asked about this in #13127

----

### All Submissions:

* [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
mrgrain pushed a commit to mrgrain/aws-cdk that referenced this issue Jul 27, 2022
…s#21201)

I ran into an issue when deploying a Java application on Fargate, where the container kept getting killed because of out-of-memory condition. Setting  memoryLimitMiB also on container fixed the problem, based on:

https://aws.amazon.com/blogs/containers/how-amazon-ecs-manages-cpu-and-memory-resources/

> [Update 12/11/2020] Some applications are container-aware and can configure themselves to take full advantage of the resources available inside the container. The latest versions of Java are a good example of this pattern. For this reason, some of these applications work best when CPU and memory resources are explicitly configured at the container level, in addition to the configuration at the task level. In other words, while containers without specific resource configurations can nominally access all task resources, aws/amazon-ecs-agent#1735.

I initially asked about this in aws#13127

----

### All Submissions:

* [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
mrgrain pushed a commit to mrgrain/aws-cdk that referenced this issue Jul 28, 2022
…s#21201)

I ran into an issue when deploying a Java application on Fargate, where the container kept getting killed because of out-of-memory condition. Setting  memoryLimitMiB also on container fixed the problem, based on:

https://aws.amazon.com/blogs/containers/how-amazon-ecs-manages-cpu-and-memory-resources/

> [Update 12/11/2020] Some applications are container-aware and can configure themselves to take full advantage of the resources available inside the container. The latest versions of Java are a good example of this pattern. For this reason, some of these applications work best when CPU and memory resources are explicitly configured at the container level, in addition to the configuration at the task level. In other words, while containers without specific resource configurations can nominally access all task resources, aws/amazon-ecs-agent#1735.

I initially asked about this in aws#13127

----

### All Submissions:

* [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
mrgrain pushed a commit to mrgrain/aws-cdk that referenced this issue Jul 28, 2022
…s#21201)

I ran into an issue when deploying a Java application on Fargate, where the container kept getting killed because of out-of-memory condition. Setting  memoryLimitMiB also on container fixed the problem, based on:

https://aws.amazon.com/blogs/containers/how-amazon-ecs-manages-cpu-and-memory-resources/

> [Update 12/11/2020] Some applications are container-aware and can configure themselves to take full advantage of the resources available inside the container. The latest versions of Java are a good example of this pattern. For this reason, some of these applications work best when CPU and memory resources are explicitly configured at the container level, in addition to the configuration at the task level. In other words, while containers without specific resource configurations can nominally access all task resources, aws/amazon-ecs-agent#1735.

I initially asked about this in aws#13127

----

### All Submissions:

* [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@peterwoodworth
Copy link
Contributor

merged in #21201

@github-actions
Copy link

github-actions bot commented Aug 5, 2022

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

mergify bot pushed a commit that referenced this issue Aug 10, 2022
…evel" (#21530)

This reverts #21201 due to #21484: Recent update breaks FargateService L3 constructs which have added additional containers to the task definition

Closes #21484
Re-opens #13127 

----

### All Submissions:

* [ ] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@mrgrain mrgrain reopened this Aug 17, 2022
@peterwoodworth peterwoodworth removed the needs-reproduction This issue needs reproduction. label Aug 17, 2022
josephedward pushed a commit to josephedward/aws-cdk that referenced this issue Aug 30, 2022
…s#21201)

I ran into an issue when deploying a Java application on Fargate, where the container kept getting killed because of out-of-memory condition. Setting  memoryLimitMiB also on container fixed the problem, based on:

https://aws.amazon.com/blogs/containers/how-amazon-ecs-manages-cpu-and-memory-resources/

> [Update 12/11/2020] Some applications are container-aware and can configure themselves to take full advantage of the resources available inside the container. The latest versions of Java are a good example of this pattern. For this reason, some of these applications work best when CPU and memory resources are explicitly configured at the container level, in addition to the configuration at the task level. In other words, while containers without specific resource configurations can nominally access all task resources, aws/amazon-ecs-agent#1735.

I initially asked about this in aws#13127

----

### All Submissions:

* [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
josephedward pushed a commit to josephedward/aws-cdk that referenced this issue Aug 30, 2022
…evel" (aws#21530)

This reverts aws#21201 due to aws#21484: Recent update breaks FargateService L3 constructs which have added additional containers to the task definition

Closes aws#21484
Re-opens aws#13127 

----

### All Submissions:

* [ ] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-ecs-patterns Related to ecs-patterns library bug This issue is a bug. effort/small Small work item – less than a day of effort p2
Projects
None yet
Development

No branches or pull requests

8 participants