Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: construct library for ECS #1058

Merged
merged 165 commits into from
Nov 6, 2018
Merged
Show file tree
Hide file tree
Changes from 129 commits
Commits
Show all changes
165 commits
Select commit Hold shift + click to select a range
939452c
Ignore swp files
SoManyHs Aug 23, 2018
7302a2f
Add ECS lib and example
SoManyHs Aug 23, 2018
4c7127a
Add service and task def constructs
SoManyHs Aug 24, 2018
98749ec
WIP adding VPC/ASG
SoManyHs Aug 24, 2018
6d0942f
Refactor ECS demo
Aug 24, 2018
bddfe56
Add more props on Service
SoManyHs Aug 24, 2018
3574611
Add TODO for AMI IDs
SoManyHs Sep 6, 2018
a63c377
Weird hack?
SoManyHs Sep 7, 2018
b00e0dd
Update example to instantiate service separately
SoManyHs Sep 7, 2018
80ba2e4
WIP Add Container Definitions
SoManyHs Sep 12, 2018
0980f52
WIP Add other Task Def properties
SoManyHs Sep 14, 2018
8648540
Merge remote-tracking branch 'official/master' into ecs-demo
Sep 28, 2018
2d53d89
Indentation 4 -> 2
Sep 28, 2018
dbdab86
Fix type errors
Sep 28, 2018
ee6b427
Use SSM parameter for ECS Optimized AMI
SoManyHs Sep 28, 2018
2b16dad
Move containerdefinition to its own file
Sep 28, 2018
16a92c0
Merge branch 'ecs-demo' of github.com:SoManyHs/aws-cdk into ecs-demo
Sep 28, 2018
ae881c2
Turning ContainerDefinition into a construct
Sep 28, 2018
0672e6a
Add enum for placement constraints
SoManyHs Oct 1, 2018
4f93a5b
Rename fleet to autoScalingGroup
SoManyHs Oct 1, 2018
9b61b89
Qualify "PlacementConstraintType" usage with package name
Oct 1, 2018
98bf81b
ECR can return itself as container image
Oct 1, 2018
ae1d525
Automatically generate execution and task roles
Oct 1, 2018
23072a9
Make taskRole public
Oct 1, 2018
e5cc7f2
Remove shellCommand from healthCheck
SoManyHs Oct 1, 2018
047eed8
Rename toContainerDefinitionJson to renderContainerDefinition
SoManyHs Oct 1, 2018
65d3f79
Some work on Service
Oct 1, 2018
11541d3
Merge branch 'ecs-demo' of github.com:SoManyHs/aws-cdk into ecs-demo
Oct 1, 2018
fb17532
Separate Fargate and ECS clusters
SoManyHs Oct 1, 2018
be8cb86
Separate ECS and Fargate Service
SoManyHs Oct 1, 2018
b8cce3a
Separate ECS/Fargate TaskDefinition
SoManyHs Oct 1, 2018
2741608
Fix lint errors
SoManyHs Oct 1, 2018
a901912
Fix lint errors
SoManyHs Oct 1, 2018
a91bda7
Merge branch 'ecs-demo' of github.com:SoManyHs/aws-cdk into ecs-demo
Oct 2, 2018
51adff5
Fix bugs in service and task def
SoManyHs Oct 2, 2018
70ae682
Fix AMI ID getter for ECS clusters
SoManyHs Oct 2, 2018
298e0a7
Add Capabilities in Linux Parameters on ContainerDefs
SoManyHs Oct 2, 2018
1541edd
Add Capabilities in Linux Parameters on ContainerDefs
SoManyHs Oct 2, 2018
a154d09
Add default arg to SSMProvider.getString
SoManyHs Oct 2, 2018
4b46685
Add Devices to LinuxParameters
SoManyHs Oct 2, 2018
9875c6e
Add Tmpfs to LinuxParameters
SoManyHs Oct 2, 2018
1044d1c
Update ecs demo with full Linux Parameters example
SoManyHs Oct 2, 2018
e7f8000
Rename renderLogDriver
SoManyHs Oct 2, 2018
e7f7b7d
Do most of Service
Oct 2, 2018
dda869d
Merge branch 'ecs-demo' of github.com:SoManyHs/aws-cdk into ecs-demo
Oct 2, 2018
97083dc
Add Ulimits
SoManyHs Oct 2, 2018
a3e3615
Fix log driver
SoManyHs Oct 2, 2018
b913f0e
Make add* functions consistently plural
SoManyHs Oct 2, 2018
fd21c0d
Make load balancers respect network mode
Oct 2, 2018
3b4695c
Merge branch 'ecs-demo' of github.com:SoManyHs/aws-cdk into ecs-demo
Oct 2, 2018
187f62b
Make add* functions consistently plural
SoManyHs Oct 2, 2018
6912e3f
Add portMappings to ContainerDefinition
SoManyHs Oct 2, 2018
2bf2d57
Missing semicolons
SoManyHs Oct 2, 2018
f477d8d
Add MountPoints to ContainerDefinition
SoManyHs Oct 2, 2018
b947f51
Change 'addContainer' to take the props and return ContainerDefinition.
Oct 2, 2018
ab83d35
Merge branch 'ecs-demo' of github.com:SoManyHs/aws-cdk into ecs-demo
Oct 2, 2018
30cb925
Add VolumesFrom to ContainerDefinition
SoManyHs Oct 2, 2018
c64e184
Add fargate example for demo purposes
SoManyHs Oct 3, 2018
566aac3
Remove Service Role, fix DesiredCount default, make task memory defin…
Oct 3, 2018
96e3105
Merge branch 'ecs-demo' of github.com:SoManyHs/aws-cdk into ecs-demo
Oct 3, 2018
48cd8d4
Starting on port mapping defaults
Oct 3, 2018
e311293
Add validation on port mappings with tests
SoManyHs Oct 3, 2018
4ffed09
Implement ingressPort method
SoManyHs Oct 3, 2018
97d3db3
Make ALB listener ACTUALLY default to 'true'
Oct 4, 2018
4f9b548
Add dependency on listener
SoManyHs Oct 4, 2018
e8aa186
Integ test for Fargate service
SoManyHs Oct 4, 2018
eb4a562
Merge branch 'ecs-demo' of github.com:SoManyHs/aws-cdk into ecs-demo
Oct 4, 2018
e06a3ec
Add VolumesFrom to ContainerDefinition
SoManyHs Oct 2, 2018
95b8d84
Add fargate example for demo purposes
SoManyHs Oct 3, 2018
5595707
Starting on port mapping defaults
Oct 3, 2018
a1b78f4
Make ALB listener ACTUALLY default to 'true'
Oct 4, 2018
6f08406
Add validation on port mappings with tests
SoManyHs Oct 3, 2018
5312362
Implement ingressPort method
SoManyHs Oct 3, 2018
6f42026
Add dependency on listener
SoManyHs Oct 4, 2018
1983858
Integ test for Fargate service
SoManyHs Oct 4, 2018
4ea6b8b
hello-cdk-fargate example works
SoManyHs Oct 4, 2018
e34d515
Add lazy evaluation of network configuration property
SoManyHs Oct 4, 2018
bf90c27
Add LoadBalancedFargateService L3 construct
SoManyHs Oct 4, 2018
ffbad9f
Start LoadBalancedFargateServiceApplet
SoManyHs Oct 4, 2018
d1b22bd
Application AutoScaling WIP
Oct 4, 2018
7840634
Merge branch 'ecs-demo' of github.com:SoManyHs/aws-cdk into ecs-demo
Oct 4, 2018
d1ec640
Make declarative example work
Oct 4, 2018
36e1ba2
Add import exports to Cluster
SoManyHs Oct 5, 2018
178898f
Fix portMappings/ingressPort
SoManyHs Oct 5, 2018
b73bdad
Clean up fargate integ test
SoManyHs Oct 5, 2018
049557e
Add integ test for load balanced service in bridge mode
SoManyHs Oct 5, 2018
2295ef2
Add integ test for awsvpc network mode on EcsCluster
SoManyHs Oct 5, 2018
f34c23c
Fix ingress/containerPort on LBs
SoManyHs Oct 5, 2018
c491a1d
Cleanup example
SoManyHs Oct 5, 2018
430f81e
Add cloudwatch metrics
SoManyHs Oct 5, 2018
dca6972
Bridge mode works with host port specified
SoManyHs Oct 5, 2018
cf528d6
Merge remote-tracking branch 'origin/master' into ecs-demo
Oct 12, 2018
ef51786
Fix libs to be in line with latest API changes from master
Oct 12, 2018
c3b543a
Merge remote-tracking branch 'origin/master' into ecs-demo
Oct 15, 2018
64b6d06
Fix build errors
Oct 17, 2018
af5a2a4
Merge remote-tracking branch 'origin/master' into ecs-demo
Oct 17, 2018
51097fd
Fix ingressPort/containerPort tests
SoManyHs Oct 22, 2018
a8f50e4
Add AssetImage: Docker image built from source
Oct 18, 2018
4baaeef
feat(aws-cdk): allow uploading Docker images as assets
Oct 18, 2018
7b19dd2
Add LoadBalancedEcsService L3 Construct
SoManyHs Oct 22, 2018
71766e1
More logging, blacklist manifest parts.
Oct 25, 2018
9bea6be
Add ability to specify cluster size
SoManyHs Oct 24, 2018
4ab455c
Merge remote-tracking branch 'origin/master' into ecs-demo
Oct 26, 2018
be9f447
feat(aws-ecs): add Task AutoScaling to Service
Oct 26, 2018
2eab513
Add unit tests for EcsCluster
SoManyHs Oct 26, 2018
57857a9
Merge branch 'master' into ecs-demo
SoManyHs Oct 29, 2018
4cf23fc
Unit tests for Cluster
SoManyHs Oct 29, 2018
37d3c4e
Set volumes on TaskDefinition
SoManyHs Oct 30, 2018
1ad3669
Clarify doc string for memoryLimit and memoryReservation
SoManyHs Oct 30, 2018
78d2a49
WIP Unit tests on EcsTaskDefinition
SoManyHs Oct 30, 2018
ca61a74
WIP Unit tests for EcsService
SoManyHs Oct 30, 2018
98345b3
WIP Unit tests for FargateService
SoManyHs Oct 30, 2018
35ebd9c
WIP unit tests Fargate Task Definition
SoManyHs Oct 30, 2018
7bbe9bb
feat(applets): integrate into toolkit
Oct 29, 2018
09a182a
Merge branch 'huijbers/autoscaling' into ecs-demo
Oct 30, 2018
2fd2e7c
Add docs and readme and a couple of tests
Oct 30, 2018
9825127
Unit test for error case on service creation
SoManyHs Oct 30, 2018
eea141e
Merge remote-tracking branch 'SoManyHs/ecs/local-image' into ecs-demo
Oct 31, 2018
67761f5
Merge remote-tracking branch 'origin/huijbers/applets' into ecs-demo
Oct 31, 2018
ccd8f34
Merge branch 'ecs-demo' of github.com:SoManyHs/aws-cdk into ecs-demo
Oct 31, 2018
15755d0
Add integ test expectations
Oct 31, 2018
c28393a
Merge branch 'ecs-demo' of github.com:SoManyHs/aws-cdk into ecs-demo
Oct 31, 2018
7faa05d
Revert "feat(applets): integrate into toolkit"
Oct 31, 2018
4f561cd
Run build
Oct 31, 2018
47e5251
Fix tests
Oct 31, 2018
6152ddc
Update readme
SoManyHs Oct 31, 2018
62ff3a4
Remove cdk.json files
SoManyHs Oct 31, 2018
b2fa1db
Update error message
SoManyHs Nov 1, 2018
410d263
Rename Ecs resources to Ec2*
SoManyHs Nov 1, 2018
360d42f
Add test for adding Ulimit
SoManyHs Nov 1, 2018
8c76aa4
Add test for adding links
SoManyHs Nov 1, 2018
45a82c8
Fix asset tests
Nov 1, 2018
631c35d
Incorporate previous PR review comments
Nov 1, 2018
013a52c
Don't forget to unignore bundled .js file
Nov 1, 2018
ce083db
Review comments
Nov 1, 2018
73680ef
Fix integ expectation locations
Nov 1, 2018
61c0edb
Update unit test in line with new 'Dockerfile' presence test
Nov 1, 2018
3a7778b
Fix test for asset image
Nov 1, 2018
6e5c3b2
Merge remote-tracking branch 'origin/master' into ecs-demo
Nov 1, 2018
36687dc
Merge remote-tracking branch 'origin/master' into ecs-demo
Nov 1, 2018
ca9fcb3
Exercise handler in unit tests
Nov 1, 2018
0dbbe63
Fix container healthCheck defaults
SoManyHs Nov 1, 2018
cc69519
Add ScratchSpace method
SoManyHs Nov 2, 2018
645b9a0
Fix aws-sdk stubbing by bringing in an external dependency
Nov 2, 2018
4ed101b
Fix handler, update expectation
Nov 2, 2018
54e4665
Minor tweaks that Clare wanted to see
Nov 2, 2018
e2d338f
Rename autoscaling methods so C# doesn't complain
Nov 2, 2018
b0f54a8
Set ECS_AWSVPC_BLOCK_IMDS=true config option
Nov 2, 2018
8f0f6b1
Update test expectation
Nov 4, 2018
15b8804
Add helper scripts to compile sources faster
Nov 4, 2018
07f1f57
Extract capacity from EC2 cluster
Nov 4, 2018
d883432
Fix rename of members on ScalableTaskCount
Nov 4, 2018
edd4d5f
Make a generic TaskDefinition for both launch types
Nov 5, 2018
ed0956a
Fix import
Nov 5, 2018
33a35b9
One EcsCluster to Rule Them All
SoManyHs Nov 5, 2018
2431901
Update README
SoManyHs Nov 6, 2018
7c0a148
Update integ test expectations
SoManyHs Nov 6, 2018
32db35d
Add moar unit tests
SoManyHs Nov 6, 2018
233a376
Rename EcsCluster => Cluster, add slightly more handler tests
Nov 6, 2018
b616248
Make a single static accessor for all types of Container images
Nov 6, 2018
ea6eeae
Small fixes
Nov 6, 2018
98990b4
Re-add missing handler
Nov 6, 2018
ffa7360
Merge remote-tracking branch 'origin/master' into ecs-demo
Nov 6, 2018
908db1f
Oops wrong name
Nov 6, 2018
64c37fc
Merge remote-tracking branch 'origin/master' into ecs-demo
Nov 6, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,5 @@ pack
coverage
.nyc_output
.LAST_BUILD
*.swp
./examples/cdk-examples-typescript/hello-cdk-ecs/cdk.json
rix0rrr marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"app": "../node_modules/.bin/cdk-applet-js fargate-service.yml"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# applet is loaded from the local ./test-applet.js file
eladb marked this conversation as resolved.
Show resolved Hide resolved
applet: @aws-cdk/aws-ecs:LoadBalancedFargateServiceApplet
image: 'amazon/amazon-ecs-sample'
cpu: "2048"
memoryMiB: "1024"
37 changes: 37 additions & 0 deletions examples/cdk-examples-typescript/hello-cdk-ecs/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
import ec2 = require('@aws-cdk/aws-ec2');
import { InstanceType } from '@aws-cdk/aws-ec2';
import ecs = require('@aws-cdk/aws-ecs');
import cdk = require('@aws-cdk/cdk');

class BonjourECS extends cdk.Stack {
constructor(parent: cdk.App, name: string, props?: cdk.StackProps) {
super(parent, name, props);

// For better iteration speed, it might make sense to put this VPC into
// a separate stack and import it here. We then have two stacks to
// deploy, but VPC creation is slow so we'll only have to do that once
// and can iterate quickly on consuming stacks. Not doing that for now.
const vpc = new ec2.VpcNetwork(this, 'MyVpc', { maxAZs: 2 });
const cluster = new ecs.Ec2Cluster(this, 'Ec2Cluster', {
vpc,
size: 3,
instanceType: new InstanceType("t2.xlarge")
});

// Instantiate ECS Service with just cluster and image
const ecsService = new ecs.LoadBalancedEc2Service(this, "Ec2Service", {
cluster,
memoryLimitMiB: 512,
image: ecs.DockerHub.image("amazon/amazon-ecs-sample"),
});

// Output the DNS where you can access your service
new cdk.Output(this, 'LoadBalancerDNS', { value: ecsService.loadBalancer.dnsName });
}
}

const app = new cdk.App();

new BonjourECS(app, 'Bonjour');

app.run();
29 changes: 29 additions & 0 deletions examples/cdk-examples-typescript/hello-cdk-fargate/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
import ec2 = require('@aws-cdk/aws-ec2');
import ecs = require('@aws-cdk/aws-ecs');
import cdk = require('@aws-cdk/cdk');

class BonjourFargate extends cdk.Stack {
constructor(parent: cdk.App, name: string, props?: cdk.StackProps) {
super(parent, name, props);

// Create VPC and Fargate Cluster
// NOTE: Limit AZs to avoid reaching resource quotas
const vpc = new ec2.VpcNetwork(this, 'MyVpc', { maxAZs: 2 });
const cluster = new ecs.FargateCluster(this, 'Cluster', { vpc });

// Instantiate Fargate Service with just cluster and image
const fargateService = new ecs.LoadBalancedFargateService(this, "FargateService", {
cluster,
image: ecs.DockerHub.image("amazon/amazon-ecs-sample"),
});

// Output the DNS where you can access your service
new cdk.Output(this, 'LoadBalancerDNS', { value: fargateService.loadBalancer.dnsName });
}
}

const app = new cdk.App();

new BonjourFargate(app, 'Bonjour');

app.run();
2 changes: 2 additions & 0 deletions examples/cdk-examples-typescript/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,9 @@
"@aws-cdk/aws-cognito": "^0.14.1",
"@aws-cdk/aws-dynamodb": "^0.14.1",
"@aws-cdk/aws-ec2": "^0.14.1",
"@aws-cdk/aws-ecs": "^0.14.1",
"@aws-cdk/aws-elasticloadbalancing": "^0.14.1",
"@aws-cdk/aws-elasticloadbalancingv2": "^0.14.1",
"@aws-cdk/aws-iam": "^0.14.1",
"@aws-cdk/aws-lambda": "^0.14.1",
"@aws-cdk/aws-neptune": "^0.14.1",
Expand Down
2 changes: 1 addition & 1 deletion packages/@aws-cdk/assets/lib/asset.ts
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ export class Asset extends cdk.Construct {
// for tooling to be able to package and upload a directory to the
// s3 bucket and plug in the bucket name and key in the correct
// parameters.
const asset: cxapi.AssetMetadataEntry = {
const asset: cxapi.FileAssetMetadataEntry = {
path: this.assetPath,
id: this.uniqueId,
packaging: props.packaging,
Expand Down
29 changes: 29 additions & 0 deletions packages/@aws-cdk/aws-ecr/lib/repository-ref.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import ecs = require('@aws-cdk/aws-ecs');
import iam = require('@aws-cdk/aws-iam');
import cdk = require('@aws-cdk/cdk');

Expand Down Expand Up @@ -44,6 +45,13 @@ export abstract class RepositoryRef extends cdk.Construct {
const parts = cdk.ArnUtils.parse(this.repositoryArn);
return `${parts.account}.dkr.ecr.${parts.region}.amazonaws.com/${parts.resourceName}`;
}

/**
* Refer to a particular image tag from this repository
*/
public getImage(tag: string = "latest"): ecs.IContainerImage {
return new EcrImage(this, tag);
}
}

export interface RepositoryRefProps {
Expand All @@ -67,3 +75,24 @@ class ImportedRepository extends RepositoryRef {
// FIXME: Add annotation about policy we dropped on the floor
}
}

class EcrImage implements ecs.IContainerImage {
public readonly imageName: string;
private readonly repositoryArn: string;

constructor(repository: RepositoryRef, tag: string) {
this.imageName = `${repository.repositoryUri}:${tag}`;
this.repositoryArn = repository.repositoryArn;
}

public bind(containerDefinition: ecs.ContainerDefinition): void {
// This image will be in ECR, so we need appropriate permissions.
containerDefinition.addToExecutionPolicy(new iam.PolicyStatement()
.addActions("ecr:BatchCheckLayerAvailability", "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage")
.addResource(this.repositoryArn));

containerDefinition.addToExecutionPolicy(new iam.PolicyStatement()
.addActions("ecr:GetAuthorizationToken", "logs:CreateLogStream", "logs:PutLogEvents")
.addAllResources());
}
}
1 change: 1 addition & 0 deletions packages/@aws-cdk/aws-ecr/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@
},
"dependencies": {
"@aws-cdk/aws-iam": "^0.14.1",
"@aws-cdk/aws-ecs": "^0.14.1",
"@aws-cdk/cdk": "^0.14.1"
},
"homepage": "https://github.com/awslabs/aws-cdk"
Expand Down
144 changes: 142 additions & 2 deletions packages/@aws-cdk/aws-ecs/README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,142 @@
## The CDK Construct Library for AWS Elastic Container Service (ECS)
This module is part of the [AWS Cloud Development Kit](https://github.com/awslabs/aws-cdk) project.
## AWS Elastic Container Service (ECS) Construct Library

This package contains constructs for working with **AWS Elastic Container
Service** (ECS). The simplest example of using this library looks like this:

```ts
// Create an ECS cluster (backed by an AutoScaling group)
const cluster = new ecs.Ec2Cluster(this, 'Cluster', {
vpc,
size: 3,
instanceType: new InstanceType("t2.xlarge")
});

// Instantiate ECS Service with an automatic load balancer
const ecsService = new ecs.LoadBalancedEc2Service(this, 'Service', {
cluster,
memoryLimitMiB: 512,
image: ecs.DockerHub.image("amazon/amazon-ecs-sample"),
});
```

### Fargate vs ECS

There are two sets of constructs in this library; one to run tasks on ECS and
one to run Tasks on fargate.

- Use the `Ec2Cluster`, `Ec2TaskDefinition` and `Ec2Service` constructs to
run tasks on EC2 instances running in your account.
- Use the `FargateCluster`, `FargateTaskDefinition` and `FargateService`
constructs to run tasks on instances that are managed for you by AWS.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From reading this, I don't understand why we need specialized classes for FargateTaskDefinition and FargateService. I am assuming I'll find out why later on, but intuitively I expected just the cluster to have a different concrete class.

The reason why this might be an issue for customers is if they want to port an ECS service/task-def from EC2 to Fargate. Ideally, if they could just change the cluster and it will Just Work, that would have been a nice capability.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know, and this was our original plan as well. But there are a number of differences that make this tricky:

  • There is a difference in how you specify task sizes. For EC2 tasks, you must specify task memory limits on the container. For Fargate tasks, you must specify instance sizes on the TaskDefinition.
  • EC2 task definitions allow different placement constraints.
  • Fargate task definitions do not allow configuring networkmode.

After discussion we decided that task migration wasn't that likely, so it was okay if it didn't work out of the box.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we document these differences in the README?

Still feels like we are missing something... For example, if a 3rd party library wants to vend a TaskDefinition for reuse, they will have to either vend two (one for Fargate and one for EC2) or users will not be able to use them interchangeably.

Maybe what we can do is define a TaskDefinition which is the union of all options, so I can define a task definition that can be used across EC2/Fargate. Does that make any sense?

Copy link

@tiffanyfay tiffanyfay Nov 1, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it should be broken up--we don't break it up in the console even. One of the things that we talk about is how you can easily migrate from EC2 to Fargate and vice versa. Also if you use awsvpc you can easily create a task definition that works for both. There isn't any field in a task definition that makes it Fargate or EC2 specific--just the values or IAM roles you give--but both can take the same things (sans task placement). Right now the only checks happen when you include the requires compatibilities field--which can take ec2 and fargate, not just one.

For service via the API it seems the differences are just giving fargate/ec2 as the launch type, platform version if fargate (and not the latest), and task placement (ec2).

I think breaking things up further than the API is a bit confusing.

Also for clusters, why not just one create cluster. And then an option to add instances as a separate function given into create cluster? Thinking this since any cluster you create as ec2 can be used for Fargate. Also if you create a cluster without instances for fargate, you can just use the addinstances function after the fact.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rix0rrr What do you mean you specify instance sizes in a task definition? You can only have task/container mem/cpu.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tiffanyfay brings up a fair point about ECS clusters that can contain both Fargate and EC2 instances in them.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Begin able to add instances also sounds like a useful feature

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you look at the current implementation in Ec2Cluster, that is currently what we are doing already, except via an ASG. @rix0rrr and I had already discussed this following this other comment regarding configurability of ASGs. The reason why we wanted to keep this separate was to hide any properties having to do with instances from the Fargate namespace.

I'm not sure what you mean by size in your addInstances example -- the size field on the Ec2Cluster construct currently refers to the size of the cluster, e.g. number of instances. I assume you mean count?

Finally, Rico and I had also already discussed that userData is something that you should be able to add separately -- I believe it was still in the context of an ASG.

In re: adding instances separately, do we actually have data supporting instances existing in clusters outside of an ASG? Both the console and the ecs-cli currently only provision capacity through ASGs today, which is why we modeled the Ec2Cluster construct this way. I'm not against the idea of adding an API that adds individual capacity, but it seems a little strange to want instances that isn't part of an ASG, but perhaps @tiffanyfay you have a concrete customer use case?

Copy link

@tiffanyfay tiffanyfay Nov 1, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, saw that and think that the ASG should be moved into the addInstances instead. So since ECS has a cluster which has both EC2 and Fargate vs them being individual clusters, I think there should be just one Cluster construct.

My thought of addInstances (or maybe it should be addInstanceASG or something) is not to scale existing types, but to add new types. Examples being:
-Create cluster with size: 3 type: t2.micro and I want to add 4 t2.medium instances in an ASG.
-Create cluster without instances (e.g. Fargate in the current code) and add 4 t2.medium instances in an ASG.

Basically I want to easily be able to have different types of instances. So actually it would need to only allow one asg in order to return something you can scale individually. Right now it can be done by adding another autoscaling launch config and asg to your CloudFormation template and updating your stack.

Kind of like the RegisterContainerInstance API plus creating the asg/instances for you.

Oh, yes, count-- I had just copy pasted.

Would it be able to be added during instance/asg creation or only after the fact and updating?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The RegisterContainerInstance is an internal API only used by the agent; it's not externally exposed to customers IIRC. I think having an addCapacity API isn't a bad idea, as long as it's well-defined and aligned with best practices. Currently, we have plans to allow instance auto-scaling (which @rix0rrr is working on).

For me I am still not convinced there is good justification for not separating the cluster types. As previously mentioned, there is an extremely low number of customers today who run both kinds of tasks/services in the same cluster -- we originally supported it to ease migration, but in today's world where both services are available as options it seems like we should be supporting a more streamlined workflow. Do you happen to have concrete customer use cases for what kind of system is running both kinds of services?

## Cluster

An `Ec2Cluster` or `FargateCluster` defines the infrastructure to run your
tasks on. If you create an ECS cluster, an AutoScalingGroup of EC2 instances
running the latest ECS Optimized AMI will implicitly be created for you.

You can run many tasks on a single cluster.

To create a cluster, go:
rix0rrr marked this conversation as resolved.
Show resolved Hide resolved

```ts
const cluster = new ecs.FargateCluster(this, 'Cluster', {
vpc: vpc
});
```

## TaskDefinition

A `TaskDefinition` describes what a single copy of a **Task** should look like.
A task definition has one or more containers; typically, it has one
main container (the *default container* is the first one that's added
to the task definition, and it will be marked *essential*) and optionally
some supporting containers which are used to support the main container,
doings things like upload logs or metrics to monitoring services.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't we merge Ec2TaskDefinition with FargateTaskDefinition?

To add containers to a `TaskDefinition`, call `addContainer()`:

```ts
const taskDefinition = new ecs.FargateTaskDefinition(this, 'TaskDef', {
memoryMiB: '512'
cpu: 256
});

taskDefinition.addContainer('main', {
// Use an image from DockerHub
image: ecs.DockerHub.image('amazon/amazon-ecs-sample')
});
```

### Images

Images supply the software that runs inside the container. Images can be
obtained from either DockerHub or from ECR repositories:

* `ecs.DockerHub.image(imageName)`: use a publicly available image from
DockerHub.
* `repository.getImage(tag)`: use the given ECR repository as the image
to start.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, we can add some sugar to make this a bit more symmetric and discoverable:

ecs.Image.dockerHub(image)
esc.Image.ecr(repository, tag)

(I would still keep repository.getImage(tag), but it would be nice if the usage from ECS's side would be symmetric)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would remove repository.getImage(tag) to avoid the dependency between ECR => ECS (that's the wrong direction)

## Service

A `Service` instantiates a `TaskDefinition` on a `Cluster` a given number of
times, optionally associating them with a load balancer. Tasks that fail will
automatically be restarted.

```ts
const taskDefinition;

const service = new ecs.Ec2Service(this, 'Service', {
cluster,
taskDefinition,
desiredCount: 5
});
```

### Include a load balancer

`Services` are load balancing targets and can be directly attached to load
balancers:

```ts
const service = new ecs.FargateService(this, 'Service', { /* ... */ });
rix0rrr marked this conversation as resolved.
Show resolved Hide resolved

const lb = new elbv2.ApplicationLoadBalancer(this, 'LB', { vpc, internetFacing: true });
const listener = lb.addListener('Listener', { port: 80 });
listener.addTargets('ECS', {
port: 80,
targets: [service]
});
```

There are two higher-level constructs available which include a load balancer for you:

* `LoadBalancedFargateService`
* `LoadBalancedEc2Service`

## Task AutoScaling

You can configure the task count of a service to match demand. Task AutoScaling is
configured by calling `autoScaleTaskCount()`:

```ts
const scaling = service.autoScaleTaskCount({ maxCapacity: 10 });
scaling.scaleOnCpuUtilization('CpuScaling', {
targetUtilizationPercent: 50
});
```

Task AutoScaling is powered by *Application AutoScaling*. Refer to that for
more information.

## Instance AutoScaling

If you're running on Fargate, AWS will manage the physical machines that your
containers are running on for you. If you're running an ECS cluster however,
your EC2 instances might fill up as your number of Tasks goes up.

To avoid placement errors, you will want to configure AutoScaling for your
EC2 instance group so that your instance count scales with demand.

TO BE IMPLEMENTED
rix0rrr marked this conversation as resolved.
Show resolved Hide resolved
Loading