Convert ECS platform plugin to use resource manager

[izaaklauer]

This converts the ECS plugin from its own resource lifecycle logic to use resource manager, and implements the Status plugin.

Addresses #1645
Closes #2061

What changes

From a user perspective, there are three changes:

More detailed deployment UI

Previous waypoint deploy console output:

New console output:

Users can now see the progress on creating or discovering all of the resources that go into an ECS deployment.

Better rollback behavior on deployment failure

If a deployment fails partway through, waypoint will now call destroy on each resource that was created during that deployment, leaving fewer orphaned resources. Example:

Note that after the rate limit exception, waypoint destroyed the ALB listener and target group before exiting.

Status

I've implemented status functions on cluster and service resources. The service resource produces additional task resources. You can view them with waypoint status -app=<app>

Example of a service coming online:

One drawback of this change: we run a status check immediately after a deployment, at which point generally the service exists, but the tasks do not yet exist. The initial status check will generally look like this, until a user enables app polling or runs the upcoming waypoint status -refresh command:

Technical notes

There are so many of individual resources that an ECS deployment creates or needs to be directly aware of. The full list is:

• cluster
• execution IAM role
• task IAM role (optional)
• internal security group
• external security group
• log group
• subnets
• target group
• ALB
• ALB Listener
• route53 record (optional)
• task definition
• service

This is also a bit of a nerve-wracking change, as it touches the entire surface area of the plugin and there aren't any tests. I tested:

• many config param combinations
• backwards compat (i.e. this can destroy 0.5.1 deployments, and 0.5.1 can destroy and release these deployments)
• app destroy
• mid-deploy failure rollback
• deleting AWS resources out of band before running a deployment destroy
• status checks performed locally and by a remote runner

That said, if anyone has any specific trial workflows in mind, please give them a whirl or let me know.

Future considerations

Improved destroy logic

We only have destroy implemented for ALB listener, target group, and service, which was the state before this change. Other resources are either app-scoped (security groups, etc), or globally scoped (log group). We have a DestroyWorkspace plugin func, which could be useful for this, but I feel like we might need a DestroyApp plugin func too, as I don't think most of these resources are workspace scoped.

Relevant issue: #805

More status functions

I've only implemented status on the cluster and service resources, which I think are the most dynamic and valuable. Implementing more status functions would result in more aws API usage, which bring us closer to hitting api rate limits for large waypoint installations. AWS rate limits to 20 RPS per region. With this change, each status check will result in 4 GET api calls, so with the default check interval of 30 seconds on the latest deploy only, waypoint won't be able to support more than 150 apps.

Once we have a plan to address the rate limit problem, we can implement status functions on more resources.

ECS Releaser plugin

The ECS releaser plugin cannot be easily replaced by the ALB plugin (context: #1577 (comment)). The ECS plugin doesn't create any resources though - only modify existing deployment resources - so I don't think it needs resource manager.

Incidental changes

Apologies for not making these smaller separate PRs.

• Closes Specifying ECS sidecar containers without health checks causes a panic #2090
• Improved error messages. Amazon errors will often be something vague like "ARN validation failed". If you're making three or four AWS calls as part of one resource, that doesn't tell you which call failed. This now wraps errors to give that context.
• Internal security group now authorizes ingress from the external security group, instead of 0.0.0.0/0. This improves the security of ECS deploys, as it prevents the public from accessing disabled tasks, or DDOSing individual tasks.
• New config parameter for ecs: ingress_port, that allows you to configure the external port the load balancer uses. It still defaults to 443 if a cert is configured, and 80 otherwise.
• Checking more aws calls for "404" equivalent responses, and doing something appropriate instead of erroring.
• All aws calls are now called withContext, so if the aws api is slow or hanging it should be possible to cancel the operation.
• Security group creation doesn't call AuthorizeIngress to add ingress rules if the security group already exists. This reduces the prodigious number of API calls we make to create a deployment.

[izaaklauer]

Using this as an argmapper marker type

[izaaklauer]

Leaving this for now because it's not new behavior, and because the future task to replace internal IDs with sequence IDs should help this considerably.

[izaaklauer]

This bit fixes #2090

[izaaklauer]

Longer term it probably makes sense to have some kind of exponential backoff with a context cancel.

[izaaklauer]

I don't love having vpcId live inside the subnet state. I tried having subnets return a "VpcId" marker type, but we're not currently incorporating the resource return values into the argmapper soup. I think this is good enough for now, and if this problem crops up repeatedly we can change the plugin sdk.

[izaaklauer]

We're only setting (not using) this for now, but it'll be valuable in the future when we decide to implement delete on security groups.

[izaaklauer]

With the improved resource visibility, we can see that this turns out to be the slowest part of the ecs deployment most of the time. Go figure.

[briancain]

:whoa: there's a lot here! This really illustrates how nice that StatusAll helper is back over on the sdk side! Nice. I haven't tried this yet but spent some time reading through the code and gave some feedback here.

[mitchellh]

Looks great! Re: rate limits, I know in Terraform we hit rate limits pretty early and while I'm not sure what it does not, back in the day we detected the error and implemented a backoff directly in the providers. I wonder if we want to do something similar here eventually.

[izaaklauer]

With the status checks, I'm mostly worried about pushing waypoint's steady-state usage over the limit. If that's too high, we'd be constantly doing backoffs.

Backoffs in the deploy and destroy funcs seems like a great idea though.