Wait for newly created services to stabilize.

[ejholmes]

I think it might be good to wait for services to stabilize only when they're being created. Couple reasons:

1. It'll make this change easier (no downtime).
2. In cases where a replacement is required, it will result in no downtime.
3. It shouldn't effect the 99% case where a service is just getting updated with a new task definition.

[ejholmes]

May want to also set a hard timeout on this. Like 5-10 minutes.

[ejholmes]

Turns out, the waiter will check up to 40 times, with a 15 second delay between checks, so this will wait up to 10 minutes, which seems about right.

https://github.com/aws/aws-sdk-go/blob/2e1a49c71fb7d8f6b7e8cca2259ed9b68d9b97ce/service/ecs/waiters.go#L41-L42

[mwildehahn]

👍

[ejholmes]

Since this can potentially block the sqs polling loop, provisioning should happen in a goroutine.

[ejholmes]

Ok, made some changes to this after testing in my environment:

1. Updated go-jmespath so that WaitUntilServicesStable actually works.
2. Provisioning now happens in a goroutine, and provisioners are given 20 minutes to do their work. Should never actually hit that, but it'll keep goroutines from growing unbounded.
3. If there's an error unmarshalling into the properties struct, that error now gets propagated back to the cloudformation ui.

I'll test this in a staging environment tomorrow and make sure that swapping over to Custom::ECSService results in 0 downtime.

[ejholmes]

This is working well in staging. Gonna merge.