Pass CloudFormation tags/parameters from file

[taraspos]

In order to use awless as part of CI job for deployment CloudFormation, would be very nice to be able to apply parameters from the file.

AWS has this kind of file for deployment with CodePipeline: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-delivery-codepipeline-cfn-artifacts.html#w2ab2c13c15c15. Would be very nice to be able to support the same file format.

I took a look at the update stack code, and everything is autogenerated there and I'm not sure how to add functionality there.

[taraspos]

@fxaguessy
@simcap

any feedback on this one? I would really like to use it in the CI. I could work on the implementation, but don't really know how to add it :)

[simcap]

To be honest I did not get it the first time I read it and I wanted to come back to it.

I'm giving it another read.

If you do not mind you could explain the workflow with your CI, with the ideal awless commands that would help.

[simcap]

If I understand correctly, in that template stack configuration file we have the Parameters section and the StackPolicy which represent the same as the update stack: parameters, policy-file (not sure though for the latter).

So we could introduce a new parameter to do update stack stackfile=/home/user/mystack.json which would slurp the Parameters and the StackPolicy and use them.

It would means we would have to slurp files in both json and yaml.

Or am I completely wrong ?

[simcap]

Ok, I see there is the Stack Template File and the Template Configuration File.

Through create stack and update stack we already support the Stack Template File through the param template-file (see awless create stack -h), but you would like awless to support the Template Configuration file through a param that could be for instance configuration-file. If present we would slurp and use the inner StackPolicy and Parameters

[taraspos]

@simcap yes, you understand it correctly.
I would like to have param stackfile (or something similar), which could be used instead of parameters, stack-policy and tags.

This way I can have 1 template file and couple stack files (for each env), because currently, I have to place dozens of parameters in the CLI command and it is very hard to read and support.

[simcap]

Ok. We shall implement that (both yaml and json support). Not sure when though. Maybe in the 0.1.5, seems it is quite useful indeed for pipelines and CI

[taraspos]

Any update on this one?

Just one more thing came to my mind, is the ability to combine stackfile and parameters.

Use case:
For example, we have CloudFormation stack with 20 parameters, which more or less static, so they can be placed in the stackfile and one parameter which change each deployment, DockerImageVersion for example, which is generated by CI.

So it would be nice to be able to specify awless update stack name=myStack stackfile=myStackFile.yml parameters=DockerImageVersion:v22

To avoid using sed to replace value for DockerImageVersion:v22 in the stackfile.

[taraspos]

By the way, if you give me a hint how to add functionality to the generated code I could work on it if I have time next week.

[fxaguessy]

Thanks for the additional feedback (and the detailed use case).
We are currently refactoring the driver component of awless, in order to be able to add such kind of feature much easily. This should be available in a few days and will allow to add manual "hooks" in the generated code.
We will tell you here when it is available, in order to implement the stackfile parameter.

[taraspos]

@fxaguessy I guess after the 0.1.6 release it should be possible to implement this, right?

Do you have some plan to work on it, or better for me to take it once I have a free hour?

[simcap]

@trane9991 yes, it will be now easier to implement that thanks to our latest re-modelling of AWS integration.

At the moment we are working on the 0.1.7 (due this week) and maybe the 0.1.8 next week for AWS re:Invent since the team will be there to present the tool.

So we will be a bit busy. I might have some time next week to have a look. Since we have to support multi format, this is not an quick feature, and it should be done properly.

We update in this issue accordingly.

[taraspos]

@simcap good luck on the re:Invent 🥇 , I wish I could go there too :D

[taraspos]

@simcap @fxaguessy usage example as promised:

So, mine use case of awless is mostly CI pipeline for deployment of CloudFormation templates.

So basically each of my projects has the template-file - <service>.template.yml and stack-file for each environment <service>.config.yml. Then I have a "script" to run the actual deployments.
Example for CloudFormation template which handles deployment of ECS service:

#!/bin/bash

TIMEOUT=${TIMEOUT:-10m}

if awless update stack --no-sync --force \
    capabilities=CAPABILITY_IAM \
    name=$STACK_NAME \
    template-file=$TEMPLATE_FILE \
    stack-file=$STACK_FILE \
    parameters=ImageTag:$VERSION_NAME
then
    awless tail stack-events \
        --follow $STACK_NAME \
        --filters=id \
        --filters=type \
        --filters=status \
        --timeout $TIMEOUT \
        --cancel-on-timeout
fi

in the stack-file I keep the static CloudFormation template parameters and tags, docker image version tag I prefer to supply via parameters because it is reflecting build number.
Stack-file looks just like in the aws docs, but in YAML.

By the way, I think that awless tail stack-events could be implemented as a part of awless update stack command with the flag --wait or --follow, this way I wouldn't need the whole if; then; fi thing.

--timeout is needed to rollback the deployment if it takes too long. Thing is, ECS can't detect if deployment is broken and will keep trying restart docker image with broken app version up to 3 hours (not sure if the number is correct, but I read it somewhere on the forums).

I think next step here is to add support of rollback-triggers (#157) to make deployment rollback smarter than just timeout.

[simcap]

Thanks a lot for the info!