[ECS] [CloudFormation]: CloudFormation support for Secrets

[talawahtech]

Tell us about your request
I want to use config values from the Parameter Store in my containers and I want to use CloudFormation to specify the params to be retrieved.

Which service(s) is this request for?
ECS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Even though ECS introduced support specifying values to be retrieved from the parameter store last November, we are not currently able to specify values for the Secrets parameter using CloudFormation.

Are you currently working around this issue?
I am using the aws cli to retrieve the values and put them in environment variables in an entrypoint script.

[abby-fuller]

working with the CloudFormation team to get this (and support for future features) out as soon as we can!

[bushong1]

Is there a ticket I can +1 to consider CloudFormation support a requirement for all new features? This CloudFormation waiting game is so frustrating when it happens with Every. Single. Feature.

[deleugpn]

I don't think that's a reasonable request. Agile at it's best.

[bushong1]

It's no different than requiring aws-cli or boto support at the launch of a new feature.

[sbe-arg]

For new readers... a few handy links here:
Basically trying to implement this https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data.html using a mix of ParameterStore and SecretsManager.

Found that TaskDef API supports it, as per here https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RegisterTaskDefinition.html.

AWS cli seems to support it as per here: https://docs.aws.amazon.com/cli/latest/reference/ecs/register-task-definition.html

But there is no secrets in cloudformation tasks as per here:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-taskdefinition.html#cfn-ecs-taskdefinition-containerdefinition

And we all want this feature...

[tvb]

... Booting up "wait game" .... hang on .....

[tatumbrett1]

+1 Cloudformation support would be great!

Thanks

[sunilkumarmohanty]

+1 this is super critical for us

[ngamradt-turner]

This is still in the "Coming Soon" bucket...is there a rough estimate on when this will get implemented? Will it be this quarter, second quarter?

[diraven]

This is a critical thing to have, really.

[sbe-arg]

Letting everyone know you can easily do as bellow: (CONFIRMED WORKING)

secrets:

  taskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
          Environment:
            - Name: SECRET_NAME
              Value: !Sub 
                - '{{resolve:secretsmanager:arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:${EnvName}:SecretString:SECRET_NAME}}'
                - { EnvName: !Ref "EnvtName" }

parameters (non-secure):

  ParameterName:
    Type: 'AWS::SSM::Parameter::Value<String>'
    Description: SSM parameter name.

[kadrach]

@sbe-arg Doesn't this bake in parameters at deploy-time? The issue is regarding specifying parameter names in CF that would be pulled into ECS at runtime.

[sbe-arg]

@kadrach you are correct. This does deploy time parameters.

[bushong1]

@sbe-arg The whole reason we +1d this thread is because that's not a great way to do it.

[sbe-arg]

To clarify the examples I posted are not a replacement to this request but what currently you can/could implement untill the feature we are all requesting becomes available.

We use this for now on top of RDS IAM so our secrets are no more than mere endpoints or service ids links.

Unsure why I'm under neg feedback. XD

[Exodus]

There's also the additional purpose of not having the secrets exposed. Passing them via the Environment property defeats that purpose as it now becomes visible in the ECS console.

@sbe-arg I think a better solution is to just give the task the iam role with enough permissions to perform an ssm parameter store get operation and as part of the docker run obtain the secret and use it. That way it's not visible. Nonetheless, it's still secret on deploy time.

[bushong1]

@Exodus this is how I've been doing it for a while now and it definitely works, but it's frustrating to have to bloat my image with Python + awscli when it really just needs a feature that has existed in AWS for months.

[Exodus]

@bushong1 my thoughts exactly. You're preaching to the choir.

[gataka]

@bushong1 My workaround is a sidecar-container, which has the aws-cli. This sidecar gets all the parameters from the parameter store and writes it to a config file, which can be mounted from the application container, the benefit is: the application container do not need to know anything about AWS. This sidecar will be unnecessary when the secrets feature is available at CloudFormation.

[rfink]

@gataka I do something pretty similar

[JoseRolles]

In my case, I am already using CloudWatch Logs Agent inside my containers and it comes with a working executable AWS CLI bin (located at /var/awslogs/bin/aws)

So in my container's startup script (using Dockerfile ENTRYPOINT exec /startup_script.sh on container run), I just run the following in a loop to grab each secret I need:

secret_response="$(/var/awslogs/bin/aws ssm get-parameter \
      --name "$ssm_parameter_name" \
      --with-decryption \
      2> >(tee --append /var/log/startup_script.log >&2) \
      )";

Note: The fourth line is using tee to both log STDERR to the console and also to a log file that is picked up by the log agent, while leaving the STDOUT from the CLI call untouched to return from the subshell to be set in the variable $secret_response.

You just have to make sure that the TaskRoleArn property for the AWS::ECS::TaskDefinition resource points to a role with enough permissions to read/decrypt the secret.

[teekennedy]

I have a different workaround approach that uses the AWS CLI to patch in secrets to one or more containerDefinitions in a given TaskDefinition.

The CloudFormation library I'm using (sceptre) allows me to run commands whenever resources are created or updated. Given an AWS profile, TaskDefinition family prefix, service name, cluster name, and a path to a JSON formatted secrets file in the form of a dictionary of container names to secrets references:

{
    "container_name": [
        {
            "name": "SECRET1",
            "valueFrom": "/aws/reference/secretsmanager/SECRET1"
        }
    ]
}

This script uses the AWS CLI to download the latest task definition, add secret references to relevant containers, and then update the service to use the patched task definition.

#!/usr/bin/env bash

USAGE="Usage: $0 aws_profile task_family_prefix service_name cluster_name /path/to/secrets.json"

if [[ $# -ne 5 ]]; then
    echo $USAGE
    exit 1
fi

PROFILE="$1"
FAMILY_PREFIX="$2"
SERVICE_NAME="$3"
CLUSTER_NAME="$4"
SECRETS_JSON_PATH="$5"
TMPDIR=$(mktemp -d)
TASK_DEF_PATH=$TMPDIR/task_def.json
CONTAINER_DEF_PATH=$TMPDIR/container_def.json

# Get the latest task definition for the given family prefix
LATEST_TASK_DEFINITION_ARN=$(aws --profile $PROFILE ecs list-task-definitions \
    --family-prefix $FAMILY_PREFIX --sort DESC --max-items 1 --output json \
    | jq -r '.taskDefinitionArns[0]')

aws --profile $PROFILE ecs describe-task-definition \
    --task-definition $LATEST_TASK_DEFINITION_ARN \
    --output json > $TASK_DEF_PATH

# Get the execution role for the task
EXECUTION_ROLE_ARN=$(jq -r '.taskDefinition.executionRoleArn' $TASK_DEF_PATH)

# Single out containerDefinitions list and patch in secrets from secrets JSON
jq_filter='[. as [$taskdef, $secrets] | $taskdef | '
jq_filter+='.taskDefinition.containerDefinitions[] | '
jq_filter+='if $secrets[.name] then .secrets = $secrets[.name] else . end]'
jq -s "$jq_filter" $TASK_DEF_PATH $SECRETS_JSON_PATH > $CONTAINER_DEF_PATH

# Create new task definition revision and get its ARN
NEW_TASK_DEFINITION_ARN=$(aws --profile $PROFILE ecs register-task-definition \
    --family $FAMILY_PREFIX --execution-role-arn $EXECUTION_ROLE_ARN \
    --container-definitions file://$CONTAINER_DEF_PATH --output json | \
    jq -r '.taskDefinition.taskDefinitionArn')

# Update the service to use the latest task definition
aws --profile $PROFILE ecs update-service --cluster $CLUSTER_NAME \
    --service $SERVICE_NAME --task-definition $NEW_TASK_DEFINITION_ARN

# Cleanup the tmpdir
rm -r $TMPDIR

Hopefully this approach is helpful to others until we get official CloudFormation support.

[dgpalmer]

+1 for cloudformation support

[bushong1]

It's April.... Any updates @abby-fuller?

[nicolai-shape]

@abby-fuller Any news? This is critical for people using CloudFormation to have a convenient and secure way to run tasks.

[patiek]

I needed this for secrets manager support. For what it is worth, and maybe to save others trouble until this feature hits, here is an aws-entrypoint.sh script for realtime replacement of environment variables containing {{realtime-secret:my-secret-value:foo}} with the value of key "foo" from my-secret-value. It also can output json structure itself via {{realtime-secret:my-secret-value}}. I use it as a drop-in entrypoint replacement for my tasks and then my environment variables just become strings like the env variable DB_USERNAME being set to value {{realtime-secret:master-db-secret:username}}.

It supports multiple-replacements too like: postgres://{{runtime-secret:db-secret:username}}:{{runtime-secret:db-secret:password}}@{{runtime-secret:db-secret:address}}/{{runtime-secret:db-secret:dbname}} being translated to postgres://someuser:secretpassword@mydbinstance/appdb at runtime by the linked aws-entrypoint.sh script.

[sbe-arg]

example working in us-east-1

          Secrets:
            - { Name: "DB_NAME", ValueFrom: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:${EnvironmentName}/rds:SecretString:DB_NAME" }
            - { Name: "DB_URL", ValueFrom: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:${EnvironmentName}/rds:SecretString:DB_URL" }
            - { Name: "DB_PORT", ValueFrom: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:${EnvironmentName}/rds:SecretString:DB_PORT" }
            - { Name: "DB_HOST", ValueFrom: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:${EnvironmentName}/rds:SecretString:DB_HOST" }

[yingmu52]

Hi this morning, I notice this feature has been taken down from ca-central-1 ??

any suggestions?

this is the error I got

Encountered unsupported property Secrets

How long do you think we can have this feature fully functioning on production ??

[waltari2001]

example working in us-east-1

          Secrets:
            - { Name: "DB_NAME", ValueFrom: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:${EnvironmentName}/rds:SecretString:DB_NAME" }

So your secret is ${EnvironmentName}/rds and you are retrieving the value of the DB_NAME key in this case? I keep getting ValidationException: Invalid name errors following the same syntax.

[sbe-arg]

Yes now secret string def does not work anymore. And u get a variable with the json payload of all your "secrets". Have now to remove ':SecretString:DB_NAME' I think is a good call to wait for official docimentation.

…

On Thu, May 30, 2019, 04:06 waltari2001 ***@***.***> wrote: example working in us-east-1 Secrets: - { Name: "DB_NAME", ValueFrom: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:${EnvironmentName}/rds:SecretString:DB_NAME" } So your secret is ${EnvironmentName}/rds and you are retrieving the value of the DB_NAME key in this case? I keep getting ValidationException: Invalid name errors. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#97?email_source=notifications&email_token=AFSO3GXZEXBTZYBO43ZKU3DPX2SX7A5CNFSM4GOYJLK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWP2LJI#issuecomment-497001893>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFSO3GSZ7C3RD7HPC2BYFXDPX2SX7ANCNFSM4GOYJLKQ> .

[bgerd]

Cool. Got a SSM Parameter lookup to work for us-east-1. Format was:

Secrets:
    - { Name: "ENV_VAR", ValueFrom: !Sub "arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/PARAMETER_NAME" }

Requires that TaskDefintion's ExecutionRoleArn has policy with ssm:GetParameters action on referenced resources

[arel]

I'm also waiting for CloudFormation support for Secrets. I had planed on using environment variables (like @sbe-arg's suggestion) before I saw this warning from Amazon:

Important

We do not recommend using plaintext environment variables for sensitive information, such as credential data.

Could @coultn or anyone else elaborate on this? Is there a risk of environment variables getting exposed when using Fargate launch type?

[arrtchiu]

Nice find on the correct syntax before it got released - you people have sure got more creativity and patience than I have.

However, this is undefined functionality and as such shouldn’t be relied upon. Amazon may change the syntax required or move it somewhere else. That’s why they haven’t documented it yet - they don’t consider it ready, and it’s therefore subject to change without notice. I mention this for the newcomers who haven’t yet had to deal with fallout of undefined behaviour/functionality. Stick to documented stuff, or beware it breaking! :)

In my case needed this and couldn’t wait for CloudFormation to add it. Luckily my requirements were small enough that swapping to Terraform was practical.

Looking forward to the day it’s fully rolled out!

Is there a risk of environment variables getting exposed

Yes - secrets manager secrets are access- controlled (via KMS if I recall correctly). If you put your secret stuff into environment variables, anyone with access to read the task definition can see the secrets.

We also don’t know how Amazon treats the environment variables that aren’t secrets under the hood. It’s entirely possible they don’t encrypt the disks they’re stored on, etc, since they have created special features specifically for secret stuff.

[jinmingda]

Just want to say thank you for all the comments in this thread. I managed to get it working with the Secrets Manger on a Fargate instance in us-east-1. I needed to create a custom Task Execution Role first with the following CloudFormation template.

  TaskExecutionRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - 'ecs-tasks.amazonaws.com'
          Action:
          - 'sts:AssumeRole'
      ManagedPolicyArns:
      - 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
      Policies:
      - PolicyName: !Sub 'EcsTaskExecutionRole-${AWS::StackName}'
        PolicyDocument:
          Version: 2012-10-17
          Statement:
          - Effect: Allow
            Action:
            - 'secretsmanager:GetSecretValue'
            Resource:
            - 'arn:aws:secretsmanager:us-east-1:<AccountID>:secret:<your secret>'

Then I used the following code in TaskDefinition > ContainerDefinitions > Secrets just like some of you previously suggested:

Secrets:
- { Name: 'SECRET', ValueFrom: 'arn:aws:secretsmanager:us-east-1:<AccountID>:secret:<your secret>' }

It successfully injects an environment variable SECRET to my container with the value (stored as plain text) fetched from the Secrets Manager. Even though the key of the environment variable is still visible in Task Definition, the value is replaced with the ARN. That's exactly what I've been look for!

[ScottBrenner]

Is ECS/Secrets support planned for CDK?

Not seeing it as a supported parameter in the task definition docs, and I can't reference secrets using environment. When I try to create a task definition with something like:

task_def.add_container(
  ...
  environment={
    "fake": ssm.ParameterStoreSecureString(parameter_name="fake", version=1).to_string(),
  }

I get an error message saying

$ cdk deploy
...
❌  cdk-fargate-cdk-1 failed: ValidationError: SSM Secure reference is not supported in: [AWS::ECS::TaskDefinition/Properties/ContainerDefinitions`

[FernandoMiguel]

Try secrets instead of environment

…

-- Fernando Miguel

On Mon, 10 Jun 2019, 06:19 Scott Brenner, ***@***.***> wrote: Is ECS/Secrets support planned for CDK? Not seeing it as a supported parameter in the task definition docs <https://docs.aws.amazon.com/cdk/api/latest/python/aws_cdk.aws_ecs/FargateTaskDefinition.html#aws_cdk.aws_ecs.FargateTaskDefinition.add_container>, and I can't reference secrets using environment. When I try to create a task definition with something like: task_def.add_container( ... environment={ "fake": ssm.ParameterStoreSecureString(parameter_name="fake", version=1).to_string(), } I get an error message saying ❌ cdk-fargate-cdk-1 failed: ValidationError: SSM Secure reference is not supported in: [AWS::ECS::TaskDefinition/Properties/ContainerDefinitions — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#97?email_source=notifications&email_token=AABJDLTLFSD2I6RQB4JBGHTPZXP6LA5CNFSM4GOYJLK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXI6IVY#issuecomment-500294743>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABJDLTHQZ7Y24N66HBCBJ3PZXP6LANCNFSM4GOYJLKQ> .

[ScottBrenner]

@FernandoMiguel Thanks, but secrets is not a valid keyword:
with

task_def.add_container(
  ...
  secrets={
    "fake": ssm.ParameterStoreSecureString(parameter_name="fake", version=1).to_string(),
  }

it says

$ cdk deploy
...
TypeError: add_container() got an unexpected keyword argument 'secrets'

[diraven]

Try Secrets instead of secrets.

[cornerman]

Just trying this out in eu-central-1 with parameter store and it works great 🎉

Though, I have one question: Can I somehow specify the version of a parameter?

I tried parameter/NAME:VERSION but then I get an error:

The Systems Manager parameter name specified for secret POSTGRES_PASSWORD is invalid. The parameter name can be up to 2048 characters and include the following letters and symbols: a-zA-Z0-9_.-,

[coultn]

This is now fully available in all ECS regions!

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/ReleaseHistory.html

[waltari2001]

example working in us-east-1

          Secrets:
            - { Name: "DB_NAME", ValueFrom: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:${EnvironmentName}/rds:SecretString:DB_NAME" }
            - { Name: "DB_URL", ValueFrom: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:${EnvironmentName}/rds:SecretString:DB_URL" }
            - { Name: "DB_PORT", ValueFrom: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:${EnvironmentName}/rds:SecretString:DB_PORT" }
            - { Name: "DB_HOST", ValueFrom: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:${EnvironmentName}/rds:SecretString:DB_HOST" }

Is it me or does the final implementation not support specifying keys in the SecretString?

[mark5cinco]

@waltari2001 Yes, it seems so. I asked AWS support and they did say that this functionality isn't supported. Too bad, I was expecting this one also.

[lukeberry99]

Really would like the ability to specify individual keys. Having to use resolve doesn't work in our case as I have to destroy and recreate the environment to update the variables

[mildebrandt]

Bumping @cornerman 's question. How do we specify the version of a parameter here?

[cornerman]

@mildebrandt I have contacted AWS Support and the response was, you cannot. It will always pull the latest version of the parameter. I was told that a feature request has been opened.

It is really inconvenient, especially because things like {{resolve::ssm}} in cloudformation do the opposite and require a version number. My workaround to version the parameters is now in the name itself, so i have parameters like passwordV1 and passwordV2 :)

[mildebrandt]

@cornerman Thanks for the update!

[tillkolter]

So the feature is not working correctly. Why is this issue closed then?

[coultn]

So the feature is not working correctly. Why is this issue closed then?

This issue was specifically for CloudFormation support for the existing ECS feature. The issue identified with version numbers is not specific to CloudFormation support, but to the underlying ECS feature.

[Kristin0]

👎