Enable adding custom user data to ECS cluster

[SoManyHs]

Currently, adding additional user data to an Autoscaling group and adding it to an ECS cluster is not a smooth experience.

The implementation of autoscalinggroup.addUserData() does not correctly process MIME multitype archives. E.g.:

    const asg = new autoscaling.AutoScalingGroup(this, 'MyFleet', {
      instanceType: new ec2.InstanceTypePair(ec2.InstanceClass.M4, ec2.InstanceSize.Large),
      machineImage: new ec2.AmazonLinuxImage(),
      associatePublicIpAddress: true,
      vpc
    });

    const userData = `
Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0

--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"

#cloud-config
cloud_final_modules:
- [scripts-user, always]

--//
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"

#!/bin/bash
/bin/echo "Hello World" >> /tmp/testfile.txt
--//
`
    asg.addUserData(userData);

    cluster.addAutoScalingGroupCapacity(asg);

Results in the following CFN for LaunchConfiguration:

  MyFleetLaunchConfig5D7F9801:
    Type: AWS::AutoScaling::LaunchConfiguration
    Properties:
      ImageId: ami-01e24be29428c15b2
      InstanceType: m4.large
      AssociatePublicIpAddress: true
      IamInstanceProfile:
        Ref: MyFleetInstanceProfile70A58496
      KeyName: hhh-2
      SecurityGroups:
        - Fn::GetAtt:
            - MyFleetInstanceSecurityGroup774E8234
            - GroupId
      UserData:
        Fn::Base64:
          Fn::Join:
            - ""
            - - |-
                #!/bin/bash

                Content-Type: multipart/mixed; boundary="//"
                MIME-Version: 1.0

                --//
                Content-Type: text/cloud-config; charset="us-ascii"
                MIME-Version: 1.0
                Content-Transfer-Encoding: 7bit
                Content-Disposition: attachment; filename="cloud-config.txt"

                #cloud-config
                cloud_final_modules:
                - [scripts-user, always]

                --//
                Content-Type: text/x-shellscript; charset="us-ascii"
                MIME-Version: 1.0
                Content-Transfer-Encoding: 7bit
                Content-Disposition: attachment; filename="userdata.txt"

                #!/bin/bash
                /bin/echo "Hello World" >> /tmp/testfile.txt
                --//

                echo ECS_CLUSTER=
              - Ref: Ec2ClusterEE43E89D
              - >-2
                 >> /etc/ecs/ecs.config
                sudo iptables --insert FORWARD 1 --in-interface docker+ --destination 169.254.169.254/32 --jump DROP

                sudo service iptables save

                echo ECS_AWSVPC_BLOCK_IMDS=true >> /etc/ecs/ecs.config

When deployed, the User Data on the ec2 instance looks like this:

#!/bin/bash

Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0

--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"

#cloud-config
cloud_final_modules:
- [scripts-user, always]

--//
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"

#!/bin/bash
/bin/echo "Hello World" >> /tmp/testfile.txt
--//

echo ECS_CLUSTER=CustomUserData-Ec2ClusterEE43E89D-1H8KFTDM00P66 >> /etc/ecs/ecs.config
sudo iptables --insert FORWARD 1 --in-interface docker+ --destination 169.254.169.254/32 --jump DROP
sudo service iptables save
echo ECS_AWSVPC_BLOCK_IMDS=true >> /etc/ecs/ecs.config

It should look like this:

Content-Type: multipart/mixed; boundary="36b51cd254c2a10606cc4ea1d7c161a960c25b43fbcf3f2275bfea986b64"
MIME-Version: 1.0

--36b51cd254c2a10606cc4ea1d7c161a960c25b43fbcf3f2275bfea986b64
Content-Disposition: attachment; filename="cloud-config.txt"
Content-Transfer-Encoding: 7bit
Content-Type: text/cloud-config; charset="us-ascii"
Mime-Version: 1.0

#cloud-config
cloud_final_modules:
- [scripts-user, always]

--36b51cd254c2a10606cc4ea1d7c161a960c25b43fbcf3f2275bfea986b64
Content-Disposition: attachment; filename="userdata.txt"
Content-Transfer-Encoding: 7bit
Content-Type: text/x-shellscript; charset="us-ascii"
Mime-Version: 1.0

#!/bin/bash
/bin/echo "Hello World" >> /tmp/testfile.txt
--36b51cd254c2a10606cc4ea1d7c161a960c25b43fbcf3f2275bfea986b64
Content-Type: text/text/x-shellscript; charset="utf-8"
Mime-Version: 1.0


#!/bin/bash
echo ECS_CLUSTER=blargh >> /etc/ecs/ecs.config

--36b51cd254c2a10606cc4ea1d7c161a960c25b43fbcf3f2275bfea986b64--

The above was generated using the ECS-cli with custom user data flag:
ecs-cli up --extra-user-data custom-user-data.txt --cluster blargh --capability-iam
Though the ECS CLI is able to leverage some go libraries for MIME multipart archive constructing/unpacking.

It would nice to have a smoother API around adding custom user data -- not sure if that's better served through an integration with a higher-level ECS construct or modifying the existing API in the Autoscaling library.

[rix0rrr]

Related: #623

[SoManyHs]

Related: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/launch_container_instance.html#instance-launch-user-data-step

[rix0rrr]

Sooo... UserData can contain multiple files in a MIME multipart block?

Would generating the same files in the regular userdata as side effect of running the shell script work?

Would generating the same files using CfnInit work?

Adding support for the latter one seems like a better use of time.

[rix0rrr]

Ah:

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/bootstrap_container_instance.html#multi-part_user_data

[SoManyHs]

CfnInit seems like it'd work -- I was coming at it from the point of view of a customer looking at the example custom user data on the docs and copy-pasting to seems like the most reasonable method on ASG. What do you suggest?

[rix0rrr]

Feels like this use case is better served by cfn-init et al.

[otterley]

@rix0rrr cfn-init is arguably the way to go, but CDK will need a mechanism to add metadata in the AWS::CloudFormation::Init namespace to a resource such as a Launch Configuration or Launch Template.

See https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html for details.

[otterley]

Related: #777

[marcb]

I'm having the same issue as @SoManyHs in that I need to pass in a multipart mime document (and even base64 encoded gzipped userdata).

cfn-init isn't functional enough for some of the use cases I am working with.

It isn't yet clear to me if there is some way to work around this when synthesising.

[rix0rrr]

Custom user data generously supplied by @hoegertn should give you a way to achieve what you need:

#4193

@marcb still interested to hear in what you need to achieve that cfn-init can not cover in the same way that userdata would?

[nmussy]

Looking at the quantity of ecs.config fields, it might also be interesting to add an interface to map them out once user date is supported.

I can open another issue if this is outside the scope of this one.

[marcb]

@rix0rrr Cloud-Init offers declarative capabilities / stages such as at instance instantiation, at boot, per instance (picked up from AMI), once, etc. Also provides some measure of OS and Cloud independence which some customers want. e.g. common security hardening policies.

That said in this instance I did refactor all the cloud-init into cfn-init bash with some packaging python code embedded in CDK - fetching logic, base64 encoding and decoding etc… as needed.

CDK has simplified the environment build a lot and the team is enjoying it - even if most have to utilised it on Windows 10… ¯\_(ツ)_/¯

[marcb]

Looking at the quantity of ecs.config fields, it might also be interesting to add an interface to map them out once user date is supported.

I can open another issue if this is outside the scope of this one.

I'd be interested in managing those within CDK...

Currently we populate /etc/ecs/ecs.config via cfn-init.

e.g.

cat <<'EOF' >> /etc/ecs/ecs.config
ECS_LOGLEVEL=debug
EOF

[SoManyHs]

This would possibly affect the implementation of custom attributes on instances: #1772

[rsmogura]

Sorry for jumping in, but maybe this can be closed in context of #8315 & #11843 ?

[namedgraph]

Is there a workaround here? I'm trying

Cluster ec2Cluster = Cluster.Builder.create(this, "EC2Cluster").
    vpc(vpc).
    build();
ec2Cluster.getAutoscalingGroup().addUserData("echo \"$PARAM\"");

but getting an NPE because getAutoscalingGroup() returns null :/

[kristianmandrup]

@namedgraph Where did you find the getAutoscalingGroup() method? I'm also trying to figure out how to access the autoscaling group from an EC2 cluster or adding user data to ECS.

[namedgraph]

@kristianmandrup https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-ecs.Cluster.html#properties

[kristianmandrup]

But there is no get method, only a property autoScalingGroup, why you are getting null

[rsmogura]

@kristianmandrup,
This property is a getter (it's a function), it can give undefined value if the capacity has not been defined in props, during creation of cluster (that's one way of defining capacity).

If you use method addCapacity to add a capacity (instead of props), then the result value is an autoscaling group (cluster can be associated with multiple ASG) and you can use it to operate on user data.

As a side note, there's new set of changes enabling capacity providers, AFIK.

[github-actions]

This issue has not received any attention in 1 year. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.