Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ec2): multipart user data #11843

Merged
merged 7 commits into from
Mar 8, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions packages/@aws-cdk/aws-ec2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -981,6 +981,51 @@ instance.userData.addExecuteFileCommand({
asset.grantRead( instance.role );
```

### Multipart user data

In addition, to above the `MultipartUserData` can be used to change instance startup behavior. Multipart user data are composed
from separate parts forming archive. The most common parts are scripts executed during instance set-up. However, there are other
kinds, too.

The advantage of multipart archive is in flexibility when it's needed to add additional parts or to use specialized parts to
fine tune instance startup. Some services (like AWS Batch) supports only `MultipartUserData`.

The parts can be executed at different moment of instance start-up and can serve a different purposes. This is controlled by `contentType` property.
For common scripts, `text/x-shellscript; charset="utf-8"` can be used as content type.

In order to create archive the `MultipartUserData` has to be instantiated. Than, user can add parts to multipart archive using `addPart`. The `MultipartBody` contains methods supporting creation of body parts.

If the very custom part is required, it can be created using `MultipartUserData.fromRawBody`, in this case full control over content type,
transfer encoding, and body properties is given to the user.

Below is an example for creating multipart user data with single body part responsible for installing `awscli` and configuring maximum size
of storage used by Docker containers:

```ts
const bootHookConf = ec2.UserData.forLinux();
bootHookConf.addCommands('cloud-init-per once docker_options echo \'OPTIONS="${OPTIONS} --storage-opt dm.basesize=40G"\' >> /etc/sysconfig/docker');

const setupCommands = ec2.UserData.forLinux();
setupCommands.addCommands('sudo yum install awscli && echo Packages installed らと > /var/tmp/setup');

const multipartUserData = new ec2.MultipartUserData();
// The docker has to be configured at early stage, so content type is overridden to boothook
multipartUserData.addPart(ec2.MultipartBody.fromUserData(bootHookConf, 'text/cloud-boothook; charset="us-ascii"'));
// Execute the rest of setup
multipartUserData.addPart(ec2.MultipartBody.fromUserData(setupCommands));

new ec2.LaunchTemplate(stack, '', {
userData: multipartUserData,
blockDevices: [
// Block device configuration rest
]
});
```

For more information see
[Specifying Multiple User Data Blocks Using a MIME Multi Part Archive](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/bootstrap_container_instance.html#multi-part_user_data)


## Importing existing subnet

To import an existing Subnet, call `Subnet.fromSubnetAttributes()` or
Expand Down
256 changes: 255 additions & 1 deletion packages/@aws-cdk/aws-ec2/lib/user-data.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { IBucket } from '@aws-cdk/aws-s3';
import { CfnElement, Resource, Stack } from '@aws-cdk/core';
import { CfnElement, Fn, Resource, Stack } from '@aws-cdk/core';
import { OperatingSystemType } from './machine-image';

/**
Expand Down Expand Up @@ -276,3 +276,257 @@ class CustomUserData extends UserData {
throw new Error('CustomUserData does not support addSignalOnExitCommand, use UserData.forLinux() or UserData.forWindows() instead.');
}
}

/**
* Options when creating `MultipartBody`.
*/
export interface MultipartBodyOptions {

/**
* `Content-Type` header of this part.
*
* Some examples of content types:
* * `text/x-shellscript; charset="utf-8"` (shell script)
* * `text/cloud-boothook; charset="utf-8"` (shell script executed during boot phase)
*
* For Linux shell scripts use `text/x-shellscript`.
*/
readonly contentType: string;

/**
* `Content-Transfer-Encoding` header specifying part encoding.
*
* @default undefined - body is not encoded
*/
readonly transferEncoding?: string;

/**
* The body of message.
*
* @default undefined - body will not be added to part
*/
readonly body?: string,
}

/**
* The base class for all classes which can be used as {@link MultipartUserData}.
*/
export abstract class MultipartBody {
/**
* Content type for shell scripts
*/
public static readonly SHELL_SCRIPT = 'text/x-shellscript; charset="utf-8"';

/**
* Content type for boot hooks
*/
public static readonly CLOUD_BOOTHOOK = 'text/cloud-boothook; charset="utf-8"';

/**
* Constructs the new `MultipartBody` wrapping existing `UserData`. Modification to `UserData` are reflected
* in subsequent renders of the part.
*
* For more information about content types see {@link MultipartBodyOptions.contentType}.
*
* @param userData user data to wrap into body part
* @param contentType optional content type, if default one should not be used
*/
public static fromUserData(userData: UserData, contentType?: string): MultipartBody {
return new MultipartBodyUserDataWrapper(userData, contentType);
}

/**
* Constructs the raw `MultipartBody` using specified body, content type and transfer encoding.
*
* When transfer encoding is specified (typically as Base64), it's caller responsibility to convert body to
* Base64 either by wrapping with `Fn.base64` or by converting it by other converters.
*/
public static fromRawBody(opts: MultipartBodyOptions): MultipartBody {
return new MultipartBodyRaw(opts);
}

public constructor() {
}

/**
* Render body part as the string.
*
* Subclasses should not add leading nor trailing new line characters (\r \n)
*/
public abstract renderBodyPart(): string[];
}

/**
* The raw part of multi-part user data, which can be added to {@link MultipartUserData}.
*/
class MultipartBodyRaw extends MultipartBody {
public constructor(private readonly props: MultipartBodyOptions) {
super();
}

/**
* Render body part as the string.
*/
public renderBodyPart(): string[] {
const result: string[] = [];

result.push(`Content-Type: ${this.props.contentType}`);

if (this.props.transferEncoding != null) {
result.push(`Content-Transfer-Encoding: ${this.props.transferEncoding}`);
}
// One line free after separator
result.push('');

if (this.props.body != null) {
result.push(this.props.body);
// The new line added after join will be consumed by encapsulating or closing boundary
}

return result;
}
}

/**
* Wrapper for `UserData`.
*/
class MultipartBodyUserDataWrapper extends MultipartBody {
private readonly contentType: string;

public constructor(private readonly userData: UserData, contentType?: string) {
super();

this.contentType = contentType || MultipartBody.SHELL_SCRIPT;
}

/**
* Render body part as the string.
*/
public renderBodyPart(): string[] {
const result: string[] = [];

result.push(`Content-Type: ${this.contentType}`);
result.push('Content-Transfer-Encoding: base64');
result.push('');
result.push(Fn.base64(this.userData.render()));

return result;
}
}

/**
* Options for creating {@link MultipartUserData}
*/
export interface MultipartUserDataOptions {
/**
* The string used to separate parts in multipart user data archive (it's like MIME boundary).
*
* This string should contain [a-zA-Z0-9()+,-./:=?] characters only, and should not be present in any part, or in text content of archive.
*
* @default `+AWS+CDK+User+Data+Separator==`
*/
readonly partsSeparator?: string;
}

/**
* Mime multipart user data.
*
* This class represents MIME multipart user data, as described in.
* [Specifying Multiple User Data Blocks Using a MIME Multi Part Archive](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/bootstrap_container_instance.html#multi-part_user_data)
*
*/
export class MultipartUserData extends UserData {
private static readonly USE_PART_ERROR = 'MultipartUserData does not support this operation. Please add part using addPart.';
rsmogura marked this conversation as resolved.
Show resolved Hide resolved
private static readonly BOUNDRY_PATTERN = '[^a-zA-Z0-9()+,-./:=?]';

private parts: MultipartBody[] = [];

private opts: MultipartUserDataOptions;

constructor(opts?: MultipartUserDataOptions) {
super();

let partsSeparator: string;

// Validate separator
if (opts?.partsSeparator != null) {
if (new RegExp(MultipartUserData.BOUNDRY_PATTERN).test(opts!.partsSeparator)) {
throw new Error(`Invalid characters in separator. Separator has to match pattern ${MultipartUserData.BOUNDRY_PATTERN}`);
} else {
partsSeparator = opts!.partsSeparator;
}
} else {
partsSeparator = '+AWS+CDK+User+Data+Separator==';
}

this.opts = {
partsSeparator: partsSeparator,
};
}

/**
* Adds a part to the list of parts.
*/
public addPart(part: MultipartBody) {
this.parts.push(part);
}

/**
* Adds a multipart part based on a UserData object
*
* This is the same as calling:
*
* ```ts
* multiPart.addPart(MultipartBody.fromUserData(userData, contentType));
* ```
*/
public addUserDataPart(userData: UserData, contentType?: string) {
this.addPart(MultipartBody.fromUserData(userData, contentType));
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why get rid of the addUserDataPart(userData, contentType) here? I rather liked it as a convenience method.

(Of course it's not strictly necessary, but it reads nicer than the current alternative)

public render(): string {
const boundary = this.opts.partsSeparator;
// Now build final MIME archive - there are few changes from MIME message which are accepted by cloud-init:
// - MIME RFC uses CRLF to separate lines - cloud-init is fine with LF \n only
// Note: new lines matters, matters a lot.
var resultArchive = new Array<string>();
resultArchive.push(`Content-Type: multipart/mixed; boundary="${boundary}"`);
resultArchive.push('MIME-Version: 1.0');

// Add new line, the next one will be boundary (encapsulating or closing)
// so this line will count into it.
resultArchive.push('');

// Add parts - each part starts with boundary
this.parts.forEach(part => {
resultArchive.push(`--${boundary}`);
resultArchive.push(...part.renderBodyPart());
});

// Add closing boundary
resultArchive.push(`--${boundary}--`);
resultArchive.push(''); // Force new line at the end

return resultArchive.join('\n');
}

public addS3DownloadCommand(_params: S3DownloadOptions): string {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rsmogura @rix0rrr L513-L532 Are these deprecated now? If they are, can we use @deprecated docstrings?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think now I understand why addCommand was not doing anything :) https://cdk-dev.slack.com/archives/C018XT6REKT/p1615356365150500

cc @fogfish

Copy link
Contributor Author

@rsmogura rsmogura Mar 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @robertd - I think it's not deprecated. You still can use other script parts as the body of archive.

However with multiply you may try to use x-include-url Content Type

https://cloudinit.readthedocs.io/en/latest/topics/format.html#include-file

And I'm glad you did find an answer for your question.

throw new Error(MultipartUserData.USE_PART_ERROR);
}

public addExecuteFileCommand(_params: ExecuteFileOptions): void {
throw new Error(MultipartUserData.USE_PART_ERROR);
}

public addSignalOnExitCommand(_resource: Resource): void {
throw new Error(MultipartUserData.USE_PART_ERROR);
}

public addCommands(..._commands: string[]): void {
throw new Error(MultipartUserData.USE_PART_ERROR);
}

public addOnExitCommands(..._commands: string[]): void {
throw new Error(MultipartUserData.USE_PART_ERROR);
}
}
Loading