Skip to content

Commit

Permalink
feat(fsx): support DataCompressionType in LustreConfiguration (#21392)
Browse files Browse the repository at this point in the history
closes #16431

CloudFormation supports the enablement of transparent LZ4 compression on Lustre filesystems, which helps the user save on consumed space, and therefore cost.  It can also help the user make better use of the provisioned bandwidth.

----

### All Submissions:

* [yes] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [no] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [yes] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [yes] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
  • Loading branch information
tcutts authored Aug 1, 2022
1 parent e2b3e23 commit 214a792
Show file tree
Hide file tree
Showing 8 changed files with 87 additions and 4 deletions.
14 changes: 14 additions & 0 deletions packages/@aws-cdk/aws-fsx/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,20 @@ const fs = new fsx.LustreFileSystem(this, "FsxLustreFileSystem", {
});
```

### Compression

By default, transparent compression of data within FSx for Lustre is switched off. To enable it, add the following to your `lustreConfiguration`:

```ts
const lustreConfiguration = {
// ...
dataCompressionType: fsx.LustreDataCompressionType.LZ4,
// ...
}
```

When you turn data compression on for an existing file system, only newly written files are compressed. Existing files are not compressed. For more information, see [Compressing previously written files](https://docs.aws.amazon.com/fsx/latest/LustreGuide/data-compression.html#migrate-compression).```

## FSx for Windows File Server

The L2 construct for the FSx for Windows File Server has not yet been implemented. To instantiate an FSx for Windows
Expand Down
25 changes: 25 additions & 0 deletions packages/@aws-cdk/aws-fsx/lib/lustre-file-system.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,21 @@ export enum LustreAutoImportPolicy {
NEW_CHANGED_DELETED = 'NEW_CHANGED_DELETED',
}

/**
* The permitted Lustre data compression algorithms
*/
export enum LustreDataCompressionType {
/**
*
* `NONE` - (Default) Data compression is turned off when the file system is created.
*/
NONE = 'NONE',
/**
* `LZ4` - Data compression is turned on with the LZ4 algorithm. Note: When you turn data compression on for an existing file system, only newly written files are compressed. Existing files are not compressed.
*/
LZ4 = 'LZ4',
}

/**
* The configuration for the Amazon FSx for Lustre file system.
*
Expand Down Expand Up @@ -102,6 +117,16 @@ export interface LustreConfiguration {
*/
readonly autoImportPolicy?: LustreAutoImportPolicy;

/**
* Sets the data compression configuration for the file system.
* For more information, see [Lustre data compression](https://docs.aws.amazon.com/fsx/latest/LustreGuide/data-compression.html) in the *Amazon FSx for Lustre User Guide* .
*
* @link http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-fsx-filesystem-lustreconfiguration.html#cfn-fsx-filesystem-lustreconfiguration-datacompressiontype
* @default - no compression
*/

readonly dataCompressionType?: LustreDataCompressionType;

/**
* Required for the PERSISTENT_1 deployment type, describes the amount of read and write throughput for each 1
* tebibyte of storage, in MB/s/TiB. Valid values are 50, 100, 200.
Expand Down
3 changes: 2 additions & 1 deletion packages/@aws-cdk/aws-fsx/test/integ.lustre-file-system.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { AmazonLinuxGeneration, AmazonLinuxImage, Instance, InstanceClass, InstanceSize, InstanceType, SubnetType, Vpc } from '@aws-cdk/aws-ec2';
import { App, RemovalPolicy, Stack } from '@aws-cdk/core';
import { LustreDeploymentType, LustreFileSystem } from '../lib';
import { LustreDeploymentType, LustreFileSystem, LustreDataCompressionType } from '../lib';

const app = new App();

Expand All @@ -11,6 +11,7 @@ const vpc = new Vpc(stack, 'VPC');
const storageCapacity = 1200;
const lustreConfiguration = {
deploymentType: LustreDeploymentType.SCRATCH_2,
dataCompressionType: LustreDataCompressionType.LZ4,
};
const fs = new LustreFileSystem(stack, 'FsxLustreFileSystem', {
lustreConfiguration,
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"version": "20.0.0",
"files": {
"06adf1bf5fb38407db9062bf544139638b8c6ac84c58a3f741f5730bba7509a2": {
"source": {
"path": "AwsCdkFsxLustre.template.json",
"packaging": "file"
},
"destinations": {
"current_account-current_region": {
"bucketName": "cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}",
"objectKey": "06adf1bf5fb38407db9062bf544139638b8c6ac84c58a3f741f5730bba7509a2.json",
"assumeRoleArn": "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cdk-hnb659fds-file-publishing-role-${AWS::AccountId}-${AWS::Region}"
}
}
}
},
"dockerImages": {}
}
Original file line number Diff line number Diff line change
Expand Up @@ -451,6 +451,7 @@
}
],
"LustreConfiguration": {
"DataCompressionType": "LZ4",
"DeploymentType": "SCRATCH_2"
},
"SecurityGroupIds": [
Expand Down
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"version":"17.0.0"}
{"version":"20.0.0"}
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"version": "17.0.0",
"version": "20.0.0",
"artifacts": {
"Tree": {
"type": "cdk:tree",
Expand Down
25 changes: 24 additions & 1 deletion packages/@aws-cdk/aws-fsx/test/lustre-file-system.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import { Template } from '@aws-cdk/assertions';
import { ISubnet, Port, SecurityGroup, Subnet, Vpc } from '@aws-cdk/aws-ec2';
import { Key } from '@aws-cdk/aws-kms';
import { Aws, Stack, Token } from '@aws-cdk/core';
import { LustreConfiguration, LustreDeploymentType, LustreAutoImportPolicy, LustreFileSystem, LustreMaintenanceTime, Weekday } from '../lib';
import { LustreConfiguration, LustreDeploymentType, LustreAutoImportPolicy, LustreFileSystem, LustreMaintenanceTime, Weekday, LustreDataCompressionType } from '../lib';

describe('FSx for Lustre File System', () => {
let lustreConfiguration: LustreConfiguration;
Expand Down Expand Up @@ -479,6 +479,29 @@ describe('FSx for Lustre File System', () => {
});
});

describe('DataCompressionType', () => {
test('dataCompressionType enabled', () => {
lustreConfiguration = {
deploymentType: LustreDeploymentType.SCRATCH_2,
dataCompressionType: LustreDataCompressionType.LZ4,
};

new LustreFileSystem(stack, 'FsxFileSystem', {
lustreConfiguration,
storageCapacityGiB: storageCapacity,
vpc,
vpcSubnet,
});

Template.fromStack(stack).hasResourceProperties('AWS::FSx::FileSystem', {
LustreConfiguration: {
DeploymentType: LustreDeploymentType.SCRATCH_2,
DataCompressionType: LustreDataCompressionType.LZ4,
},
});
});
});

describe('perUnitStorageThroughput', () => {
test.each([
50,
Expand Down

0 comments on commit 214a792

Please sign in to comment.