Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(lambda): efs filesystems #8602

Merged
merged 63 commits into from
Jul 6, 2020
Merged
Show file tree
Hide file tree
Changes from 58 commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
25762f4
init support
pahud Jun 17, 2020
16e1311
use CfnAccessPoint
pahud Jun 17, 2020
cac456f
add integ
pahud Jun 17, 2020
5b76922
remove comments
pahud Jun 17, 2020
5f85413
update integ
pahud Jun 17, 2020
0b56916
change permisson to 755
pahud Jun 17, 2020
484caba
Update packages/@aws-cdk/aws-lambda/lib/function.ts
pahud Jun 18, 2020
e77d4a6
fix
pahud Jun 18, 2020
1b27ade
Merge branch 'lambdafs-efs' of github.com:pahud/aws-cdk into lambdafs…
pahud Jun 18, 2020
4deb43d
fix
pahud Jun 18, 2020
3a1e8fd
Merge branch 'master' into lambdafs-efs
pahud Jun 23, 2020
53492f0
- use AccessPoint construct
pahud Jun 23, 2020
64af74f
minor fix
pahud Jun 23, 2020
c534ff0
fix
pahud Jun 24, 2020
b3c503a
fix
pahud Jun 24, 2020
73d3d22
add integ test and update dependency
pahud Jun 24, 2020
e756770
update README
pahud Jun 24, 2020
0bdbca6
Update packages/@aws-cdk/aws-lambda/README.md
pahud Jun 24, 2020
1687560
update sample in the README
pahud Jun 24, 2020
f1d25b6
fix
pahud Jun 24, 2020
52d1f2d
minor fix
pahud Jun 24, 2020
aec9b6e
fix package.json
pahud Jun 24, 2020
c43d6f0
update integ test
pahud Jun 24, 2020
7b87dd4
Merge branch 'master' into lambdafs-efs
pahud Jun 26, 2020
035affa
simplify the filesystem configuration class
pahud Jun 27, 2020
2e8fcde
update README
pahud Jun 27, 2020
c3b5aee
fix tests
pahud Jun 27, 2020
3f9f709
minor fix
pahud Jun 27, 2020
6e22e90
fix
pahud Jun 29, 2020
3a5cd81
fix
pahud Jun 29, 2020
7de5399
fix
pahud Jun 30, 2020
0595cd9
add docstring
pahud Jun 30, 2020
46065b3
add securityGroups
pahud Jun 30, 2020
1e5afa1
Merge branch 'master' into lambdafs-efs
pahud Jun 30, 2020
93fb996
update README
pahud Jun 30, 2020
07435dd
update package.jsonA
pahud Jun 30, 2020
7d855f2
Merge branch 'master' into lambdafs-efs
pahud Jun 30, 2020
d246979
update README and integ test
pahud Jul 1, 2020
460ff7a
add doc string
pahud Jul 1, 2020
764210d
minor
pahud Jul 1, 2020
3298ce7
update integ test
pahud Jul 1, 2020
740dcd8
Merge branch 'master' into lambdafs-efs
pahud Jul 1, 2020
94b64c1
Merge branch 'master' into lambdafs-efs
pahud Jul 1, 2020
4ce6567
refactor the permission settiong
pahud Jul 1, 2020
1bea0cf
Merge branch 'lambdafs-efs' of https://github.com/pahud/aws-cdk into …
pahud Jul 1, 2020
d37cead
minor
pahud Jul 1, 2020
c9c03a0
minor
pahud Jul 1, 2020
fa1679a
revert
pahud Jul 1, 2020
0065bfa
Merge branch 'master' into lambdafs-efs
pahud Jul 1, 2020
2114987
revert
pahud Jul 1, 2020
7fb7ac8
fix
pahud Jul 2, 2020
f1bd125
fix
pahud Jul 2, 2020
db34025
rename filesystem to fileSystem
pahud Jul 2, 2020
44cb692
Merge branch 'master' into lambdafs-efs
pahud Jul 2, 2020
f1fb5fa
fix yarn.lock
pahud Jul 2, 2020
3cb9b0d
update package.json
pahud Jul 2, 2020
2845c6d
Update packages/@aws-cdk/aws-lambda/lib/filesystem.ts
Jul 6, 2020
f83a571
Update packages/@aws-cdk/aws-lambda/lib/filesystem.ts
Jul 6, 2020
956f2b0
fix
pahud Jul 6, 2020
2434b57
fix README
pahud Jul 6, 2020
f93c45e
fix README
pahud Jul 6, 2020
4e553d9
Merge branch 'master' into lambdafs-efs
mergify[bot] Jul 6, 2020
20563a3
Merge branch 'master' into lambdafs-efs
mergify[bot] Jul 6, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions packages/@aws-cdk/aws-efs/lib/access-point.ts
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,11 @@ export class AccessPoint extends Resource implements IAccessPoint {
*/
public readonly accessPointId: string;

/**
* The filesystem of the access point
*/
public readonly fileSystem: IFileSystem;

constructor(scope: Construct, id: string, props: AccessPointProps) {
super(scope, id);

Expand All @@ -165,5 +170,6 @@ export class AccessPoint extends Resource implements IAccessPoint {
resource: 'access-point',
resourceName: this.accessPointId,
});
this.fileSystem = props.fileSystem;
}
}
18 changes: 16 additions & 2 deletions packages/@aws-cdk/aws-efs/lib/efs-file-system.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import * as ec2 from '@aws-cdk/aws-ec2';
import * as kms from '@aws-cdk/aws-kms';
import { Construct, IResource, RemovalPolicy, Resource, Size, Tag } from '@aws-cdk/core';
import { ConcreteDependable, Construct, IDependable, IResource, RemovalPolicy, Resource, Size, Tag } from '@aws-cdk/core';
import { AccessPoint, AccessPointOptions } from './access-point';
import { CfnFileSystem, CfnMountTarget } from './efs.generated';

Expand Down Expand Up @@ -83,6 +83,12 @@ export interface IFileSystem extends ec2.IConnectable, IResource {
* @attribute
*/
readonly fileSystemId: string;

/**
* Dependable that can be depended upon to ensure the mount targets of the filesystem are ready
*/
readonly mountTargetsAvailable: IDependable;

}

/**
Expand Down Expand Up @@ -205,6 +211,7 @@ export class FileSystem extends Resource implements IFileSystem {
securityGroups: [attrs.securityGroup],
defaultPort: ec2.Port.tcp(FileSystem.DEFAULT_PORT),
});
public readonly mountTargetsAvailable = new ConcreteDependable();
}

return new Import(scope, id);
Expand All @@ -225,6 +232,10 @@ export class FileSystem extends Resource implements IFileSystem {
*/
public readonly fileSystemId: string;

public readonly mountTargetsAvailable: IDependable;

private readonly _mountTargetsAvailable = new ConcreteDependable();

/**
* Constructor for creating a new EFS FileSystem.
*/
Expand Down Expand Up @@ -263,15 +274,18 @@ export class FileSystem extends Resource implements IFileSystem {

// We now have to create the mount target for each of the mentioned subnet
let mountTargetCount = 0;
this.mountTargetsAvailable = [];
subnets.subnetIds.forEach((subnetId: string) => {
new CfnMountTarget(this,
const mountTarget = new CfnMountTarget(this,
'EfsMountTarget' + (++mountTargetCount),
{
fileSystemId: this.fileSystemId,
securityGroups: Array.of(securityGroup.securityGroupId),
subnetId,
});
this._mountTargetsAvailable.add(mountTarget);
});
this.mountTargetsAvailable = this._mountTargetsAvailable;
}

/**
Expand Down
42 changes: 42 additions & 0 deletions packages/@aws-cdk/aws-lambda/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,48 @@ correct log retention period (never expire, by default).
*Further note* that, if the log group already exists and the `logRetention` is not set, the custom resource will reset
the log retention to never expire even if it was configured with a different value.

### FileSystem Access

You can configure a function to mount an Amazon Elastic File System (Amazon EFS) to a
directory in your runtime environment with the `filesystems` property. To access Amaozn EFS
from lambda function, the Amazon EFS access point will be required.

The following sample allows the lambda function to mount the Amazon EFS access point to `/mnt/msg` in the runtime environment and access the filesystem with the POSIX identity defined in `posixUser`.

```ts
// create a new Amaozn EFS filesystem
const fileSystem = new efs.FileSystem(stack, 'Efs', { vpc });

// create a new access point from the filesystem
const accessPoint = fileSystem.addAccessPoint('AccessPoint', {
// set /export/lambda as the root of the access point
path: '/export/lambda',
// as /export/lambda does not exist in a new efs filesystem, the efs will create the directory with the following createAcl
createAcl: {
eladb marked this conversation as resolved.
Show resolved Hide resolved
ownerUid: '1001',
ownerGid: '1001',
permissions: '750',
},
// enforce the POSIX identity so lambda function will access with this identity
posixUser: {
uid: '1001',
gid: '1001',
},
});

const fn = new lambda.Function(stack, 'MyLambda', {
code,
handler,
runtime,
vpc,
filesystems: [
// mount the access point to /mnt/msg in the lambda runtime enironment
lambda.FileSystem.fromEfsAccessPoint(accessPoint, '/mnt/msg'),
pahud marked this conversation as resolved.
Show resolved Hide resolved
],
});
```


### Singleton Function

The `SingletonFunction` construct is a way to guarantee that a lambda function will be guaranteed to be part of the stack,
Expand Down
68 changes: 68 additions & 0 deletions packages/@aws-cdk/aws-lambda/lib/filesystem.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
import { Connections } from '@aws-cdk/aws-ec2';
import * as efs from '@aws-cdk/aws-efs';
import { IDependable } from '@aws-cdk/core';

/**
* FileSystem configurations for the Lambda function
* @experimental
*/
export interface FileSystemConfig {
pahud marked this conversation as resolved.
Show resolved Hide resolved
/**
* mount path in the lambda runtime environment
*/
readonly localMountPath: string;

/**
* ARN of the access point
*/
readonly arn: string;

/**
* array of IDependable that lambda function depends on
*
* @default - no dependency
*/
readonly dependency?: IDependable[]

/**
* connections object used to allow ingress traffic from lambda function
*
* @default - no connections required to add extra ingress rules for Lambda function
*/
readonly connections?: Connections;

/**
* additional managed policies required for the lambda function
*
* @default - no additional policies required
*/
readonly managedPolicies?: string[]
}

/**
* Represents the filesystem for the Lambda function
* @experimental
*/
export class FileSystem {
/**
* mount the filesystem from Amazon EFS
* @param ap the Amazon EFS access point
* @param mountPath the target path in the lambda runtime environment
*/
public static fromEfsAccessPoint(ap: efs.AccessPoint, mountPath: string): FileSystem {
return new FileSystem({
localMountPath: mountPath,
arn: ap.accessPointArn,
dependency: [ ap.fileSystem.mountTargetsAvailable ],
connections: ap.fileSystem.connections,
managedPolicies: [
'AmazonElasticFileSystemClientFullAccess',
],
});
}

/**
* @param config the FileSystem configurations for the Lambda function
*/
protected constructor(public readonly config: FileSystemConfig) { }
}
58 changes: 56 additions & 2 deletions packages/@aws-cdk/aws-lambda/lib/function.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import { CfnResource, Construct, Duration, Fn, Lazy, Stack } from '@aws-cdk/core
import { Code, CodeConfig } from './code';
import { EventInvokeConfigOptions } from './event-invoke-config';
import { IEventSource } from './event-source';
import { FileSystem } from './filesystem';
import { FunctionAttributes, FunctionBase, IFunction } from './function-base';
import { calculateFunctionHash, trimFromStart } from './function-hash';
import { Version, VersionOptions } from './lambda-version';
Expand Down Expand Up @@ -274,6 +275,13 @@ export interface FunctionProps extends FunctionOptions {
* the handler.
*/
readonly handler: string;

/**
* The filesystem configuration for the lambda function
*
* @default - will not mount any filesystem
*/
readonly filesystems?: FileSystem[];
}

/**
Expand Down Expand Up @@ -460,6 +468,8 @@ export class Function extends FunctionBase {

protected readonly canCreatePermissions = true;

protected readonly filesystems: FileSystem[] = [];

private readonly layers: ILayerVersion[] = [];

private _logGroup?: logs.ILogGroup;
Expand Down Expand Up @@ -489,6 +499,21 @@ export class Function extends FunctionBase {
managedPolicies.push(iam.ManagedPolicy.fromAwsManagedPolicyName('service-role/AWSLambdaVPCAccessExecutionRole'));
}

// add additonal managed policies when necessary
if (props.filesystems) {
const pushedPolicy: string[] = [];
for (const fs of props.filesystems) {
if (fs.config.managedPolicies) {
pahud marked this conversation as resolved.
Show resolved Hide resolved
fs.config.managedPolicies.forEach(p => {
if (pushedPolicy.indexOf(p) === -1) {
managedPolicies.push(iam.ManagedPolicy.fromAwsManagedPolicyName(p));
pushedPolicy.push(p);
}
});
}
}
}

this.role = props.role || new iam.Role(this, 'ServiceRole', {
assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com'),
managedPolicies,
Expand Down Expand Up @@ -570,6 +595,26 @@ export class Function extends FunctionBase {
}

this.currentVersionOptions = props.currentVersionOptions;

if (props.filesystems) {
// max 1 filesystem allowed
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is unnecessary future proofing.. If only a single file system is allowed, then why do we need the property to be filesystems? Shall we change it to filesystem? In the future if we want to add additional filesystems, we can either deprecate the singular property or add additionalFileSystems.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Let me revise it.

Copy link
Contributor Author

@pahud pahud Jul 6, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @eladb

It's filesystem now.

And, to even scope down the required iam policies according to the doc, I changed managedPolicies to policies and bake the required policy statements like this in the fromEfsAccessPoint

      policies: [
        new iam.PolicyStatement({
          actions: [ 'elasticfilesystem:ClientMount' ],
          resources: [ '*' ],
          conditions: {
            StringEquals: {
              'elasticfilesystem:AccessPointArn': ap.accessPointArn,
            },
          },
        }),
        new iam.PolicyStatement({
          actions: ['elasticfilesystem:ClientWrite'],
          resources: [ Stack.of(ap).formatArn({
            service: 'elasticfilesystem',
            resource: 'file-system',
            resourceName: ap.fileSystem.fileSystemId,
          }) ],
        }),
      ],

Let me know if it looks better now.

if (props.filesystems.length > 1) {
throw new Error('max 1 filesystem allowed');
}

for (const fs of props.filesystems) {
// add dependency when necessary
if (fs.config.dependency) {
this.node.addDependency(...fs.config.dependency);
}
}

resource.addPropertyOverride('FileSystemConfigs',
pahud marked this conversation as resolved.
Show resolved Hide resolved
props.filesystems.map(fs => ({
LocalMountPath: fs.config.localMountPath,
Arn: fs.config.arn,
})));
}
}

/**
Expand Down Expand Up @@ -701,7 +746,7 @@ export class Function extends FunctionBase {
// sort environment so the hash of the function used to create
// `currentVersion` is not affected by key order (this is how lambda does
// it).
const variables: { [key: string]: string } = { };
const variables: { [key: string]: string } = {};
for (const key of Object.keys(this.environment).sort()) {
variables[key] = this.environment[key];
}
Expand Down Expand Up @@ -745,6 +790,15 @@ export class Function extends FunctionBase {

this._connections = new ec2.Connections({ securityGroups });

if (props.filesystems) {
for (const fs of props.filesystems) {
// add an ingress rule from the filesystem security group to allow ingress traffic from the lambda function
if (fs.config.connections) {
fs.config.connections.allowDefaultPortFrom(this);
}
}
}

// Pick subnets, make sure they're not Public. Routing through an IGW
// won't work because the ENIs don't get a Public IP.
// Why are we not simply forcing vpcSubnets? Because you might still be choosing
Expand Down Expand Up @@ -841,4 +895,4 @@ export function verifyCodeConfig(code: CodeConfig, runtime: Runtime) {
if (code.inlineCode && !runtime.supportsInlineCode) {
throw new Error(`Inline source not allowed for ${runtime.name}`);
}
}
}
1 change: 1 addition & 0 deletions packages/@aws-cdk/aws-lambda/lib/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ export * from './layers';
export * from './permission';
export * from './runtime';
export * from './code';
export * from './filesystem';
export * from './lambda-version';
export * from './singleton-lambda';
export * from './event-source';
Expand Down
2 changes: 2 additions & 0 deletions packages/@aws-cdk/aws-lambda/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@
"dependencies": {
"@aws-cdk/aws-cloudwatch": "0.0.0",
"@aws-cdk/aws-ec2": "0.0.0",
"@aws-cdk/aws-efs": "0.0.0",
"@aws-cdk/aws-events": "0.0.0",
"@aws-cdk/aws-iam": "0.0.0",
"@aws-cdk/aws-logs": "0.0.0",
Expand All @@ -99,6 +100,7 @@
"peerDependencies": {
"@aws-cdk/aws-cloudwatch": "0.0.0",
"@aws-cdk/aws-ec2": "0.0.0",
"@aws-cdk/aws-efs": "0.0.0",
"@aws-cdk/aws-events": "0.0.0",
"@aws-cdk/aws-iam": "0.0.0",
"@aws-cdk/aws-logs": "0.0.0",
Expand Down
Loading