Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(batch): new L2 Constructs #24775

Merged
merged 75 commits into from
Apr 7, 2023
Merged
Show file tree
Hide file tree
Changes from 40 commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
42bf0b8
deprecate old constructs + add most of managed CE
comcalvi Mar 6, 2023
6e31c93
multinode jobs still docstrings + eks type modeling needs to be done
comcalvi Mar 8, 2023
cd3a344
job definition typing done
comcalvi Mar 8, 2023
9b936ab
all typing
comcalvi Mar 8, 2023
bf696de
builds! And some tests have been written
comcalvi Mar 10, 2023
c76fdc1
working compute env defaults! except for spot...
comcalvi Mar 11, 2023
b1481c4
spotfleetrole
comcalvi Mar 13, 2023
8e3c927
runtime error checking + fixed subnets
comcalvi Mar 13, 2023
fadbcaa
eksclusternamespace default...need to unit + integ test
comcalvi Mar 15, 2023
2a2aabd
namespace required + tests work
comcalvi Mar 15, 2023
cf742a1
ecs lazy...still need to figure out how to add validations
comcalvi Mar 15, 2023
6adf6c1
eks lazy
comcalvi Mar 15, 2023
a63c62b
container checkpoint
comcalvi Mar 15, 2023
7bf967e
working job defn base
comcalvi Mar 15, 2023
6b8c386
EcsVolume
comcalvi Mar 16, 2023
1ebcb82
tmp
comcalvi Mar 16, 2023
9ffb2b6
need to ensure EksVolumes get automounted. Next up is multinode jobs,
comcalvi Mar 16, 2023
63798aa
multinode!
comcalvi Mar 17, 2023
ef0ce0b
kubernetes cluster now required...need to emit warning when user link…
comcalvi Mar 20, 2023
c03407f
so...eks clusters aren't trivial to config, so the user has to do it …
comcalvi Mar 21, 2023
e41fae6
test
comcalvi Mar 21, 2023
c097ac2
unmanaged tests
comcalvi Mar 22, 2023
2ed534c
tmp, before we ruin everything
comcalvi Mar 22, 2023
144b193
TS magic
comcalvi Mar 22, 2023
d4aade0
ecs volumes
comcalvi Mar 22, 2023
6c60622
eksContainers
comcalvi Mar 22, 2023
27b9363
buncha tests
comcalvi Mar 23, 2023
0237fde
policy test
comcalvi Mar 23, 2023
cb3497e
basically done, needs extra error checking around volumes
comcalvi Mar 23, 2023
88874da
tmp
comcalvi Mar 24, 2023
b1c45c3
integ tests, some readme, more unit tests
comcalvi Mar 24, 2023
62ca79c
need to finish the addVolume() methods
comcalvi Mar 27, 2023
95f9e5b
addVolume() methods + docs
comcalvi Mar 28, 2023
4e70e64
docs
comcalvi Mar 28, 2023
f7e038b
tmp
comcalvi Mar 28, 2023
4488f38
from methods + docstring
comcalvi Mar 28, 2023
07b6132
lint pt n
comcalvi Mar 28, 2023
8f89a26
yarn lint
comcalvi Mar 29, 2023
a9ba303
yarn lint (ec2)
comcalvi Mar 29, 2023
e54a4bd
Merge branch 'main' into 485
mergify[bot] Mar 29, 2023
784b605
comments
comcalvi Mar 29, 2023
f92e99f
integ tests
comcalvi Mar 30, 2023
15a5b4c
refactor
comcalvi Mar 30, 2023
8699f30
some readme complaints, will finish these tomorrow
comcalvi Mar 30, 2023
9c561a0
rename containerDefinition to container
comcalvi Mar 30, 2023
fec0757
readme + test fixes
comcalvi Mar 30, 2023
6babcca
remove build file
comcalvi Mar 30, 2023
89554cb
comment
comcalvi Mar 30, 2023
5e037db
comments
comcalvi Mar 30, 2023
fa08646
public / private / function
comcalvi Mar 31, 2023
3b2f98d
resource props
comcalvi Mar 31, 2023
252fb82
number -> Size
comcalvi Mar 31, 2023
92f8370
comments
comcalvi Mar 31, 2023
6549484
useJobDefinitionRole -> useJobRole
comcalvi Mar 31, 2023
1547bd2
property defaults
comcalvi Mar 31, 2023
fda4412
propagate tags does not apply to eks
comcalvi Mar 31, 2023
1ca2f70
EKS does not use spotfleetrole + spotfleetrole is only required if al…
comcalvi Mar 31, 2023
8722424
execution role is created by default if logging is enabled on ec2 con…
comcalvi Mar 31, 2023
71003c6
repo restructure conflicts
comcalvi Apr 3, 2023
83139a4
repo restructure
comcalvi Apr 3, 2023
039cd40
integ test
comcalvi Apr 3, 2023
a4ff001
new alpha testing package
comcalvi Apr 3, 2023
18b7d54
need to add test verify it throws / doesn't throw
comcalvi Apr 3, 2023
a2554cf
tmp
comcalvi Apr 5, 2023
5b173ca
working tests
comcalvi Apr 5, 2023
f5fb0f1
integ tests
comcalvi Apr 6, 2023
6f741f9
refactor
comcalvi Apr 6, 2023
bb2e023
doc string formatting
comcalvi Apr 6, 2023
a9455bb
cleanup
comcalvi Apr 6, 2023
b9584d1
todos
comcalvi Apr 6, 2023
e666e5d
rosetta
comcalvi Apr 6, 2023
52cf5ef
Merge branch 'main' into 485
comcalvi Apr 6, 2023
42848ef
undeeded change
comcalvi Apr 6, 2023
6ddef3d
woops we needed that
comcalvi Apr 6, 2023
34b5ba5
Merge branch 'main' into 485
mergify[bot] Apr 7, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
724 changes: 466 additions & 258 deletions packages/@aws-cdk/aws-batch/README.md

Large diffs are not rendered by default.

119 changes: 119 additions & 0 deletions packages/@aws-cdk/aws-batch/lib/compute-environment-base.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
import * as iam from '@aws-cdk/aws-iam';
import { IResource, Resource } from '@aws-cdk/core';
import { Construct } from 'constructs';
import { CfnComputeEnvironmentProps } from './batch.generated';


/**
* Represents a ComputeEnvironment
*/
export interface IComputeEnvironment extends IResource {
/**
* The name of the ComputeEnvironment
*
* @default - generated by CloudFormation
*
* @attribute
*/
readonly computeEnvironmentName?: string;
comcalvi marked this conversation as resolved.
Show resolved Hide resolved

/**
* The ARN of this compute environment.
*
* @attribute
*/
readonly computeEnvironmentArn: string;

/**
* The role Batch uses to perform actions on your behalf
* in your account, such as provision instances to run your jobs
*
* @default - a serviceRole will be created for managed CEs, none for unmanaged CEs
*/
readonly serviceRole?: iam.IRole;

/**
* Whether or not this ComputeEnvironment can accept jobs from a Queue.
* Enabled ComputeEnvironments can accept jobs from a Queue and
* can scale instances up or down.
* Disabled ComputeEnvironments cannot accept jobs from a Queue or
* scale instances up or down.
*
* If you change a ComputeEnvironment from enabled to disabled while it is executing jobs,
* Jobs in the `STARTED` or `RUNNING` states will not
* be interrupted. The ComputeEnvironment will no longer scale instances down;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the worst behavior

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ComputeEnvironment will scale down to minvCpus when DISABLED and running jobs complete on existing instances above min.

The Batch docs read

Managed compute environments that are in the DISABLED state don't scale out.

aka up.

If user want a non-zero minvCpus ComputeEnvironment to scale to zero for their use-case, set minvCpus = 0 when setting state to DISABLED.

* that is, any instances that were scaled up will remain if the ComputeEnvironment is disabled.
*
* To ensure you aren't billed for unused capacity, remove the ComputeEnvironment from your stack.
*/
readonly enabled: boolean;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this property exposed? What do we do for ComputeEnvironment.fromComputeEnvironmentArn() etc?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We set it to true and give empty values where needed:

  public static fromManagedEc2EcsComputeEnvironmentArn(
    scope: Construct, id: string, managedEc2EcsComputeEnvironmentArn: string,
  ): IManagedEc2EcsComputeEnvironment {
    const stack = Stack.of(scope);
    const computeEnvironmentName = stack.splitArn(managedEc2EcsComputeEnvironmentArn, ArnFormat.SLASH_RESOURCE_NAME).resourceName!;

    class Import extends Resource implements IManagedEc2EcsComputeEnvironment {
      public readonly computeEnvironmentArn = managedEc2EcsComputeEnvironmentArn;
      public readonly computeEnvironmentName = computeEnvironmentName;
      public readonly enabled = true;
      public readonly instanceClasses = [];
      public readonly instanceTypes = [];
      public readonly maxvCpus = 1;
      public readonly connections = { } as any;

      public addInstanceClass(_instanceClass: ec2.InstanceClass): void {
        throw new Error(`cannot add instance class to imported ComputeEnvironment '${id}'`);
      }
      public addInstanceType(_instanceType: ec2.InstanceType): void {
        throw new Error(`cannot add instance type to imported ComputeEnvironment '${id}'`);
      }
    }

    return new Import(scope, id);
  }

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand, but who needs it? What other construct cannot operate without this exposed?

Because it seems to be designed for use by subclasses, but the subclass initialization is awkward currently.

}

/**
* Props common to all ComputeEnvironments
*/
export interface ComputeEnvironmentProps {
/**
* The name of the ComputeEnvironment
*
* @default - generated by CloudFormation
*/
readonly computeEnvironmentName?: string;

/**
* The role Batch uses to perform actions on your behalf
* in your account, such as provision instances to run your jobs
*
* @default - a serviceRole will be created for managed CEs, none for unmanaged CEs
*/
readonly serviceRole?: iam.IRole;

/**
* Whether or not this ComputeEnvironment can accept jobs from a Queue.
* Enabled ComputeEnvironments can accept jobs from a Queue and
* can scale instances up or down.
* Disabled ComputeEnvironments cannot accept jobs from a Queue or
* scale instances up or down.
*
* If you change a ComputeEnvironment from enabled to disabled while it is executing jobs,
* Jobs in the `STARTED` or `RUNNING` states will not
* be interrupted. The ComputeEnvironment will no longer scale instances down;
* that is, any instances that were scaled up will remain if the ComputeEnvironment is disabled.
comcalvi marked this conversation as resolved.
Show resolved Hide resolved
*
* To ensure you aren't billed for unused capacity, remove the ComputeEnvironment from your stack.
*
* @default true
*/
readonly enabled?: boolean;
}

/**
* Abstract base class for ComputeEnvironments
*
* @internal
*/
export abstract class ComputeEnvironmentBase extends Resource implements IComputeEnvironment {
public readonly computeEnvironmentName?: string | undefined;
public readonly serviceRole?: iam.IRole | undefined;
public readonly enabled: boolean;
public abstract readonly computeEnvironmentArn: string;

protected resourceProps: CfnComputeEnvironmentProps;

constructor(scope: Construct, id: string, props?: ComputeEnvironmentProps) {
super(scope, id, {
physicalName: props?.computeEnvironmentName,
});

this.computeEnvironmentName = props?.computeEnvironmentName;
comcalvi marked this conversation as resolved.
Show resolved Hide resolved
this.serviceRole = props?.serviceRole;
this.enabled = props?.enabled || props?.enabled === undefined ? true : false;
comcalvi marked this conversation as resolved.
Show resolved Hide resolved

this.resourceProps = {
computeEnvironmentName: this.computeEnvironmentName,
serviceRole: this.serviceRole?.roleArn,
state: this.enabled ? 'ENABLED' : 'DISABLED',
type: 'dummy',
};
comcalvi marked this conversation as resolved.
Show resolved Hide resolved
}
}
Loading