feat(stepfunctions): add service integrations #1646

rix0rrr · 2019-01-31T14:18:29Z

Add service integrations for SNS, SQS and ECS to StepFunctions.

Integration tests are done, but

Unit Tests still need to be added

Pull Request Checklist

Testing
- Unit test added
- CLI change?: coordinate update of integration tests with team
- cdk-init template change?: coordinated update of integration tests with team
Docs
- jsdocs: All public APIs documented
- README: README and/or documentation topic updated
Title and Description
- Change type: title prefixed with fix, feat will appear in changelog
- Title: use lower-case and doesn't end with a period
- Breaking?: last paragraph: "BREAKING CHANGE: <describe what changed + link for details>"
- Issues: Indicate issues fixed via: "Fixes #xxx" or "Closes #xxx"
Sensitive Modules (requires 2 PR approvers)
- IAM Policy Document (in @aws-cdk/aws-iam)
- EC2 Security Groups and ACLs (in @aws-cdk/aws-ec2)
- Grant APIs (only if not based on official documentation with a reference)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license.

eladb · 2019-01-31T18:19:28Z

packages/@aws-cdk/aws-ecs/lib/base/base-run-task.ts

+/**
+ * A StepFunctions Task to run a Task on ECS or Fargate
+ */
+export class BaseRunTask extends stepfunctions.Task implements ec2.IConnectable {


Why is this called "Base"?

Ah it should be abstract.

eladb · 2019-01-31T18:23:44Z

packages/@aws-cdk/aws-ec2/lib/vpc-ref.ts

@@ -192,7 +192,7 @@ export abstract class VpcNetworkBase extends Construct implements IVpcNetwork {
  /**
   * Return the subnets appropriate for the placement strategy
   */
-  public subnets(placement: VpcPlacementStrategy = {}): IVpcSubnet[] {
+  public subnets(placement: VpcPlacementStrategy = {}, required = SubnetQuery.Required): IVpcSubnet[] {


Can you provide some more details in the PR description about the changes needed in the VPC?

Yeah this was a bad idea. It should default the other way.

eladb · 2019-01-31T18:24:31Z

packages/@aws-cdk/aws-ec2/lib/vpc-ref.ts

+  /**
+   * Require at least one subnet
+   */
+  Required = 'Required',


Wouldn't it make more sense to make this a boolean? (required)

I suppose. Options with a boolean field is better, this is awkward.

eladb · 2019-01-31T18:24:46Z

packages/@aws-cdk/aws-ecs/lib/base/base-run-task.ts

+import { TaskDefinition } from './task-definition';
+
+/**
+ * Properties for SendMessageTask


SendMessage?

I want to make it clear it's a Task in a stepfunctions workflow.

eladb · 2019-01-31T18:26:19Z

packages/@aws-cdk/aws-ecs/lib/base/base-service.ts

    }
    if (securityGroup === undefined) {
      securityGroup = new ec2.SecurityGroup(this, 'SecurityGroup', { vpc });
    }
    const subnets = vpc.subnets(vpcPlacement);
    this.connections.addSecurityGroup(securityGroup);

+    if (assignPublicIp === undefined && subnets.every(s => vpc.isPublicSubnet(s))) {


add a little comment explaining this logic

packages/@aws-cdk/aws-stepfunctions/README.md

eladb · 2019-02-04T13:54:16Z

packages/@aws-cdk/aws-stepfunctions/README.md

+  queue that you poll on a compute fleet you manage yourself)
+* `sns.PublishTask` -- publish a message to an SNS topic
+* `sqs.SendMessageTask` -- send a message to an SQS queue
+* `ecs.FargateRunTask`/`ecs.Ec2RunTask` -- run a container task


I rather these would have a common prefix, so maybe (along with my comment on "Task" above): RunStepFunctionsTaskOnFargate and RunStepFunctionsTaskOnEc2

eladb · 2019-02-04T13:55:50Z

packages/@aws-cdk/aws-stepfunctions/lib/states/task.ts

+    /**
+     * The resource that represents the work to be executed
+     *
+     * Can be either a Lambda Function or an Activity.


I guess this comment can be updated

eladb · 2019-02-04T13:55:59Z

packages/@aws-cdk/aws-stepfunctions/lib/states/task.ts

+     *
+     * @default No parameters
+     */
+    parameters?: { [name: string]: any };


any => string?

Can be nested objects too.

eladb · 2019-02-04T13:57:52Z

packages/@aws-cdk/aws-sqs/lib/send-message-task.ts

+/**
+ * A StepFunctions Task to send a message to an SQS Queue
+ */
+export class SendMessageTask extends stepfunctions.Task {


Should we add queue.toStepFunction() as a convenience/discovery mechanism?

@skinny85, this is similar to what we have for pipelines (toCodePipeline()).

eladb · 2019-02-05T11:52:07Z

@rix0rrr let's land this. Can I help somehow?

eladb · 2019-02-06T12:25:41Z

@rix0rrr ping pong

rix0rrr · 2019-02-08T18:43:22Z

Yeah. I'm also not sure about the XxxTask suffix (in relation to SFN), but I really have nothing better.

StepFunctionsTaskRunOnFargate is horrible. I guess we could do SfnRunFargateTask, but "sfn" is not super official, and it's also inconsistent with tasks inside the aws-stepfunctions package.

Alternatively we put them in a different package so it's obvious they're "extended" tasks. @aws-cdk/aws-stepfunctions-tasks maybe.

There's actually a conflation here, because the step in the Workflow is called a Task, and a running instance of a piece of code on Fargate is also called a Task. The official name should be something like FargateRunTaskStepFunctionsTask. Let's not put 2 Tasks in the class name though.

eladb · 2019-02-11T11:02:25Z

I think we have a precedence with codepipeline actions and I think the current naming convention is CodePipelineActionSomething.

I know it’s ugly and long but there is a benefit in the common prefix and deconflation

eladb · 2019-02-11T11:07:07Z

It’s actually not that consistent with pipelines. Let’s coordinate with @skinny85 and come up with a convention?

skinny85 · 2019-02-12T00:04:40Z

So for CodePipeline, the convention is currently:

<servicename>.Pipeline<ActionType>Action (for example, s3.PipelineSourceAction) for actions outside the CodePipeline package
codepipeline.<Service><ActionType>Action (for example, codepipeline.GitHubSourceAction) for actions inside the CodePipeline package

However, as we've discussed earlier, this convention is:

a) inconsistent (different inside vs outside the CodePipeline package),
b) does not translate to languages without prefixed namespaces (like Java and C#), and
c) depends on the customer using the same prefix for their import as we recommend (codepipeline).

Given all of the above issues, I'm more than happy to iterate on it and come up with something better, that fits both CodePipeline and Step Functions.

eladb · 2019-02-12T09:00:13Z

@skinny85 that will be great if we can formalize this. @rix0rrr also had a few other naming conventions in mind that we wanted to formalize. How about kicking off a PR that adds a section to the design/aws-guidelines doc and we can iterate there?

adjmaker · 2019-02-20T15:52:41Z

Hello, what's going on with this PR?

Is @rix0rrr still working on this?

…ons-service-integrations

eladb · 2019-05-06T13:29:28Z

packages/@aws-cdk/aws-ec2/lib/vpc-ref.ts

@@ -115,7 +115,7 @@ export enum SubnetType {
   * This can be good for subnets with RDS or
   * Elasticache endpoints
   */
-  Isolated = 1,
+  Isolated = 'Isolated',


nit: usually we use lowercase for the values when they are string

eladb · 2019-05-06T13:31:12Z

packages/@aws-cdk/aws-ecs/package.json

@@ -86,6 +86,7 @@
    "@aws-cdk/aws-secretsmanager": "^0.29.0",
    "@aws-cdk/aws-servicediscovery": "^0.29.0",
    "@aws-cdk/aws-sns": "^0.29.0",
+    "@aws-cdk/aws-stepfunctions": "^0.29.0",


revert I presume?

eladb · 2019-05-06T13:31:26Z

packages/@aws-cdk/aws-iam/lib/policy-document.ts

@@ -533,6 +533,10 @@ export class PolicyStatement extends cdk.Token {
        return undefined;
      }

+      if (cdk.unresolved(values)) {


Use Token.isToken

eladb · 2019-05-06T13:32:24Z

packages/@aws-cdk/aws-stepfunctions-tasks/lib/base-run-task.ts

+import cdk = require('@aws-cdk/cdk');
+
+/**
+ * Properties for SendMessageTask


Seems like this doc needs a refresher

eladb · 2019-05-06T13:33:32Z

packages/@aws-cdk/aws-stepfunctions-tasks/lib/base-run-task.ts

+/**
+ * Properties for SendMessageTask
+ */
+export interface CommonRunTaskProps extends stepfunctions.BasicTaskProps {


I kind of expected this to be under some ecs.ts and have some ECS context in the interface name

Introducing the `@aws-cdk/aws-stepfunctions-tasks` package, with service integrations for ECS, SNS and SQS. Lambda and Activity integrations have been moved there as well. Add jest bindings for `haveResource` and `haveResourceLike`. Allow specifying `placementConstraints` and `placementStrategies` in the constructor of `EC2Service`. Exposing parts of `TokenMap` so that the reverse mapping of string => Token can be used by non-core libraries, if they need to embed non-literal data representation facilities into complex data structures (used in the SFN tasks to represent data that is extracted from the state JSON via a JSONPath). BREAKING CHANGES * If your using Lambdas or Activities in StepFunctions workflows, you must now instantiate `InvokeFunction` or `InvokeActivity` from the `@aws-cdk/aws-stepfunctions-tasks` package around it. * `PlacementConstraint` used in task definitions is now a union object, constructed using factory functions on the `PlacementConstraint` class. * Empty placement constraints and strategies will no longer render in the CloudFormation template. This may appear as diffs showing an empty line.

Rico Huijbers and others added 5 commits January 28, 2019 16:41

Implement service integration for SNS & SQS

e33629a

WIP

28d74ce

WIP

df9fb7b

Add support for running ECS tasks in StepFunctions

b1dcd17

Make FunctionTask and ActivityTask to match new naming pattern

a4e74e8

rix0rrr requested review from SoManyHs and a team as code owners January 31, 2019 14:18

eladb reviewed Feb 4, 2019

View reviewed changes

eladb mentioned this pull request Feb 4, 2019

feat: re-structure the CodePipeline Construct library API #1590

Merged

4 tasks

rix0rrr self-assigned this Feb 4, 2019

rix0rrr mentioned this pull request Feb 12, 2019

naming conventions: cross-service integrations #1743

Closed

eladb mentioned this pull request Feb 24, 2019

Support all Step Function task resources #1314

Closed

7 tasks

rix0rrr added 6 commits April 23, 2019 15:06

Merge remote-tracking branch 'origin/master' into huijbers/stepfuncti…

b501289

…ons-service-integrations

WIP

29c6e0c

Merge remote-tracking branch 'origin/master' into huijbers/stepfuncti…

106e04e

…ons-service-integrations

Get ECS library to build

55a15e1

Move metrics away

b3cd0bc

Adding tests

7766d0e

rix0rrr force-pushed the huijbers/stepfunctions-service-integrations branch from 0455c51 to 7766d0e Compare April 27, 2019 11:48

Don't inherit from task, do composition instead

d0c5283

eladb reviewed May 6, 2019

View reviewed changes

Refactor

8db693a

eladb approved these changes May 8, 2019

View reviewed changes

Remove peerDependency

755ef40

rix0rrr merged commit e4ac767 into master May 8, 2019

rix0rrr deleted the huijbers/stepfunctions-service-integrations branch May 8, 2019 14:40

astj mentioned this pull request May 30, 2019

Update aws cdk to v0.32.0 aereal/cdk-mackerel-container-agent#52

Merged

1 task

NGL321 added the contribution/core This is a PR that came from AWS. label Sep 27, 2019

This was referenced Jan 20, 2020

[Snyk] Upgrade @aws-cdk/aws-route53 from 0.22.0 to 0.39.0 MechanicalRock/tech-radar#20

Closed

[Snyk] Upgrade @aws-cdk/aws-s3 from 0.22.0 to 0.39.0 MechanicalRock/tech-radar#21

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(stepfunctions): add service integrations #1646

feat(stepfunctions): add service integrations #1646

rix0rrr commented Jan 31, 2019

eladb Jan 31, 2019

rix0rrr Feb 8, 2019

eladb Jan 31, 2019

rix0rrr Feb 8, 2019

eladb Jan 31, 2019

rix0rrr Feb 8, 2019

eladb Jan 31, 2019

rix0rrr Feb 8, 2019

eladb Jan 31, 2019

eladb Feb 4, 2019

eladb Feb 4, 2019

eladb Feb 4, 2019

rix0rrr Feb 8, 2019

eladb Feb 4, 2019

eladb commented Feb 5, 2019

eladb commented Feb 6, 2019

rix0rrr commented Feb 8, 2019 •

edited

Loading

eladb commented Feb 11, 2019

eladb commented Feb 11, 2019

skinny85 commented Feb 12, 2019

eladb commented Feb 12, 2019

adjmaker commented Feb 20, 2019

eladb May 6, 2019

eladb May 6, 2019

eladb May 6, 2019

eladb May 6, 2019

eladb May 6, 2019

feat(stepfunctions): add service integrations #1646

feat(stepfunctions): add service integrations #1646

Conversation

rix0rrr commented Jan 31, 2019

Pull Request Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eladb commented Feb 5, 2019

eladb commented Feb 6, 2019

rix0rrr commented Feb 8, 2019 • edited Loading

eladb commented Feb 11, 2019

eladb commented Feb 11, 2019

skinny85 commented Feb 12, 2019

eladb commented Feb 12, 2019

adjmaker commented Feb 20, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rix0rrr commented Feb 8, 2019 •

edited

Loading