Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Subscription Filter not added to cloudformation stack #4553

Closed
fedeBollo opened this issue Oct 17, 2019 · 11 comments
Closed

Subscription Filter not added to cloudformation stack #4553

fedeBollo opened this issue Oct 17, 2019 · 11 comments
Assignees
Labels
@aws-cdk/aws-logs Related to Amazon CloudWatch Logs bug This issue is a bug. needs-reproduction This issue needs reproduction.

Comments

@fedeBollo
Copy link

fedeBollo commented Oct 17, 2019

❓ Subscription Filter Object not create relative CloudFormation resources

The Question

Using SubscriptionFilter object, the relative resource are not created in cloudformation stack. Relative error are not showed.
Synth and deploy command doesn't return any error.

-->

Environment

  • **CDK CLI Version:1.13.1
  • **Module Version: aws-logs
  • **OS: MacOSX
  • **Language: Javascipt

Other information

Here is the code:

let cloudWatchGroup = cl_logs.LogGroup.fromLogGroupArn(this, 'lambda-log-group', res);
let filterPattern = cl_logs.FilterPattern.allEvents();
const subscriptionFilter = new cl_logs.SubscriptionFilter(this, 'elastic-search-subscription',{ destination: lambdaDestination, filterPattern: filterPattern, logGroup: cloudWatchGroup, });
`

@nmussy
Copy link
Contributor

nmussy commented Oct 17, 2019

Hey @fedeBollo,

Trying to read your reproduction stack was a little hard, so here is what I came up with, tell me if I'm too far off your use-case:

import {FilterPattern, LogGroup, SubscriptionFilter} from '@aws-cdk/aws-logs';
import {Code, Function as LambdaFn, Runtime} from '@aws-cdk/aws-lambda';
import {LambdaDestination} from '@aws-cdk/aws-logs-destinations';
import {Construct, Stack, StackProps} from '@aws-cdk/core';
// ...
const cloudWatchGroup = LogGroup.fromLogGroupArn(
  this,
  'lambda-log-group',
  'arn:aws:logs:us-east-1:123456789012:log-group:my-log-group',
);

const filterPattern = FilterPattern.allEvents();

const lambda = new LambdaFn(this, 'function', {
  runtime: Runtime.NODEJS_8_10,
  handler: 'hello.handler',
  code: Code.fromInline('() => {}'),
});

new SubscriptionFilter(this, 'elastic-search-subscription', {
  destination: new LambdaDestination(lambda),
  filterPattern: filterPattern,
  logGroup: cloudWatchGroup,
});

When I run cdk synth, I get the following:

// ...
  elasticsearchsubscription333E993D:
    Type: AWS::Logs::SubscriptionFilter
    Properties:
      DestinationArn:
        Fn::GetAtt:
          - functionF19B1A04
          - Arn
      FilterPattern: ""
      LogGroupName: test
    Metadata:
      aws:cdk:path: TestStack10/elastic-search-subscription/Resource

Is this not what you are getting?

EDIT: I've edited my example to use a valid log group ARN, without it the logGroupName property wasn't set.

@fedeBollo
Copy link
Author

Thanks for the reply.

Actually editing my code, I get the errore showed here:

/app/www/infocert/iqp_cloudwatch_event/node_modules/@aws-cdk/core/lib/construct.js:344
for (const reference of node._references) {
^

TypeError: node._references is not iterable

This error is show if I use "fromFunctionArn" method and also if I create a new LambdaFunction Object as you made in your examples.

Maybe is something depending on jsii implementation for Javascript?

Thanks for the help, really appreciate.

@nmussy
Copy link
Contributor

nmussy commented Oct 17, 2019

No problem @fedeBollo,

I'm fairly certain the JSII TypeScript runtime is the same as the JavaScript one, except for the exported d.ts types.

Could you send a complete and minimal reproduction of your error?

@SomayaB SomayaB added needs-reproduction This issue needs reproduction. @aws-cdk/aws-logs Related to Amazon CloudWatch Logs labels Oct 17, 2019
@SomayaB SomayaB added the bug This issue is a bug. label Oct 17, 2019
@fedeBollo
Copy link
Author

Hello @nmussy, sorry for the late answer.
Here is the complete trace error.

{project_dir}/node_modules/@aws-cdk/core/lib/construct.js:344
            for (const reference of node._references) {
                                         ^

TypeError: node._references is not iterable
    at recurse ({project_dir}/node_modules/@aws-cdk/core/lib/construct.js:344:42)
    at recurse ({project_dir}/node_modules/@aws-cdk/core/lib/construct.js:348:17)
    at ConstructNode.get references [as references] ({project_dir}/node_modules/@aws-cdk/core/lib/construct.js:351:9)
    at IqpCloudwatchEventStack.prepare ({project_dir}/node_modules/@aws-cdk/core/lib/stack.js:349:37)
    at Function.prepare ({project_dir}/node_modules/@aws-cdk/core/lib/construct.js:89:27)
    at Function.synth ({project_dir}/node_modules/@aws-cdk/core/lib/construct.js:52:14)
    at App.synth ({project_dir}/node_modules/@aws-cdk/core/lib/app.js:71:52)
    at Object.<anonymous> ({project_dir}/bin/iqp_cloudwatch_event.js:15:5)
    at Module._compile (internal/modules/cjs/loader.js:689:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)

Here you can find the complete CDK code, with all the method inside the constructor:

let lambdaArn = "lambda-arn";
        let lambdaElasticSearch = lambda.Function.fromFunctionArn(this, 'lambda-elasticsearch', lambdaArn);
        let cloudWatchGroup = cl_logs.LogGroup.fromLogGroupArn(this, 'log-group', "log-group-arn");
        let filterPattern = cl_logs.FilterPattern.allEvents();
        const subscriptionFilter = new cl_logs.SubscriptionFilter(this, 'elastic-search-subscription',{
            destination: new cl_logs_destination.LambdaDestination(lambdaElasticSearch),
            filterPattern: filterPattern,
            logGroup: cloudWatchGroup,
        });
        new cdk.CfnOutput(this, 'cloudwatch-subscription', {
            value: cloudWatchGroup.logGroupName
        });
`

@nmussy
Copy link
Contributor

nmussy commented Oct 18, 2019

Alright, I've been able to reproduce the issue, it does seem to be JS-related. Starting from scratch:

$ cdk init --language javascript
$ npm i @aws-cdk/aws-lambda @aws-cdk/aws-logs @aws-cdk/aws-logs-destinations
const cdk = require('@aws-cdk/core');
const lambda = require('@aws-cdk/aws-lambda');
const logs = require('@aws-cdk/aws-logs');
const logs_destinations = require('@aws-cdk/aws-logs-destinations');


class DeployJsStack extends cdk.Stack {
  /**
   *
   * @param {cdk.Construct} scope
   * @param {string} id
   * @param {cdk.StackProps=} props
   */
  constructor(scope, id, props) {
    super(scope, id, props);

    const lambdaElasticSearch = lambda.Function.fromFunctionArn(
        this,
        'lambda-elasticsearch',
        'arn:b:c:d:e:f',
    );
    const cloudWatchGroup = logs.LogGroup.fromLogGroupArn(
        this,
        'log-group',
        'arn:aws:logs:us-east-1:123456789012:log-group:my-log-group',
    );
    const filterPattern = logs.FilterPattern.allEvents();
    new logs.SubscriptionFilter(
        this,
        'elastic-search-subscription',
        {
          destination: new logs_destinations.LambdaDestination(lambdaElasticSearch),
          filterPattern: filterPattern,
          logGroup: cloudWatchGroup,
        },
    );

    new cdk.CfnOutput(this, 'cloudwatch-subscription', {
      value: cloudWatchGroup.logGroupName,
    });
  }
}

module.exports = { DeployJsStack }
$ npx cdk synth

node_modules\@aws-cdk\core\lib\construct.ts:423
      for (const reference of node._references) {
                                   ^
TypeError: node._references is not iterable
    at recurse (node_modules\@aws-cdk\core\lib\construct.ts:423:36)
    at recurse (node_modules\@aws-cdk\core\lib\construct.ts:428:9)
    at ConstructNode.get references [as references] (node_modules\@aws-cdk\core\lib\construct.ts:432:5)
    at DeployJsStack.prepare (node_modules\@aws-cdk\core\lib\stack.ts:487:33)
    at Function.prepare (node_modules\@aws-cdk\core\lib\construct.ts:84:28)
    at Function.synth (node_modules\@aws-cdk\core\lib\construct.ts:41:10)
    at App.synth (node_modules\@aws-cdk\core\lib\app.ts:128:36)
    at process.App.process.once (node_modules\@aws-cdk\core\lib\app.ts:111:45)
    at Object.onceWrapper (events.js:281:20)
    at process.emit (events.js:193:13)

Really not sure what's going on there

@fedeBollo
Copy link
Author

Thanks @nmussy. If you know a workaround also, please let me know, or I just wait for the fix.

Appreciate your help.

@nmussy
Copy link
Contributor

nmussy commented Oct 20, 2019

Switching to TypeScript seems like it would solve your issue.

If you don't feel like learning it, you could add allowJs: true to your tsconfig.json to keep your existing JS file, but the issue might still be there? I haven't tested it.

@fedeBollo
Copy link
Author

Hello @nmussy, your assumption are correct: switching to TypeScript with the same JS code and the allowJs field, solve the issue. Seems a non-related code error, but something with JS kernel or interpreter.
Here is the template:

Resources:
  elasticsearchsubscription333E993D:
    Type: AWS::Logs::SubscriptionFilter
    Properties:
      DestinationArn: arn:aws:lambda:eu-west-1:956251722582:function:LogsToElasticsearch_svts-monitoring
      FilterPattern: ""
      LogGroupName: /aws/lambda/IqpCloudwatchEventStack-iqpcloudwatchrulelambdaDDB-50WCW1RC9OOM:*
    Metadata:
      aws:cdk:path: SubscriptionTypescriptStack/elastic-search-subscription/Resource
  CDKMetadata:
    Type: AWS::CDK::Metadata
    Properties:
      Modules: aws-cdk=1.11.0,@aws-cdk/assets=1.13.1,@aws-cdk/aws-cloudwatch=1.13.1,@aws-cdk/aws-ec2=1.13.1,@aws-cdk/aws-events=1.13.1,@aws-cdk/aws-iam=1.13.1,@aws-cdk/aws-kms=1.13.1,@aws-cdk/aws-lambda=1.13.1,@aws-cdk/aws-logs=1.13.1,@aws-cdk/aws-logs-destinations=1.13.1,@aws-cdk/aws-s3=1.13.1,@aws-cdk/aws-s3-assets=1.13.1,@aws-cdk/aws-sqs=1.13.1,@aws-cdk/aws-ssm=1.13.1,@aws-cdk/core=1.13.1,@aws-cdk/cx-api=1.13.1,@aws-cdk/region-info=1.13.1,jsii-runtime=node.js/v10.15.1
    Condition: CDKMetadataAvailable
Outputs:
  cloudwatchsubscription:
    Value: /aws/lambda/IqpCloudwatchEventStack-iqpcloudwatchrulelambdaDDB-50WCW1RC9OOM:*
Conditions:
  CDKMetadataAvailable:
    Fn::Or:
      - Fn::Or:
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-east-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-northeast-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-northeast-2
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-south-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-southeast-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-southeast-2
          - Fn::Equals:
              - Ref: AWS::Region
              - ca-central-1
          - Fn::Equals:
              - Ref: AWS::Region
              - cn-north-1
          - Fn::Equals:
              - Ref: AWS::Region
              - cn-northwest-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-central-1
      - Fn::Or:
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-north-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-west-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-west-2
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-west-3
          - Fn::Equals:
              - Ref: AWS::Region
              - me-south-1
          - Fn::Equals:
              - Ref: AWS::Region
              - sa-east-1
          - Fn::Equals:
              - Ref: AWS::Region
              - us-east-1
          - Fn::Equals:
              - Ref: AWS::Region
              - us-east-2
          - Fn::Equals:
              - Ref: AWS::Region
              - us-west-1
          - Fn::Equals:
              - Ref: AWS::Region
              - us-west-2

I will check through the code and if I find something I will let you know.
Thanks again.

@nmussy
Copy link
Contributor

nmussy commented Oct 21, 2019

Alright, I've found the issue. The template is being initialized with outdated aws-cdk and @aws-cdk/core packages if your cdk CLI is outdated. Upgrading them solves the issue.

$ cdk --version
0.36.0 (build 6d38487)

$ cdk init --language javascript
Applying project template app for javascript
Executing npm install...
npm notice created a lockfile as package-lock.json. You should commit this file.

$ npm outdated
Package        Current   Wanted   Latest  Location
@aws-cdk/core   0.36.2   0.36.2   1.13.1  deploy
@types/node    8.10.45  8.10.45  12.11.1  deploy
aws-cdk         0.36.2   0.36.2   1.13.1  deploy

$ npm i aws-cdk@latest @aws-cdk/core@latest
+ aws-cdk@1.13.1
+ @aws-cdk/core@1.13.1

$ npm outdated
Package      Current   Wanted   Latest  Location
@types/node  8.10.45  8.10.45  12.11.1  deploy

I'm used to using local packages with npx, which is why mine is so old. Yours is probably more recent, but doesn't have the recently merged #2821, which removed _references from ConstructNode.

To upgrade your global CDK CLI version:

npm i -g aws-cdk@latest

Not sure what we should do about this issue.

@fedeBollo
Copy link
Author

Thansk @nmussy, this solve issue! I have an "auto-upgrade" script that periodically check if some of my packages is outdated and automatically upgrade it, maybe something is not working there...
This complete a really complicated stack I'm actually developing, really appreciate your help.

Thanks again!

@nmussy
Copy link
Contributor

nmussy commented Oct 21, 2019

No problem. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-logs Related to Amazon CloudWatch Logs bug This issue is a bug. needs-reproduction This issue needs reproduction.
Projects
None yet
Development

No branches or pull requests

4 participants