AWS::Lambda::Function LogGroup

[PatMyron]

2. Scope

c) new attribute for existing resource is desired

3. Expected behavior

easily manage Lambda logs RetentionInDays, etc.

5. Links

AWS::Lambda::Function docs
AWS::Logs::LogGroup docs

https://stackoverflow.com/questions/39231592/specify-log-group-for-an-aws-lambda
https://stackoverflow.com/questions/50382048/how-to-setup-cloudwatch-log-for-a-lambda-created-in-cloudformation
https://stackoverflow.com/questions/45364967/set-expiration-of-cloudwatch-log-group-for-lambda-function
https://forums.aws.amazon.com/thread.jspa?messageID=811315
aws/aws-cdk#11549

6. Category

1. Compute (EC2, ECS, EKS, Lambda...)

[rclark]

This would also allow your Lambda function's IAM execution role to have better-scoped permissions than logs:* on resource *.

[benbridts]

This might be something that is better suited as an issue for the Serverless Application Model (SAM). You can already do this in CloudFormation, but it uses multiple resources.

AWSTemplateFormatVersion: '2010-09-09'
Description: LogGroup Retention Example
Para[Resource import](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import.html) meters:
  LogRetentionInDays:
    Type: String
    Default: ''
    AllowedValues: ['', 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 3653]

Conditions:
  LogRetentionInDaysSet: !Not [!Equals [!Ref LogRetentionInDays, '']]

Resources:
  LambdaRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service: lambda.amazonaws.com
          Action: sts:AssumeRole
      Path: !Ref RolePath
  LambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Code: src/
      Handler: app.lambda_handler
      MemorySize: 128
      Role: !GetAtt LambdaRole.Arn
      Runtime: python3.7
      Timeout: 3
      Description: !Sub "${AWS::StackName} lambda"
  LambdaLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub "/aws/lambda/${LambdaFunction}"
      RetentionInDays: !If [LogRetentionInDaysSet, !Ref LogRetentionInDays, !Ref AWS::NoValue]
  LambdaLogPermissions:
    Type: AWS::IAM::Policy
    Properties:
      Roles:
      - !Ref LambdaRole
      PolicyName: !Sub "${AWS::Region}-LambdaLogGroup"
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Action:
          - logs:CreateLogStream
          - logs:PutLogEvents
          Resource:
          - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/${LambdaFunction}"
          - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/${LambdaFunction}:*"
          - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/${LambdaFunction}:*:*"

Edit: and you can use Resource Import to get an existing LogGroup into CloudFormation.

[scottbrown]

I wrote about this method here which is very similar to ikben's answer, and there's a small bug in the CFN return value for LogGroup that doesn't alllow you to use the resource's return value in an IAM policy.

https://typicalrunt.me/2019/09/20/enforcing-least-privilege-when-logging-lambda-functions-to-cloudwatch/

[jdrydn]

Never mind RetentionInDays - being able to have one log group for a collection of Lambda functions (where each write to their own stream but collectively you can search one group for data) would be great!

[jplock]

@scottbrown thank you for your insightful post. One minor correction is that you’ve got a / in the ARN after log-group when it should be a :. Otherwise works fine

[notbrain]

Has anyone used the techniques above from @scottbrown and @ikben (or similar) to allow renaming of lambda functions but still create log streams in the same named log group? We're doing a bunch of renaming for account agnostic deploys and having to manage multiple historical log groups has become cumbersome.

[Ricapar]

@notbrain This has always been a challenge with Log Group management. At some point you have to assume they will be orphaned from CloudFormation, but should probably still make every attempt to get the initial setup right.

If you cause a resource replacement on the Lambda, then it's going to get a new name. With a new name you'll get a new Log Group, meaning by default CloudFormation will try to delete the now no-longer-needed Log Group regardless of what the retention time is - something that security folks may not be happy with.

If preserving logs is important to you, I'd recommend putting the DeletionPolicy/UpdateReplacePolicy to "Retain" on the AWS::Logs::LogGroup resource to account for any resource replacement.

The example from @ikben above is 100% what I do when creating Lambdas, or any other resource that writes to CWL. The problem with most services that write to CloudWatch Logs is that if you're sloppy with the IAM Permissions and give it something like logs:* on *, then it will just go ahead and create the Log Group resource automatically. Once that happens you can't control that log group via CloudFormation anymore.

For Lambda, for CodeBuilds, ECS Clusters, AWS Transfer, whatever - don't give the logging resource the logs:CreateLogGroup permission on their role if you intend to manage that Log Group via CloudFormation. And there's very little reason for why a Lambda should have logs:* on *.

For me this extends beyond just setting the retention time - setting up a subscription filter to send the logs to Elastic Search or Kinesis is challenging if you don't create the Log Group in CloudFormation.

Lambda and a few other services don't give you the option of choosing what the Log Group name is though. The good thing is that it is predictable - it's always /aws/lambda/${LambdaName}. This is why @ikben's example works. A few other services are equally as predictable and you can pre-empt the Log Group creation via figuring out the naming convention and finding the right thing to !Ref or !GetAtt.

It's a good habit to figure out how those work and plan your templates and IAM accordingly.

[petrgazarov]

Note Cloudformation will complain if LogGroup with the name /aws/lambda/${LambdaName} already exists. So, I had to delete the existing LogGroup (and archive all logs to S3) to get @benbridts's answer to work.

[benbridts]

Note Cloudformation will complain if LogGroup with the name /aws/lambda/${LambdaName} already exists. So, I had to delete the existing LogGroup (and archive all logs to S3) to get @benbridts's answer to work.

AWS::Log::LogGroup is now supported by resource import. I'm sorry, @petrgazarov, I didn't think about updating this issue with that information sooner. It would have saved you some effort.

[petrgazarov]

@benbridts resource import worked great, thanks!

[sharmanaman]

Anyone imported this log Group in (AWS SAM) templates?

[danilobuerger]

Why was this closed? It doesn't look like its possible reading the docs.

[rnhurt]

This is never getting fixed, is it?

[danilobuerger]

This was closed again without it being possible so far. @aygold92 could you please reopen it again?

[danilobuerger]

This was closed again without it being possible so far. @aygold92 could you please reopen it again?

[PatMyron]

Apparently released in November:
https://aws.amazon.com/blogs/compute/introducing-advanced-logging-controls-for-aws-lambda-functions/

[rnhurt]

Ok, the CFN docs have been updated with Logging functionality. I haven't actually tried it yet but it looks like we might have a winner! 🎉

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-function-loggingconfig.html

[rgoltz]

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-function-loggingconfig.html

RetentionInDays 🥲

[benbridts]

@rgoltz Instead of setting the config via the Lambda Function, you can create your own (custom) loggroup, and configure all properties of that. This allows you to start logging to a log group managed in CFN, without having to delete the current log group first