Lambda functions break when instrumenting with Datadog Layers when using NODEJS_LATEST runtime

[maherio]

Expected Behavior

I can create a Lambda function using Runtime.NODEJS_LATEST and instrument it with Datadog layers.
Or at the very least, my function is not instrumented at all and can still successfully execute.

Actual Behavior

• Datadog silently skips applying the layers (debug logs include Unsupported runtime: ${Token[TOKEN.1308]}).
• However it also continues trying to instrument the lambda, which results in the lambda executions failing since it is expecting the layer to be there (/opt/datadog_wrapper: does not exist).

Steps to Reproduce the Problem

const lambdaFunction = new Function(this, 'MyLambdaFunction', {
  runtime: Runtime.NODEJS_LATEST,
  ...
});

const datadog = new Datadog(this, "Datadog", {
  nodeLayerVersion: 115,
  extensionLayerVersion: 63,
  ...
});
datadog.addLambdaFunctions([lambdaFunction])

Specifications

• Node version: NODEJS_LATEST

[lym953]

@maherio Thanks for reporting. Could you share more of this code

const lambdaFunction = new Function(this, 'MyLambdaFunction', {
  runtime: Runtime.NODEJS_LATEST,
  ...
});

For example, what other params are you passing? Are you setting a bundling image?

[lym953]

Could you also provide:

1. version of aws-lambda package (by running npm -v aws-cdk-lib/aws-lambda)
2. version of Datadog CDK Construct
3. attributes passed to new Datadog()
4. what command you ran to deploy the stack

Thanks!

[cgatt]

Hi, I can chime in with some more info here. This is caused by aws/aws-cdk#30108, which was introduced in cdk version 2.146.0. When using Runtime.NODE_LATEST, CDK now uses a Fn::FindInMap to fetch the appropriate latest runtime version for the region.
This means that lambda.runtime.name is now a token, and as such the runtimeLookup in applyLayers fails to find the appropriate runtime family:


A potential alternative is the use the Runtime.family property instead of Runtime.name to determine the correct layer to apply.

[lym953]

@cgatt Thanks for the info! Let me try to reproduce it.

[lym953]

@cgatt Could you share your code for setting the runtime?

I'm unable to reproduce the issue using:

import * as lambda from "aws-cdk-lib/aws-lambda";

const nodeFunction = new Function(this, "hello-node", {
  runtime: lambda.Runtime.NODEJS_LATEST,
  ...
});

Detailed code in: https://github.com/DataDog/datadog-cdk-constructs/blob/9e60822878ea6385e8be9280519627871aa321fd/examples/typescript-stack/lib/cdk-typescript-stack-base.ts

This is because NODEJS_LATEST resolves to nodejs18.x in aws-cdk code:
https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/aws-lambda/lib/runtime.ts#L119

[cgatt]

Hi @lym953, This will only take effect from aws-cdk-lib version 2.146.0, while this repo is currently pinned to aws-cdk-lib 2.134.0. In order to recreate this issue, you'll need to bump this repo to a recent version of CDK or create a test project running latest CDK. The latest version of Node in most regions is now nodejs20.x.

[cgatt]

Hi @lym953,
My apologies, I was incorrect. The behaviour I was referring to occurs when using determineLatestNodeRuntime() function from the cdk-lambda, not NODEJS_LATEST. This function is most notably used by AwsCustomResource. It appears to only occur on stacks where env.region is not set.
See PR for more details: aws/aws-cdk#30108
A quick minimal example:

import { determineLatestNodeRuntime, InlineCode } from 'aws-cdk-lib/aws-lambda';
import { NodejsFunction } from 'aws-cdk-lib/aws-lambda-nodejs';
import { App, Stack } from 'aws-cdk-lib';
import { Datadog } from 'datadog-cdk-constructs-v2';

const app = new App();
const stack = new Stack(app, 'TestStack');

const lambda = new NodejsFunction(stack, 'MyLambda', {
  runtime: determineLatestNodeRuntime(this),
  code: InlineCode.fromInline("module.exports.handler = (event) => { console.log('Event received: ', JSON.stringify(event)); }"),
  handler: 'index.handler',
});

const datadog = new Datadog(stack, 'Datadog', {
  nodeLayerVersion: 115,
  extensionLayerVersion: 63,
  enableDatadogASM: true, // This causes AWS_LAMBDA_EXEC_WRAPPER to be set even if adding layers fails
  apiKey: 'foo',
});
datadog.addLambdaFunctions([lambda]);

app.synth();

I've not been able to recreate the behaviour with Runtime.NODEJS_LATEST, though theoretically any stack structure that results in the runtime being a token could cause this.

[lym953]

@cgatt Thanks for the additional info! I was able to reproduce it. Working on a fix.

[lym953]

For the short term, I'm going to make CDK Construct skip instrumenting the Lambda function to avoid runtime error.
For the long term, I'm going to create an internal ticket in our backlog for figuring out if there's a way to support this case. https://datadoghq.atlassian.net/browse/SVLS-5895

cc PM @sumedham to track this feature request.