Skip to content

Conversation

@DivMode
Copy link

@DivMode DivMode commented Nov 3, 2025

Problem

DynamoDB Streams and other AWS event sources lack metadata carriers for trace propagation. Unlike SQS (message attributes) or API Gateway (HTTP headers), DynamoDB Streams only contain the changed record data.

The Trace Propagation Issue

When using OpenTelemetry with DynamoDB Streams:

  1. Producer writes to DynamoDB with trace context stored in record:

    {
      "data": "data",
      "_otel": {
        "traceparent": "00-7d7836d35ab7da657d35c2b5f889c3be-de0931d210e214b3-01"
      }
    }
  2. Consumer Lambda needs to extract and restore this context

  3. Problem: AwsLambdaInstrumentation creates automatic wrapper span with NEW trace ID before handler code runs:

    Trace A: 7d7836d35ab7da657d35c2b5f889c3be (original fetch)
    Trace B: 49792b11678269a91638c0717450480b (NEW trace - disconnected!)
    
  4. Result: Distributed trace is broken - cannot correlate stream processing with original operation

Solution

This PR extends the functionality added in the base implementation to make aws-lambda and aws-sdk instrumentations respect OTEL_NODE_DISABLED_INSTRUMENTATIONS.

Previously, PR #1653 added the ability to disable instrumentations via OTEL_NODE_DISABLED_INSTRUMENTATIONS, but the AWS Lambda and AWS SDK instrumentations were always loaded unconditionally in the createInstrumentations() function. This meant users couldn't disable these instrumentations even when setting the environment variable.

Usage:

export OTEL_NODE_DISABLED_INSTRUMENTATIONS=aws-lambda

Effect: Disables automatic Lambda wrapper span, allowing handler to manually restore trace context from event data before creating spans.

Use Case: DynamoDB Streams Trace Propagation

Producer (stores trace context):

await dynamodb.put({
  Item: {
    data: "data",
    _otel: {
      traceparent: currentTraceContext.traceparent
    }
  }
});

Consumer (restores trace context with disabled AwsLambdaInstrumentation):

// Lambda layer: OTEL_NODE_DISABLED_INSTRUMENTATIONS=aws-lambda
// No automatic wrapper span created!

export const handler = async (event: DynamoDBStreamEvent) => {
  const record = event.Records[0];
  const otelContext = unmarshall(record.dynamodb.NewImage)._otel;

  // Manually extract trace context
  const carrier = { traceparent: otelContext.traceparent };
  const extractedContext = propagation.extract(ROOT_CONTEXT, carrier);

  // Execute in extracted context (continues ORIGINAL trace!)
  await context.with(extractedContext, async () => {
    await OTELSpan.withSpan("process stream record", async () => {
      // All operations here inherit original trace ID
      await processRecord(record);
    });
  });
};

Result: Full trace continuity from producer → DynamoDB → stream → consumer

Related Issues

Event Sources Affected

This enables manual trace propagation for AWS event sources without metadata carriers:

Event Source Metadata Carrier Auto-Extracted Needs Manual Propagation
API Gateway ✅ HTTP headers ✅ YES ❌ NO
SQS ✅ Message attributes ✅ YES ❌ NO
SNS ✅ Message attributes ✅ YES ❌ NO
EventBridge ✅ Event metadata ✅ YES ❌ NO
DynamoDB Streams None ❌ NO YES (must store in record data)
Kinesis Streams None ❌ NO YES (must store in payload)

Industry Pattern

This approach aligns with industry best practices for DynamoDB Streams trace propagation:

  • Datadog (2024): "Distributed Tracing with Amazon DynamoDB" - Stores trace metadata in DynamoDB attributes, extracts in consumer
  • SigNoz (2024): "OpenTelemetry Context Propagation" - Manual inject/extract for message queues without auto-propagation
  • Better Stack (2023): "Implementing Distributed Tracing with OpenTelemetry" - Producer stores traceparent, consumer extracts and applies

Testing

  • Verified aws-lambda and aws-sdk added to defaultInstrumentationList
  • Verified conditional loading based on getActiveInstumentations()
  • Tested in development environment with OTEL_NODE_DISABLED_INSTRUMENTATIONS=aws-lambda
  • Confirmed trace propagation works for DynamoDB Streams in production

Benefits

  1. ✅ Enables distributed tracing across DynamoDB Streams (previously impossible)
  2. ✅ Provides control over when Lambda instrumentation runs (user choice)
  3. ✅ Maintains all other auto-instrumentations (AWS SDK, HTTP, net/TLS)
  4. ✅ No performance overhead (conditional loading, not runtime checks)
  5. ✅ Follows OpenTelemetry context propagation best practices
  6. ✅ Implements feature requested in issue NodeJS: Add option to disable AWS Lambda auto instrumentation. #1803

…L_NODE_DISABLED_INSTRUMENTATIONS

Extends PR open-telemetry#1653 to make AWS-specific instrumentations (aws-lambda, aws-sdk)
also respect the OTEL_NODE_DISABLED_INSTRUMENTATIONS environment variable.

Previously, these instrumentations were always loaded regardless of the
disable flag. Now they follow the same conditional loading pattern as
other instrumentations.

Fixes: Users can now disable AWS Lambda auto-instrumentation in dev mode
@DivMode DivMode requested a review from a team as a code owner November 3, 2025 06:26
@serkan-ozal serkan-ozal added the javascript Pull requests that update Javascript code label Nov 6, 2025
@serkan-ozal
Copy link
Contributor

Hi @DivMode,

Could you please fix the lint issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

javascript Pull requests that update Javascript code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants