[question] How to instrument SQS lambdas properly?

[Dreamescaper]

Component

OpenTelemetry.Instrumentation.AWSLambda

Question details

SQSEvent might contains multiple messages. How to handle them properly?

I can see that ExtractParentContext supports SQSEvent.SQSMessage input. So does it make sense to wrap each message processing, like this?

foreach (var message in sqsEvent.Records)
{
	await AWSLambdaWrapper.TraceAsync(tracerProvider, (message, context) => HandleAsync(...), message, lambdaContext);
}

Or maybe wrap both SQSEvent and SQSEvent.SQSMessage?

await AWSLambdaWrapper.TraceAsync(tracerProvider, async (sqsEvent, lambdaContext) =>
{
    foreach (var message in sqsEvent.Records)
    {
        await AWSLambdaWrapper.TraceAsync(tracerProvider, (message, context) => HandleAsync(...), message, lambdaContext);
    }
}, sqsEvent, lambdaContext);

[github-actions]

Tagging component owner(s).

@srprash @ppittle @muhammad-othman @rypdal @Oberon00

[Oberon00]

In the docs for Dynatrace we recommend calling TraceAsync only with the actual event object, not the messages: https://docs.dynatrace.com/docs/shortlink/aws-lambda-otel-dotnet#sqs-sns-in

After all, there is only one Lambda invocation, so calling TraceAsync multiple times seems semantically wrong. I think what would be needed would be an additional API that creates the right spans to represent that we are now processing a message but not invoking the Lambda again. Related new issue seems to be #2244 (though the SDK defines a different message type than Lambda libraries). Otherwise I'd recommend following the OTel messaging semantic conventions yourself. https://github.com/open-telemetry/semantic-conventions/blob/main/docs/messaging/README.md

[xactlance]

@Oberon00 But isn't each record it's own message with it's own MessageAttributes and thus it's own traceparent and tracestate? I personally do the second approach:

foreach (var message in sqsEvent.Records)
{
     await AWSLambdaWrapper.TraceAsync(tracerProvider, (message, context) => HandleAsync(...), message, lambdaContext);
}

I would love to know if this is wrong.

[Oberon00]

I'm not 100% sure but won't it get the faas.trigger, etc. activity tags? That way, it would mean it was a separate Lambda invocation which it wasn't. So the faas attributes would need to be skipped at least