Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(aws-lambda): disableAwsContextPropagation config option #546

Merged
merged 8 commits into from
Jun 30, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ In your Lambda function configuration, add or update the `NODE_OPTIONS` environm
| --- | --- | --- |
| `requestHook` | `RequestHook` (function) | Hook for adding custom attributes before lambda starts handling the request. Receives params: `span, { event, context }` |
| `responseHook` | `ResponseHook` (function) | Hook for adding custom attributes before lambda returns the response. Receives params: `span, { err?, res? } ` |
| `disableAwsPropagation` | `boolean` | By default, this instrumentation will try to get the context using `propagator-aws-xray`, set this to `true` to disable this behavior |
nirsky marked this conversation as resolved.
Show resolved Hide resolved

### Hooks Usage Example

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,10 @@ export class AwsLambdaInstrumentation extends InstrumentationBase {
) {
const httpHeaders =
typeof event.headers === 'object' ? event.headers : {};
const parent = AwsLambdaInstrumentation._determineParent(httpHeaders);
const parent = AwsLambdaInstrumentation._determineParent(
httpHeaders,
Boolean(plugin._config.disableAwsPropagation)
nirsky marked this conversation as resolved.
Show resolved Hide resolved
);

const name = context.functionName;
const span = plugin.tracer.startSpan(
Expand Down Expand Up @@ -298,29 +301,34 @@ export class AwsLambdaInstrumentation extends InstrumentationBase {
}

private static _determineParent(
httpHeaders: APIGatewayProxyEventHeaders
httpHeaders: APIGatewayProxyEventHeaders,
disableAwsPropagation: boolean
): OtelContext {
let parent: OtelContext | undefined = undefined;
const lambdaTraceHeader = process.env[traceContextEnvironmentKey];
if (lambdaTraceHeader) {
parent = awsPropagator.extract(
otelContext.active(),
{ [AWSXRAY_TRACE_ID_HEADER]: lambdaTraceHeader },
headerGetter
);
}
if (parent) {
const spanContext = trace.getSpan(parent)?.spanContext();
if (
spanContext &&
(spanContext.traceFlags & TraceFlags.SAMPLED) === TraceFlags.SAMPLED
) {
// Trace header provided by Lambda only sampled if a sampled context was propagated from
// an upstream cloud service such as S3, or the user is using X-Ray. In these cases, we
// need to use it as the parent.
return parent;

if (!disableAwsPropagation) {
const lambdaTraceHeader = process.env[traceContextEnvironmentKey];
if (lambdaTraceHeader) {
parent = awsPropagator.extract(
otelContext.active(),
{ [AWSXRAY_TRACE_ID_HEADER]: lambdaTraceHeader },
headerGetter
);
}
if (parent) {
const spanContext = trace.getSpan(parent)?.spanContext();
if (
spanContext &&
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry to have this idea this late but does it work to add a check that that span context is valid and no need for the option? I just realized that I think in the problematic case the context has sampled = false, but actually doesn't have a valid span ID either and that could be enough of a signal

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by checking that the span context is valid? Doesn't the if (spanContext ... check covers it?
Anyway, for our usage, we'd be interested to have this ability even if x-ray is turned on, I'll try to describe the scenario, hope it'll be clear:

Consider the following:

image

  • Mircrosevice sends span with id x to the collector
  • API Gateway sends span with id y to x-ray
  • Lambda sends span with id z and parent span id of y.

We end up only having spans x and z in the collector, resulting in a broken trace.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if spanContext is only in the case that there is no span in the context, in practice I don't know if it can actually happen with this code.

There is an isSpanContextValid function

https://www.github.com/open-telemetry/opentelemetry-js-api/tree/main/src%2Fapi%2Ftrace.ts

For that use case, is the idea for the customer to have the trace both in x-ray and another backend via the collector and show the same trace in them?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this use case, the customer is interested only in having x and z where the parentSpanId of z is x.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case they can disable x-ray right? Is there a reason to enable it and be charged even without needing y?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of our customers may have an existing x-ray set-up and may want to use our product in addition to other usages, without integrating x-ray traces into our product.
Our product provides some other value on top of observability.

Anyway, IMO it's up for the user to decide, each one has different considerations and it's reasonable to let the user decide how to configure this

(spanContext.traceFlags & TraceFlags.SAMPLED) === TraceFlags.SAMPLED
) {
// Trace header provided by Lambda only sampled if a sampled context was propagated from
// an upstream cloud service such as S3, or the user is using X-Ray. In these cases, we
// need to use it as the parent.
return parent;
}
}
}

// There was not a sampled trace header from Lambda so try from HTTP headers.
const httpContext = propagation.extract(
otelContext.active(),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,5 @@ export type ResponseHook = (
export interface AwsLambdaInstrumentationConfig extends InstrumentationConfig {
requestHook?: RequestHook;
responseHook?: ResponseHook;
disableAwsPropagation?: boolean;
}
Original file line number Diff line number Diff line change
Expand Up @@ -526,6 +526,57 @@ describe('lambda handler', () => {
// Parent unsampled so no spans exported.
assert.strictEqual(spans.length, 0);
});

it('ignores sampled lambda context if "disableAwsPropagation" config option is true', async () => {
process.env[traceContextEnvironmentKey] = sampledAwsHeader;
initializeHandler('lambda-test/async.handler', {
disableAwsPropagation: true,
});

const result = await lambdaRequire('lambda-test/async').handler(
'arg',
ctx
);
assert.strictEqual(result, 'ok');
const spans = memoryExporter.getFinishedSpans();
const [span] = spans;
assert.strictEqual(spans.length, 1);
assertSpanSuccess(span);
assert.notDeepStrictEqual(
span.spanContext().traceId,
sampledAwsSpanContext.traceId
);
assert.strictEqual(span.parentSpanId, undefined);
});

it('takes sampled http context over sampled lambda context if "disableAwsPropagation" config option is true', async () => {
process.env[traceContextEnvironmentKey] = sampledAwsHeader;
initializeHandler('lambda-test/async.handler', {
disableAwsPropagation: true,
});

const proxyEvent = {
headers: {
traceparent: sampledHttpHeader,
},
};

const result = await lambdaRequire('lambda-test/async').handler(
proxyEvent,
ctx
);

assert.strictEqual(result, 'ok');
const spans = memoryExporter.getFinishedSpans();
const [span] = spans;
assert.strictEqual(spans.length, 1);
assertSpanSuccess(span);
assert.strictEqual(
span.spanContext().traceId,
sampledHttpSpanContext.traceId
);
assert.strictEqual(span.parentSpanId, sampledHttpSpanContext.spanId);
});
});

describe('hooks', () => {
Expand Down