-
Notifications
You must be signed in to change notification settings - Fork 93
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Document AWS Lambda compatibility #4
Comments
Hi joebowbeer, It's on our roadmap to support AWS Lambda, but we do not support it today. Stay tuned for updates soon! Meanwhile, see our public roadmap here: https://github.com/orgs/aws-observability/projects/4 |
Hello, Timing of OTEL auto-instrumentation.We followed a similar approach to what is described in the guide Tracing with the AWS Distro for OpenTelemetry JavaScript SDK, although my setup file is named When working with simple examples, it's easy to add a line similar to the following at the top of your code. This allows opentelemetry to intercept calls to require('./telemetry.js'); Unfortunately this approach an fall apart pretty quickly if you add babel/webpack into the mix to minimize the size of your lambda functions. A recommended alternative is to launch your application with a node argument that allows node --require ./telemetry.js src/index.js This turned out to be mildly challenging in a Lambda environment. I was eventually able to launch our NodeJS lambda functions using this approach, but it involved:
telemetry-wrapper#!/bin/bash
# the path to the interpreter and all of the originally intended arguments
args=("$@")
# the extra options to pass to the interpreter
extra_args=("--require" "/opt/telemetry.js")
# insert the extra options
args=("${args[@]:0:$#-1}" "${extra_args[@]}" "${args[@]: -1}")
# start the runtime with the extra options
exec "${args[@]}"
OTEL Exporters need to flush before shutting downIt's not clear to me if this is a responsibility for AWS or OTEL, but the OTEL exporters don't seem to have the oportuntiy to flush and pending spans before a lambda shuts down. I was able to sole this with a call While doing research to find a working solution to the timing issue, I found Sentry's approach to Lambda's pretty clean, hopefully this use case is something that OpenTelemetry gives some attention to in the future. Connecting OTEL to the external X-Ray trace.If you turn on X-Ray support for a Lambda, you automatically get high level instrumentation out-of-the-box. If you setup the Recording exceptions does not seem to workFor some reason, recording exceptions does not seem to work as expected. I haven't had time to dig deeper into what's going on, only I don't see any indication of the error or stack trace in x-ray. Other thoughts:I found this discussion interesting, but I'm not sure if a custom tracer would have helped with the issues I ran into: It would be nice if this opentelemetry plugin supported a Lambda environment: Below is an example that demonstrates some of the topics discussed above: import "core-js/stable";
import "regenerator-runtime/runtime";
const path = require('path');
const { NoRecordingSpan } = require('@opentelemetry/core');
const api = require('@opentelemetry/api');
const deployment = require('../package.json');
const axios = require('axios');
function instrumentHandler(handler) {
return async (event, context, callback) => {
const tracer = api.trace.getTracer(deployment.name, deployment.version);
// Currently AWSXRayPropagator expected to be passed HTTP headers,
// not a Lambda environment map.
const mockHttpRequestHeaders = {
'X-Amzn-Trace-Id': process.env._X_AMZN_TRACE_ID
};
// propagate remote AWS X-Ray span to current execution context
await api.context.with(api.propagation.extract(mockHttpRequestHeaders), async () => {
const remoteSpan = new NoRecordingSpan(api.context.active());
const handlerName = path.basename(process.env._HANDLER)
const handlerSpan = tracer.startSpan(handlerName, {
parent: remoteSpan,
kind: api.SpanKind.CONSUMER
});
let handlerReturn = null;
let handlerError = null;
try {
handlerReturn = await tracer.withSpan(handlerSpan, async () => {
return handler(event, context, callback);
});
} catch (error) {
handlerError = error;
// FIXME: Recorded exceptions to not make it to x-ray, attached to the respective
// span. By allowing the exception to bubble up, Lambda's x-ray integration
// will ultimately record the exception at a higher level span.
handlerSpan.recordException(error);
}
handlerSpan.end();
// ensure exporter(s) have a chance to flush spans before
// lambda fn shuts down or freezes
await api.trace.getTracerProvider().getDelegate().shutdown();
if (handlerError) {
throw handlerError
}
return handlerReturn;
});
};
}
// eslint-disable-next-line no-unused-vars
export const handler = instrumentHandler(async (event, context, callback) => {
console.log(process.env);
const response = await axios.get('https://ifconfig.co/json')
console.log(response.data);
}); |
After further investigation, it seems that the approach highlighted above likely does not handle the scenario in which Lambda's are frozen, and potentially never unfrozen. See the following open issue for details: |
A external-type lambda extension is needed? |
@joebowbeer - Possibly, it looks like some effort has been put towards using Lambda extensions for OTEL support. TBH I'm not familiar enough with Lambda's life-cycle or how the OTEL Collector works to know what additional challenges might be involved in the extension approach. Maybe someone from one of the following projects can provide recommendations. https://github.com/open-telemetry/opentelemetry-lambda-extension https://github.com/aws-observability/aws-otel-lambda |
Done. |
Is it possible to use AWS Distro for OpenTelemetry to instrument AWS Lambda functions?
From aws-otel.github.io :
If so, I recommend updating the documentation.
The text was updated successfully, but these errors were encountered: