Slow start-up?

[a-h]

I tried out migrating away from AWS's X-Ray SDK for Lambda, but the Open Telemetry Lambda layer appears to add a significant amount to cold start time, which I didn't expect.

It was suggested I cross post, since this is actually the repo with the layers in.

aws-observability/aws-otel-lambda#228 (comment)

Here's the data for reference:

I don't see any documentation on performance, comparison to X-Ray performance etc.

Is there a plan to reduce this? I didn't expect to have to add a Lambda layer to get OpenTelemetry working, I thought it would be included in the Lambda runtime as a first class thing, rather than being a sort-of add-on.

[mhausenblas]

Hi, ADOT PM here. Thanks a lot, we're already in the process to dive deep on this and I will report back once we have some shareable data (ETA: early August 2022).

[adambartholomew]

Is there an expected startup time for the ADOT collector and instrumentation? Running the latest nodejs layer (1.6.0:1) I am still witnessing 2+ second startups. Tested with both 1024MB and 1536MB memory.

Same test requests go from 3 seconds to 100ms after warming up:

Sample initialization:

[mhausenblas]

@adambartholomew we've identified the issue with cold starts and are considering ways to address this. Thanks for sharing your data points and we currently do not publish expected startup times.

[sam-goodwin]

Any update on this? Is there an eta for a fix? Can we expect a solution that is comparable to native EMF?

[adambiggs]

Also eager to hear any updates. Is there any workarounds in the meantime?

[Sutty100]

AWS Snapstart would seem to remove this as a problem. Benchmarking I've done shows cold starts are largely removed as a factor when using SnapStart

[a-h]

@Sutty100 - SnapStart is Java only, and specifically only java11, so it doesn't solve anything for most people using Lambda.

[RichiCoder1]

AWS Snapstart would seem to remove this as a problem. Benchmarking I've done shows cold starts are largely removed as a factor when using SnapStart

I believe even if/when it's expanded too, it doesn't currently support or address Lambda Extensions so it wouldn't benefit this issue right now either.

[bilalq]

Related issue in aws-otel-lambda repo: aws-observability/aws-otel-lambda#228

[RichiCoder1]

@mhausenblas not to be too noisy, but is there any update to this? Or a plan to provide an update? This makes using the Otel layer close to a no-go for a number of latency sensitive cases.

[mhausenblas]

@RichiCoder1 no problem at all, yes we're working on it and should be able to share details soon. Overall, our plan is to address the issues in Q1, what we need to verify is to what extent.

[tsloughter]

To give some feedback, I believe this is believed to be due to auto-instrumentation. So you may be able to improve your startup now by building your own, narrower, layer.

[a-h]

@tsloughter - do you mean "the 200ms cold start time is caused by auto-instrumentation"?

I can't see how that could be the case. Since https://opentelemetry.io/docs/instrumentation/go/libraries/ says:

Go does not support truly automatic instrumentation like other languages today.

And the Lambda layer is written in Go.