Problem with long running functions

[marcioemiranda]

Hello Heitor,

I have the tracer decorator applied to a lambda that may take close to 15 minutes to execute.

If the execution happens in a short time, everything is fine and trace renders correctly in XRay.
However, if the execution takes close to 15 minutes, the trace shows as "Pending" and Service Map doesn't render correctly.

This lambda function reads a stream from S3 and executes a lot of Dynamo BatchWriteItem requests.

The function uses threading, but I don't think this is the cause of the problem. It seems that the problem is related to a long running task. I've found the following reference in the documentation:

https://docs.aws.amazon.com/xray/latest/devguide/xray-api-segmentdocuments.html

For long-running requests, you can send an in-progress segment to notify X-Ray that the request was received, and then send subsegments separately to trace them before completing the original request.

When the request is complete, close the segment by resending it with an end_time. The complete segment overwrites the in-progress segment.

The snippets above suggest I should have called create_segment in the end passing end_time. I've taken a look at the code of the tracer and couldn't find any create_segment there.

Any thoughts?

[heitorlessa]

You can't create a segment in Lambda as that's the root segment - Lambda creates it, and within your Lambda (or Powertools) we can only create subsegment from it.

Is that still showing as "Pending"? Within the trace, what info does it show? if the trace was received and has some annotations then it worked, otherwise X-Ray received half of the trace.

Let me know and I can have a chat with X-Ray team about this and ask you trace ID and other details via DM (@heitor_lessa on Twitter).

[marcioemiranda]

Hello Heitor.

Attached is a screen shot showing this behavior

Trace ID is 1-5e841491-3d01a1f3bdeb2ed21d8930cb

This sample execution ended successfully by the lambda function.

[heitorlessa]

@marcioemiranda - Could you send me the raw trace (raw data) via email? lessa at amazon dot com.

That Pending would generally happen in two conditions: A) Function hasn't returned successfully and subsegments weren't closed, B) Something happened on our end (Lambda svc) when sending trace data

[marcioemiranda]

Hello Heitor,

I've just sent the email. thanks!

[heitorlessa]

I'm gonna close this here as some work will be done in the app as we per email. Feel free to reopen if you find the same result otherwise