[serverless] Add S3 span pointers

[nhulston]

What does this PR do?

Part 2 of #3123

Adds span pointers in S3 for putObject, copyObject, and completeMultipartUpload requests.

Span pointers are similar to Span Links, but for cases when it is impossible to pass the Trace ID and Span ID between the spans that need to be linked.

The changes in this tracer handle the 'upstream' case, for when a Lambda/EC2 instance/anything else using the tracer makes an S3 request. The changes in this PR in the serverless agent handles the 'downstream' case, for when a Lambda is triggered by an S3 update.

Both sides calculate a unique hash for the S3 update. When the calculated hashes for the upstream and downstream sides match, the Datadog backend will automatically link the two traces together.

When clicking on the linked span, a new tab opens linking to the downstream Lambda function that was triggered by this S3 object update. When the link in the downstream Lambda is clicked, it links back to the upstream trace.

Also, this PR moves the span.Finish() call from startTraceMiddleware to deserializeTraceMiddleware. We must do this because we can't add span links to a span that has already finished. Also, the span should be finished once the request has received a response (that's how the other tracers work), not when the request is sent.

Motivation

This is a new feature the Serverless team is implementing in every Lambda runtime.

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.
• Add an appropriate team label so this PR gets put in the right place for the release notes.
• Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.
• For internal contributors, a matching PR should be created to the v2-dev branch and reviewed by @DataDog/apm-go.

Unsure? Have a question? Request a review!

[pr-commenter]

Benchmarks

Benchmark execution time: 2025-02-07 22:01:06

Comparing candidate commit dee04b6 in PR branch nicholas.hulston/s3-span-pointers with baseline commit e08fe35 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 61 metrics, 1 unstable metrics.

[nhulston]

the aws span should live until we receive a response, not when the request is sent. (this is how AWS spans work in other tracers)

(from my testing this has barely any actual impact on the span end time)

This has to be done because we can't add links to a span that has already finished.