Lambda CreateFunction and UpdateFunctionCode consume high amounts of memory due to logging

[flostadler]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Environment and Versions

aws-sdk-go-base: v2.0.0-beta.53

Expected Behavior

Logging of HTTP requests or responses shouldn't lead to excessive memory usage.

Actual Behavior

When creating or updating Lambdas with the code as a Zip file, the base64 encoded archive gets included in the request.
The request body gets copied and converted to a string no matter what log level is set (https://github.com/hashicorp/aws-sdk-go-base/blob/main/logger.go#L136).
Under the hood this calls out to the stdlib's httputil.DumpRequestOut https://github.com/hashicorp/aws-sdk-go-base/blob/main/logging/http.go#L119.

httputil.DumpRequestOut allocates memory for the whole body, just so it's truncated to maxRequestBodyLen (1024 chars) later (see: https://github.com/hashicorp/aws-sdk-go-base/blob/main/logging/http.go#L134).

When creating a lot of Lambda functions with inline code archives, this can drastically increase the memory usage.
What makes this worse is that the higher memory pressure can lead to request failures and retries on machines with lower amounts of memory. When this happens more requests (because of the retries) get logged and more memory is allocated. I believe this might be the root cause of: hashicorp/terraform#9364.

I am investigating the removal of the limitation introduced in hashicorp/terraform#9364, that forces Lambdas to be created in serial.
While testing with 50 Lambdas a 25MB I was observing the memory usage of the provider using pprof. I found that more memory was used on logging the Lambda requests in (*defaultRequestBodyLogger).Log (~3.8GB) then to serialize the bodies in awsRestjson1_serializeDocumentFunctionCode.

S3 PutObject & UploadPart have a similar problem and for this the sdk wrapper has a special request logger that redacts the body: https://github.com/hashicorp/aws-sdk-go-base/blob/main/logging/http.go#L148. I think doing the same thing for Lambda CreateFunction & UpdateFunctionCode would improve the memory usage of the provider and provide a path to reverting on the decision in hashicorp/terraform#9364 and ultimately speeding up Lambda function creation/updates.

Steps to Reproduce

1. Create a lambda function with a big inline code archive (e.g. 25MB).
2. Run terraform apply and attach pprof to provider
3. curl http://localhost:6060/debug/pprof/heap > heap.out to take a heap dump
4. go tool pprof heap.out to analyze the heap dump (e.g. web)

Debug Output

n/a

References

• "terraform apply" uses 3 GB memory and 50% of my PC for uploading 18 small lambdas terraform#9364
• https://github.com/hashicorp/aws-sdk-go-base/blob/main/logging/http.go#L119