[Bug]: aws_lambda_function crashes with fatal error: out of memory

[JonathanRys]

Terraform Core Version

1.4.6

AWS Provider Version

4.67.0

Affected Resource(s)

aws_lambda_function

Expected Behavior

It should have deployed my lambda

Actual Behavior

runtime: out of memory: cannot allocate 671088640-byte block (1065025536 in use)
fatal error: out of memory

Zip file size: 305 MB (320,278,949 bytes)

Relevant Error/Panic Output Snippet

Error: The terraform-provider-aws_v4.67.0_x5.exe plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Terraform Configuration Files

data "aws_iam_policy_document" "assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role" "iam_for_lambda" {
  name               = "iam_for_lambda"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

data "archive_file" "lambda" {
  type        = "zip"
  source_dir = "package"
  output_path = "lambda_function_payload.zip"
}

resource "aws_lambda_function" "pinecone-lambda" {
  # If the file is not in the current working directory you will need to include a
  # path.module in the filename.
  filename      = "lambda_function_payload.zip"
  function_name = "pinecone-lambda"
  role          = aws_iam_role.iam_for_lambda.arn
  handler       = "pinecone-lambda.lambda_handler"
  memory_size   = 256

  source_code_hash = data.archive_file.lambda.output_base64sha256

  runtime = "python3.10"

  environment {
    variables = {
      ...
    }
  }
}

Steps to Reproduce

This worked fine with source_file instead of source_dir except that my libraries weren't available :(. I can't mix the two, so I moved the lambda handler and packages into a folder and used source_dir instead, that's when I got this error.

my requirements.txt:

pinecone-client==2.2.1
pymongo==4.3.3
nltk==3.8.1
sentence_transformers==2.2.2

Debug Output

https://gist.github.com/JonathanRys/43815164fdca16f1a3752ad43a9c221e#file-crash-log

Panic Output

see above

Important Factoids

Processor Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz 4.01 GHz
Installed RAM 16.0 GB

System type 64-bit operating system, x64-based processor
Pen and touch No pen or touch input is available for this display

Edition Windows 10 Pro
Version 22H2
Installed on ‎11/‎17/‎2021
OS build 19045.2846
Experience Windows Feature Experience Pack 120.2212.4190.0

I tried removing the preinstalled packages and installing requirements.txt on the server like so:

variable "lambda_root" {
  type = string
  description = "The relative path to the source of the lambda"
  default = "./lambda"
}

resource "null_resource" "install_dependencies" {
  provisioner "local-exec" {
    command = "pip install -r ${var.lambda_root}/requirements.txt -t ${var.lambda_root}/"
  }
  triggers = {
    dependencies_versions = filemd5("${var.lambda_root}/requirements.txt")
    source_versions = filemd5("${var.lambda_root}/pinecone-lambda.py")
  }
}

data "archive_file" "lambda" {
  type        = "zip"
  source_dir = var.lambda_root
  output_path = "lambda_function_payload.zip"
}

but also got an out of memory error that way. I tried to zip it myself and got a smaller file: 290 MB (304,642,515 bytes), but it also failed with the same error. I tried it with 512MB of memory and it still failed. I tried using pip install --only-binary but it made no difference. I tried to download it from S3, but it is apparently too large. That is probably the source of the issue but I would expect a more graceful error from the plugin

│ Error: creating Lambda Function (pinecone-lambda): operation error Lambda: CreateFunction, https response error StatusCode: 400, RequestID: 01ab078c-1344-4b61-8403-d4ad952b637f, InvalidParameterValueException: Unzipped size must be smaller than 262144000 bytes

References

May be related to #29153

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[a087674]

This is not a bug within the provider. per documentation, package lambdas cannot exceed 50MB compressed and 250MB uncompressed. Your zip file is violating both limits.

You can move to image based lambdas by building your own container.

[justinretzolk]

Hey @JonathanRys 👋 I think this might be a combination of a couple of things. There is a bug that we're working to address (hashicorp/aws-sdk-go-base#490) that has caused some out of memory errors like you saw here (#31644, #31570) that I suspect you're also hitting. Based on the above comment, once we've patched that bug, you'll probably start to see other errors with the configuration until you take the actions @a087674 laid out.

With that in mind, would you feel comfortable with us closing this issue out?

[gdavison]

hashicorp/aws-sdk-go-base#490 addresses a problem with large response payloads and logging, so it won't solve this one. This error has to do with serializing the large Zip payload in the request.

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.