Function not found error is not properly retried when creating an aws_lambda_function

[Veetaha]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

• terraform CLI: v1.1.7
• AWS provider: v4.17.1

Affected Resource(s)

• aws_lambda_function

Terraform Configuration Files

We have the lambda function defined in a shared module, and we have a ton of them. I think this error isn't related to a specific lambda function config. However, here is a snippet of what that resource looks like in our template

resource "aws_lambda_function" "this" {
  function_name = "jobs-status-service-tenant"

  // We deploy Rust-based lambdas with a custom runtime
  runtime = "provided.al2"
  handler = "doesnt.matter"

  memory_size = 512
  timeout = 60 * 15 # seconds

  // There is a dependency on a log group, but I think it's irrelevant for this bug
  // depends_on = [aws_cloudwatch_log_group.lambda_log_group]

  environment {
    variables = {
        ELASTIO_LOG : var.log_filter
        ELASTIO_TASK : "service-tenant",
    }
  }

  tags = {
    "elastio:component"      = "jobs-status",
    "elastio:component_kind" = "handler",
  }

  role             = /* some lambda role we take from other deployment of a CFN stack's outputs */
  source_code_hash = "we-have-a-bug-in-calculating-source-code-hash-so-it-is-incorrect-here-faceplam"

  // This bucket is public
  s3_bucket        = "elastio-artifacts-us-east-2"
  s3_key           = "release/region/0.17.0/red-stack-lambda.zip"
}

Debug Output

  ╷
  │Error: ResourceNotFoundException: Function not found: arn:aws:lambda:us-east-1:REDACTED_ACCOUNT_ID:function:elastio-exp-1655437669-xncuhk-jobs-status-service-tenant
  │{
  │  RespMetadata: {
  │    StatusCode: 404,
  │    RequestID: "f9a85836-c2ba-45c7-96ed-75fef1cd7eaf"
  │  },
  │  Message_: "Function not found: arn:aws:lambda:us-east-1:REDACTED_ACCOUNT_ID:function:elastio-exp-1655437669-xncuhk-jobs-status-service-tenant",
  │  Type: "User"
  │}
  │
  │  with module.jobs_status.module.service_tenant.aws_lambda_function.this,
  │  on ../../../modules/aws_lambda_rust/main.tf line 32, in resource "aws_lambda_function" "this":
  │  32: resource "aws_lambda_function" "this" {
  │
  ╵

Expected Behavior

The lambda function should be deployed successfully on a clean deployment

Actual Behavior

The lambda function fails to be deployed with the error specified in the debug output.

I took a look at CloudTrail and found three GetFunction20150331v2 events for the given function there.
There timestamps were:

• 2022-06-16T21:48:52Z
• 2022-06-16T21:48:39Z
• 2022-06-16T21:48:38Z

It looks like these 3 series with the timeout of 14 seconds was not enough?

Redacted events from CloudTrail

[
    {
        "eventVersion": "1.08",
        "userIdentity": {
            "type": "IAMUser",
            "principalId": "REDACTED",
            "arn": "arn:aws:iam::REDACTED:user/REDACTED",
            "accountId": "REDACTED",
            "accessKeyId": "REDACTED",
            "userName": "REDACTED"
        },
        "eventTime": "2022-06-16T21:48:52Z",
        "eventSource": "lambda.amazonaws.com",
        "eventName": "GetFunction20150331v2",
        "awsRegion": "us-east-1",
        "sourceIPAddress": "REDACTED",
        "userAgent": "APN/1.0 HashiCorp/1.0 Terraform/1.1.7 (+https://www.terraform.io) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.44.25 (go1.17.6; linux; amd64)",
        "requestParameters": {
            "functionName": "elastio-exp-1655437669-xncuhk-jobs-status-service-tenant"
        },
        "responseElements": null,
        "requestID": "REDACTED",
        "eventID": "REDACTED",
        "readOnly": true,
        "eventType": "AwsApiCall",
        "managementEvent": true,
        "recipientAccountId": "REDACTED",
        "eventCategory": "Management"
    },
    {
        "eventVersion": "1.08",
        "userIdentity": {
            "type": "IAMUser",
            "principalId": "REDACTED",
            "arn": "arn:aws:iam::REDACTED:user/REDACTED",
            "accountId": "REDACTED",
            "accessKeyId": "REDACTED",
            "userName": "REDACTED"
        },
        "eventTime": "2022-06-16T21:48:39Z",
        "eventSource": "lambda.amazonaws.com",
        "eventName": "GetFunction20150331v2",
        "awsRegion": "us-east-1",
        "sourceIPAddress": "REDACTED",
        "userAgent": "APN/1.0 HashiCorp/1.0 Terraform/1.1.7 (+https://www.terraform.io) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.44.25 (go1.17.6; linux; amd64)",
        "requestParameters": {
            "functionName": "elastio-exp-1655437669-xncuhk-jobs-status-service-tenant"
        },
        "responseElements": null,
        "requestID": "REDACTED",
        "eventID": "REDACTED",
        "readOnly": true,
        "eventType": "AwsApiCall",
        "managementEvent": true,
        "recipientAccountId": "REDACTED",
        "eventCategory": "Management"
    },
    {
        "eventVersion": "1.08",
        "userIdentity": {
            "type": "IAMUser",
            "principalId": "REDACTED",
            "arn": "arn:aws:iam::REDACTED:user/REDACTED",
            "accountId": "REDACTED",
            "accessKeyId": "REDACTED",
            "userName": "REDACTED"
        },
        "eventTime": "2022-06-16T21:48:38Z",
        "eventSource": "lambda.amazonaws.com",
        "eventName": "GetFunction20150331v2",
        "awsRegion": "us-east-1",
        "sourceIPAddress": "REDACTED",
        "userAgent": "APN/1.0 HashiCorp/1.0 Terraform/1.1.7 (+https://www.terraform.io) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.44.25 (go1.17.6; linux; amd64)",
        "requestParameters": {
            "functionName": "elastio-exp-1655437669-xncuhk-jobs-status-service-tenant"
        },
        "responseElements": null,
        "requestID": "REDACTED",
        "eventID": "REDACTED",
        "readOnly": true,
        "eventType": "AwsApiCall",
        "managementEvent": true,
        "recipientAccountId": "REDACTED",
        "eventCategory": "Management"
    }
]

Steps to Reproduce

Cross your fingers and run terraform apply for a lambda function in a clean environment (nonincremental deployment). If it takes AWS too much time to deploy this function, you will see the Function not found error.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.