Concurrency error for AWS API Gateway Resources

[martinezp]

I am trying to achieve a way to create a boilerplate code with a set of default error responses I can use for all my end-points in aws.

By creating an error submodule I can call passing the status code as a parameter. And a module creating several instances of that submodule. I expect to have the ability to recreate all the potential responses from my lambda functions in all end-points of my API. Without duplicate the code.

Unfortunately, I couldn't find a way to establish a dependency hierarchy within the modules that saves me from concurrent errors with aws_api_gateway_method_response resource.

Terraform Version

0.9.5

Affected Resource(s)

Module and aws_api_gateway_method_response

If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this.

Terraform Configuration Files

Submodule

variable "rest_api_id" {
}
variable "resource_id" {
}
variable "http_method" {
}
variable "response_models" {
}
variable "status_code" {
}
variable dependency {
}

resource "aws_api_gateway_method_response" "error_response" {
  lifecycle {
    create_before_destroy = true
  }
  rest_api_id = "${var.rest_api_id}"
  resource_id = "${var.resource_id}"
  http_method = "${var.http_method}"
  status_code = "${var.status_code}"
  response_models = {
    "application/json" = "${var.response_models}"
  }
  response_parameters = {
    "method.response.header.Access-Control-Allow-Headers" = true,
    "method.response.header.Access-Control-Allow-Methods" = true,
    "method.response.header.Access-Control-Allow-Origin" = true
  }
}

resource "aws_api_gateway_integration_response" "error_response" {
  lifecycle {
    create_before_destroy = true
  }
  rest_api_id = "${var.rest_api_id}"
  resource_id = "${var.resource_id}"
  http_method = "${var.http_method}"
  status_code = "${aws_api_gateway_method_response.error_response.status_code}"
  selection_pattern = ".*\"httpStatus\":${var.status_code}.*"
  response_templates {
    "application/json" = <<EOF
      #set ($errorMessageObj = $util.parseJson($input.path('$.errorMessage')))
      {
        "code" : "$errorMessageObj.code",
        "message" : "$errorMessageObj.message",
        "request-id" : "$errorMessageObj.requestId"
      }
    EOF
  }
  response_parameters = {
    "method.response.header.Access-Control-Allow-Headers" = "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'",
    "method.response.header.Access-Control-Allow-Methods" = "'POST,OPTIONS,GET,PUT,PATCH,DELETE'",
    "method.response.header.Access-Control-Allow-Origin" = "'*'"
  }
}

Module

variable "rest_api_id" {
}
variable "resource_id" {
}
variable "http_method" {
}
variable "response_models" {
}
module "400" {
  source = "./error"

  rest_api_id = "${var.rest_api_id}"
  resource_id = "${var.resource_id}"
  http_method = "${var.http_method}"
  response_models = "${var.response_models}"
  status_code = "400"
}

module "403" {
  source = "./error"

  rest_api_id = "${var.rest_api_id}"
  resource_id = "${var.resource_id}"
  http_method = "${var.http_method}"
  response_models = "${var.response_models}"
  status_code = "403"
}

module "404" {
  source = "./error"

  rest_api_id = "${var.rest_api_id}"
  resource_id = "${var.resource_id}"
  http_method = "${var.http_method}"
  response_models = "${var.response_models}"
  status_code = "404"
}

module "500" {
  source = "./error"

  rest_api_id = "${var.rest_api_id}"
  resource_id = "${var.resource_id}"
  http_method = "${var.http_method}"
  response_models = "${var.response_models}"
  status_code = "500"
}

Module reference to be used by different end-points

module "get_fixture" {
  source = "./modules/api/responses"

  rest_api_id = "${aws_api_gateway_rest_api.football.id}"
  resource_id = "${aws_api_gateway_resource.fixture_id.id}"
  http_method = "${aws_api_gateway_method.get_fixture.http_method}"
  response_models = "${aws_api_gateway_model.football.name}"
}

Debug Output

1 error(s) occurred:

• module.get_fixtures.module.404.aws_api_gateway_method_response.error_response: 1 error(s) occurred:

• aws_api_gateway_method_response.error_response: Error creating API Gateway Method Response: ConflictException: Unable to complete operation due to concurrent modification. Please try again later.
  status code: 409, request id: 94cb69e3-3a46-11e7-826a-472059cb8059

References

• Depends_on for module #1178, Same concurrency error without using modules
• aws_api_gateway_method_response Concurrency #11395, Module dependencies
• inter module dependency #14432, Module dependencies

[apparentlymart]

Hi @martinezp! Sorry for the problems here.

The API gateway API is rather notorious for this sort of problem unfortunately, and many other programs trying to automate things with this API have run into this sort of problem.

While in the long run we would like to have better ways to control the sequencing of actions across modules, I think the better way to address this immediate concern (following the example of many other projects using this API) is to introduce some mutex locking for the API gateway resources so that we force these writes to be serialized. Terraform's features would still need to be used to control the ordering of these serialized operations, but we'd guarantee then that two mutually-non-dependent resources would still not try to create or modify at the same time.

So I'm going to re-tag this as an AWS provider bug, since I think we can get to a solution much more easily that way than blocking on some still-rather-hypothetical core Terraform features.

[JBirdVegas]

@apparentlymart, I have a similar issue; thanks for the hints but would you mind sharing a link to an example of such a mutex in action?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.