Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

aws_batch_job_definition crashes terraform when setting timeout.attempt_duration_seconds = null #18066

Closed
jsyrjala opened this issue Mar 12, 2021 · 3 comments · Fixed by #19505
Closed

aws_batch_job_definition crashes terraform when setting timeout.attempt_duration_seconds = null #18066

jsyrjala opened this issue Mar 12, 2021 · 3 comments · Fixed by #19505
Assignees
@ewbankkit
Labels
bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. service/batch Issues and PRs that pertain to the batch service.
Milestone
v3.43.0

Comments

@jsyrjala
Copy link

jsyrjala commented Mar 12, 2021 •
edited
Loading

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

terraform 0.12.29
terraform_provider_aws 3.31.0 (where crash happens)
terraform_provider_aws 2.70.0 (where is no crash)

Affected Resource(s)

  • aws_batch_job_definition

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key: https://keybase.io/hashicorp

I have a module that creates ja aws_batch_job_definition. The timeout parameter is passed as a parameter to the module.
Default value for the timeout is null.

variable "timeout_attempt_duration_seconds" {
  description = "The time duration in seconds after which the job terminates if it has not finished"
  type        = string
  default     = null
}

resource "aws_batch_job_definition" "definition" {
  name                 = var.name
  type                 = "container"
  container_properties = var.container_properties

  parameters = var.parameters

  timeout  {
    attempt_duration_seconds = var.timeout_attempt_duration_seconds
  }
}

Debug Output

Panic Output

Expected Behavior

It should create aws_batch_job_definition and not set timeout for it.

Actual Behavior

With terraform_provider_aws 2.70.0 everything works

With terraform_provider_aws 3.31.0 and setting the timeout to some value => everything works.

With terraform_provider_aws 3.31.0 and not setting the timeout terraform crashed:

module.batch_job.aws_batch_job_definition.definition: Destruction complete after 1s
module.batch_job.aws_batch_job_definition.definition: Creating...

Error: rpc error: code = Unavailable desc = transport is closing



Error: rpc error: code = Unavailable desc = transport is closing



Error: rpc error: code = Unavailable desc = transport is closing


panic: interface conversion: interface {} is nil, not map[string]interface {}
2021-03-12T11:20:02.759+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: 
2021-03-12T11:20:02.759+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: goroutine 136 [running]:
2021-03-12T11:20:02.759+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.expandJobDefinitionTimeout(...)
2021-03-12T11:20:02.759+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-aws/aws/resource_aws_batch_job_definition.go:312
2021-03-12T11:20:02.759+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsBatchJobDefinitionCreate(0xc000ad2280, 0x5bb9780, 0xc001ee8000, 0x0, 0xffffffffffffffff)
2021-03-12T11:20:02.759+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-aws/aws/resource_aws_batch_job_definition.go:140 +0x9e5
2021-03-12T11:20:02.759+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).create(0xc00107d380, 0x7320be8, 0xc0020f40c0, 0xc000ad2280, 0x5bb9780, 0xc001ee8000, 0x0, 0x0, 0x0)
2021-03-12T11:20:02.759+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.4.4/helper/schema/resource.go:278 +0x88
2021-03-12T11:20:02.759+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc00107d380, 0x7320be8, 0xc0020f40c0, 0xc0022bae70, 0xc0020b02c0, 0x5bb9780, 0xc001ee8000, 0xc001998038, 0x59f1bc0, 0xc0021ca840, ...)
2021-03-12T11:20:02.759+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.4.4/helper/schema/resource.go:396 +0x674
2021-03-12T11:20:02.759+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc00000c768, 0x7320be8, 0xc0020f40c0, 0xc0022920a0, 0xc0020f40c0, 0x66b4f00, 0xc0021ca100)
2021-03-12T11:20:02.759+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.4.4/helper/schema/grpc_provider.go:955 +0x8ef
2021-03-12T11:20:02.759+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).ApplyResourceChange(0xc0016b0380, 0x7320c90, 0xc0020f40c0, 0xc0022bad20, 0xc0016b0380, 0xc0021ca150, 0xc0011d7ba0)
2021-03-12T11:20:02.759+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.2.1/tfprotov5/server/server.go:332 +0xb5
2021-03-12T11:20:02.760+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler(0x66b4f00, 0xc0016b0380, 0x7320c90, 0xc0021ca150, 0xc001aa6060, 0x0, 0x7320c90, 0xc0021ca150, 0xc000136700, 0x65f)
2021-03-12T11:20:02.760+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.2.1/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:380 +0x214
2021-03-12T11:20:02.760+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001d3340, 0x733fc18, 0xc0011ce600, 0xc000ae4000, 0xc0011a31d0, 0xa6b8be0, 0x0, 0x0, 0x0)
2021-03-12T11:20:02.760+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1194 +0x52b
2021-03-12T11:20:02.760+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: google.golang.org/grpc.(*Server).handleStream(0xc0001d3340, 0x733fc18, 0xc0011ce600, 0xc000ae4000, 0x0)
2021-03-12T11:20:02.760+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1517 +0xd0c
2021-03-12T11:20:02.760+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0008d4000, 0xc0001d3340, 0x733fc18, 0xc0011ce600, 0xc000ae4000)
2021-03-12T11:20:02.760+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:859 +0xab
2021-03-12T11:20:02.760+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: created by google.golang.org/grpc.(*Server).serveStreams.func1
2021-03-12T11:20:02.760+0200 [DEBUG] plugin.terraform-provider-aws_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:857 +0x1fd
2021/03/12 11:20:02 [DEBUG] aws_security_group.xxx: apply errored, but we're indicating that via the Error pointer rather than returning it: rpc error: code = Unavailable desc = transport is closing
2021/03/12 11:20:02 [TRACE] <root>: eval: *terraform.EvalMaybeTainted
2021/03/12 11:20:02 [TRACE] <root>: eval: *terraform.EvalWriteState
2021/03/12 11:20:02 [TRACE] EvalWriteState: recording 0 dependencies for aws_security_group.xxx
2021/03/12 11:20:02 [TRACE] EvalWriteState: writing current state object for aws_security_group.xxx
2021-03-12T11:20:02.766+0200 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021/03/12 11:20:02 [DEBUG] module.batch_job.aws_batch_job_definition.definition: apply errored, but we're indicating that via the Error pointer rather than returning it: rpc error: code = Unavailable desc = transport is closing
2021/03/12 11:20:02 [TRACE] module.batch_job: eval: *terraform.EvalMaybeTainted
2021/03/12 11:20:02 [TRACE] EvalMaybeTainted: module.batch_job.aws_batch_job_definition.definition encountered an error during creation, so it is now marked as tainted
2021/03/12 11:20:02 [TRACE] module.batch_job: eval: *terraform.EvalWriteState
2021/03/12 11:20:02 [TRACE] EvalWriteState: removing state object for module.batch_job.aws_batch_job_definition.definition
2021/03/12 11:20:02 [TRACE] module.batch_job: eval: *terraform.EvalApplyProvisioners
2021/03/12 11:20:02 [TRACE] EvalApplyProvisioners: aws_batch_job_definition.definition has no state, so skipping provisioners
2021/03/12 11:20:02 [TRACE] module.batch_job: eval: *terraform.EvalMaybeTainted
2021/03/12 11:20:02 [TRACE] EvalMaybeTainted: module.batch_job.aws_batch_job_definition.definition encountered an error during creation, so it is now marked as tainted
2021/03/12 11:20:02 [TRACE] module.batch_job: eval: *terraform.EvalWriteState
2021/03/12 11:20:02 [TRACE] EvalWriteState: removing state object for module.batch_job.aws_batch_job_definition.definition
2021/03/12 11:20:02 [TRACE] module.batch_job: eval: *terraform.EvalIf
2021/03/12 11:20:02 [TRACE] module.batch_job: eval: *terraform.EvalIf
2021/03/12 11:20:02 [TRACE] module.batch_job: eval: *terraform.EvalWriteDiff
2021/03/12 11:20:02 [TRACE] module.batch_job: eval: *terraform.EvalApplyPost
2021/03/12 11:20:02 [ERROR] module.batch_job: eval: *terraform.EvalApplyPost, err: rpc error: code = Unavailable desc = transport is closing
2021/03/12 11:20:02 [ERROR] module.batch_job: eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2021/03/12 11:20:02 [TRACE] [walkApply] Exiting eval tree: module.batch_job.aws_batch_job_definition.definition
2021/03/12 11:20:02 [TRACE] vertex "module.batch_job.aws_batch_job_definition.definition": visit complete
2021/03/12 11:20:02 [DEBUG] aws_security_group.sql-runner: apply errored, but we're indicating that via the Error pointer rather than returning it: rpc error: code = Unavailable desc = transport is closing
2021/03/12 11:20:02 [TRACE] <root>: eval: *terraform.EvalMaybeTainted
2021/03/12 11:20:02 [TRACE] <root>: eval: *terraform.EvalWriteState
2021/03/12 11:20:02 [TRACE] EvalWriteState: recording 0 dependencies for aws_security_group.sql-runner
2021/03/12 11:20:02 [TRACE] EvalWriteState: writing current state object for aws_security_group.sql-runner
2021/03/12 11:20:02 [TRACE] dag/walk: upstream of "module.batch_job.output.job_definition_arn" errored, so skipping
2021/03/12 11:20:02 [TRACE] dag/walk: upstream of "output.batch_job_definition_arn" errored, so skipping
2021-03-12T11:20:02.767+0200 [DEBUG] plugin: plugin process exited: path=/var/lib/jenkins/jobs/sql-runner-deploy/workspace/sql-runner/deployment/terraform/.terraform/plugins/linux_amd64/terraform-provider-aws_v3.31.0_x5 pid=4896 error="exit status 2"
2021/03/12 11:20:02 [TRACE] <root>: eval: *terraform.EvalApplyProvisioners
2021/03/12 11:20:02 [TRACE] EvalApplyProvisioners: aws_security_group.sql-runner is not freshly-created, so no provisioning is required
2021/03/12 11:20:02 [TRACE] <root>: eval: *terraform.EvalMaybeTainted
2021/03/12 11:20:02 [TRACE] <root>: eval: *terraform.EvalWriteState
2021/03/12 11:20:02 [TRACE] EvalWriteState: recording 0 dependencies for aws_security_group.sql-runner
2021/03/12 11:20:02 [TRACE] EvalWriteState: writing current state object for aws_security_group.sql-runner
2021/03/12 11:20:02 [TRACE] <root>: eval: *terraform.EvalIf
2021/03/12 11:20:02 [TRACE] <root>: eval: *terraform.EvalIf
2021/03/12 11:20:02 [TRACE] <root>: eval: *terraform.EvalWriteDiff
2021/03/12 11:20:02 [TRACE] <root>: eval: *terraform.EvalApplyPost
2021/03/12 11:20:02 [ERROR] <root>: eval: *terraform.EvalApplyPost, err: rpc error: code = Unavailable desc = transport is closing
2021/03/12 11:20:02 [TRACE] <root>: eval: *terraform.EvalApplyProvisioners
2021/03/12 11:20:02 [TRACE] EvalApplyProvisioners: aws_security_group.xxx is not freshly-created, so no provisioning is required
2021/03/12 11:20:02 [TRACE] <root>: eval: *terraform.EvalMaybeTainted
2021/03/12 11:20:02 [TRACE] <root>: eval: *terraform.EvalWriteState
2021/03/12 11:20:02 [TRACE] EvalWriteState: recording 0 dependencies for aws_security_group.xxx
2021/03/12 11:20:02 [TRACE] EvalWriteState: writing current state object for aws_security_group.xxx
2021/03/12 11:20:02 [ERROR] <root>: eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2021/03/12 11:20:02 [TRACE] [walkApply] Exiting eval tree: aws_security_group.sql-runner
2021/03/12 11:20:02 [TRACE] vertex "aws_security_group.sql-runner": visit complete
2021/03/12 11:20:02 [TRACE] <root>: eval: *terraform.EvalIf
2021/03/12 11:20:02 [TRACE] <root>: eval: *terraform.EvalIf
2021/03/12 11:20:02 [TRACE] <root>: eval: *terraform.EvalWriteDiff
2021/03/12 11:20:02 [TRACE] <root>: eval: *terraform.EvalApplyPost
2021/03/12 11:20:02 [ERROR] <root>: eval: *terraform.EvalApplyPost, err: rpc error: code = Unavailable desc = transport is closing
2021/03/12 11:20:02 [ERROR] <root>: eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2021/03/12 11:20:02 [TRACE] [walkApply] Exiting eval tree: aws_security_group.xxx
2021/03/12 11:20:02 [TRACE] vertex "aws_security_group.xxx": visit complete
2021/03/12 11:20:02 [TRACE] dag/walk: upstream of "module.compute_environment.var.security_group_ids" errored, so skipping
2021/03/12 11:20:02 [TRACE] dag/walk: upstream of "module.compute_environment.aws_batch_compute_environment.environment" errored, so skipping
2021/03/12 11:20:02 [TRACE] dag/walk: upstream of "module.compute_environment.output.arn" errored, so skipping
2021/03/12 11:20:02 [TRACE] dag/walk: upstream of "module.batch_job.var.compute_environments" errored, so skipping
2021/03/12 11:20:02 [TRACE] dag/walk: upstream of "module.batch_job.aws_batch_job_queue.queue" errored, so skipping
2021/03/12 11:20:02 [TRACE] dag/walk: upstream of "module.batch_job.output.job_queue_arn" errored, so skipping
2021/03/12 11:20:02 [TRACE] dag/walk: upstream of "output.batch_job_queue_arn" errored, so skipping
2021/03/12 11:20:02 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping
2021/03/12 11:20:02 [TRACE] dag/walk: upstream of "provider.aws (close)" errored, so skipping
2021/03/12 11:20:02 [TRACE] dag/walk: upstream of "root" errored, so skipping
2021/03/12 11:20:02 [DEBUG] Uploading remote state to S3: {
  redacted
}
2021-03-12T11:20:02.913+0200 [DEBUG] plugin: plugin exited



!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

SECURITY WARNING: the "crash.log" file that was created may contain 
sensitive information that must be redacted before it is safe to share 
on the issue tracker.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Steps to Reproduce

  1. terraform apply

Important Factoids

References
@ghost ghost added the service/batch Issues and PRs that pertain to the batch service. label Mar 12, 2021
@github-actions github-actions bot added needs-triage Waiting for first response or review from a maintainer. bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. labels Mar 12, 2021
@ewbankkit
Copy link
Contributor

Relevant code:

terraform-provider-aws/aws/resource_aws_batch_job_definition.go

Lines 310 to 312 in b4205b9

func expandJobDefinitionTimeout(item []interface{}) *batch.JobTimeout {
timeout := &batch.JobTimeout{}
data := item[0].(map[string]interface{})

terraform-provider-aws/aws/resource_aws_batch_job_definition.go

Lines 139 to 141 in b4205b9

if v, ok := d.GetOk("timeout"); ok {
input.Timeout = expandJobDefinitionTimeout(v.([]interface{}))
}

Needs length and nil check as described in the Data Handling and Conversion guide.

@ewbankkit ewbankkit removed the needs-triage Waiting for first response or review from a maintainer. label Mar 12, 2021
@ewbankkit ewbankkit self-assigned this May 24, 2021
@ewbankkit ewbankkit mentioned this issue May 24, 2021
Merged
@github-actions github-actions bot added this to the v3.43.0 milestone May 26, 2021
@ghost
Copy link

ghost commented Jun 1, 2021

This has been released in version 3.43.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

@github-actions
Copy link

github-actions bot commented Jul 2, 2021

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jul 2, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@ewbankkit ewbankkit

Labels
bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. service/batch Issues and PRs that pertain to the batch service.
Projects
None yet
Milestone
v3.43.0
2 participants
@jsyrjala @ewbankkit