Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS - Terraform plan with directory services causes panic / crash with segmentation violation code=0x1 addr=0x18 pc=0x40e2395 #17168

Closed
ghost opened this issue Jan 19, 2021 · 6 comments · Fixed by #19820
Labels
bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic.
Milestone

Comments

@ghost
Copy link

ghost commented Jan 19, 2021

This issue was originally opened by @screwnet as hashicorp/terraform#27538. It was migrated here as a result of the provider split. The original body of the issue is below.


Terraform Version

Terraform v0.14.4

Terraform Configuration Files

main.tf

terraform {

  required_version = "~> 0.14.3"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.24"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

data "aws_directory_service_directory" "aws-ad" {
  directory_id = "d-10_char_alphanumeric_id"
}

Debug Output

https://gist.github.com/screwnet/b856bba058839d1d415ec3deb975a1bc

Crash Output

https://gist.github.com/screwnet/c845f79b170f0befa8a02a578edc0fed

Expected Behavior

The code should have executed without crashing during terraform plan operation

Actual Behavior

When:

terraform plan

is executed, the code causes Terraform to crash with panic.

Steps to Reproduce

Copy the terraform code above to a test file and run following commands in the same directory:

terraform init
terraform validate
terraform plan

Additional Context

Any invalid values in the resource properties were inserted as a part of cleaning logs.
The AD directory is shared from a different AWS account (via VPC peering and handshake) and was not created in this account. The executed users had admin privileges on DS, EC2 services in the destination account where the code was run.
All routing, security group and peering is working fine as well as aws cli:

$aws ds describe-directories --directory-ids "d-10_char_alphanumeric_id"

{
    "DirectoryDescriptions": [
        {
            "DirectoryId": "d-10_char_alphanumeric_id",
            "Name": "example.com",
            "ShortName": "EXAMPLE",
            "Size": "Small",
            "Edition": "Standard",
            "Alias": "d-10_char_alphanumeric_id",
            "AccessUrl": "d-10_char_alphanumeric_id.awsapps.com",
            "Stage": "Active",
            "ShareStatus": "Shared",
            "ShareMethod": "HANDSHAKE",
            "LaunchTime": 1606833348.919,
            "StageLastUpdatedDateTime": 1608647353.947,
            "Type": "SharedMicrosoftAD",
            "SsoEnabled": false,
            "DesiredNumberOfDomainControllers": 0,
            "OwnerDirectoryDescription": {
                "DirectoryId": "d-actual_AD_10_char_alphanumeric_id",
                "AccountId": "actual_AD_hosted_AWS_Account_ID",
                "DnsIpAddrs":["192.168.1.1","192.168.2.1"],
                "VpcSettings": {
                    "VpcId": "vpc-actual_AD_hosted_AWS_VPC_IDe",
                    "SubnetIds": [["subnet-sn-id","subnet-sn-id2"],
                    "AvailabilityZones": [
                        "us-east-1a",
                        "us-east-1b"
                    ]
                }
            }
        }
    ]
}

References

@ghost ghost added bug Addresses a defect in current functionality. service/directoryservice labels Jan 19, 2021
@github-actions github-actions bot added the needs-triage Waiting for first response or review from a maintainer. label Jan 19, 2021
@ewbankkit
Copy link
Contributor

ewbankkit commented Jan 19, 2021

2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: panic: runtime error: invalid memory address or nil pointer dereference
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x40e2395]
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: 
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: goroutine 80 [running]:
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: github.com/terraform-providers/terraform-provider-aws/aws.dataSourceAwsDirectoryServiceDirectoryRead(0xc0001b5500, 0x5a6dc40, 0xc001f98000, 0xa59b760, 0x6881b08)
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-aws/aws/data_source_aws_directory_service_directory.go:188 +0xd75
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc0001f5140, 0x71935c0, 0xc0000d8640, 0xc0001b5500, 0x5a6dc40, 0xc001f98000, 0x0, 0x0, 0x0)
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.4.0/helper/schema/resource.go:290 +0x88
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).ReadDataApply(0xc0001f5140, 0x71935c0, 0xc0000d8640, 0xc001af4580, 0x5a6dc40, 0xc001f98000, 0xc001f98000, 0xc001af4580, 0x0, 0x0)
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.4.0/helper/schema/resource.go:498 +0xfd
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadDataSource(0xc000fdab40, 0x71935c0, 0xc0000d8640, 0xc001af4300, 0xc0000d8640, 0x62b54c0, 0x654fee0)
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.4.0/helper/schema/grpc_provider.go:1105 +0x4d4
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).ReadDataSource(0xc0018320e0, 0x71935c0, 0xc0000d8640, 0xc0009dc140, 0xc0018320e0, 0xc00065a210, 0xc00032bba0)
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.1.0/tfprotov5/server/server.go:246 +0xe5
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadDataSource_Handler(0x654fee0, 0xc0018320e0, 0x7193680, 0xc00065a210, 0xc00136c0c0, 0x0, 0x7193680, 0xc00065a210, 0xc001ec40f0, 0xeb)
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.1.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:416 +0x214
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: google.golang.org/grpc.(*Server).processUnaryRPC(0xc000024e00, 0x71b6380, 0xc00104e600, 0xc0002b8300, 0xc00065a810, 0xa55a0d0, 0x0, 0x0, 0x0)
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1194 +0x522
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: google.golang.org/grpc.(*Server).handleStream(0xc000024e00, 0x71b6380, 0xc00104e600, 0xc0002b8300, 0x0)
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1517 +0xd05
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0009d8770, 0xc000024e00, 0x71b6380, 0xc00104e600, 0xc0002b8300)
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:859 +0xa5
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: created by google.golang.org/grpc.(*Server).serveStreams.func1
2021-01-18T21:42:35.676+0530 [DEBUG] plugin.terraform-provider-aws_v3.24.1_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:857 +0x1fd
2021-01-18T21:42:35.679+0530 [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/aws/3.24.1/linux_amd64/terraform-provider-aws_v3.24.1_x5 pid=24436 error="exit status 2"
2021-01-18T21:42:35.679+0530 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"

var securityGroupId *string
if aws.StringValue(dir.Type) == directoryservice.DirectoryTypeAdconnector {
securityGroupId = dir.ConnectSettings.SecurityGroupId
} else {
securityGroupId = dir.VpcSettings.SecurityGroupId
}
d.Set("security_group_id", aws.StringValue(securityGroupId))

@ewbankkit ewbankkit added the crash Results from or addresses a Terraform crash or kernel panic. label Jan 19, 2021
@screwnet
Copy link
Contributor

screwnet commented Jan 20, 2021

Observation:
As this piece of code (describe ds directory) in data was working flawlessly with tested native (non shared) AD in AWS, this seems to be something caused by AWS API call reply for a shared AD. Comparing the code block quoted and my AWS CLI call result, I could see 'VpcSettings.SecurityGroupId' is never returned from AWS for the shared directory as is ConnectSettings.SecurityGroupId. But this is expected at (Line 188)
Because it is a shared directory and AWS is not returning any of the SecurityGroupId members even in their API calls ( aws ds describe-directories api-reference refer the sample output given), we need to have a validation check for directory type?
If the VPC of main directory is not returning enough data, we should set the value as an exception and remove it before returning the API response.

Please let me know your thoughts. I am getting used to Terraform debugging, so there might be some noob faults in the above. Sorry for that.

@bill-rich bill-rich removed the needs-triage Waiting for first response or review from a maintainer. label Jan 22, 2021
@lnpad007
Copy link

I am having a similar issue while using the data source for aws_directory_service_directory that is shared by other aws account. (I am using the shared Directory ID / not the owner's Directory ID)

the setup was tried with below configurations -
provider versions - 3.28.0 / 3.27.0
Terraform Versions - 0.12.29 and 0.12.30

data "aws_directory_service_directory" "my_ads" { directory_id = "d-0000000" }

CRASH LOG

2021-02-19T12:24:40.191+0100 [INFO]  plugin.terraform-provider-aws_v3.28.0_x5: 2021/02/19 12:24:40 [DEBUG] Received DS directory: {
  AccessUrl: "d-000000.awsapps.com",
  Alias: "d-000000",
  DesiredNumberOfDomainControllers: 0,
  DirectoryId: "d-000000",
  Edition: "Standard",
  LaunchTime: 2021-02-18 22:07:18.422 +0000 UTC,
  Name: "my.xxx.local",
  OwnerDirectoryDescription: {
    AccountId: "xxxxxxxx",
    DirectoryId: "d-000000",
    DnsIpAddrs: ["xx.xx.xx.xx","xx.xx.xx.xx"],
    VpcSettings: {
      AvailabilityZones: ["eu-central-1a","eu-central-1b"],
      SubnetIds: ["subnet-xxxxx","subnet-xxxxx"],
      VpcId: "vpc-xxxxx"
    }
  },
  ShareMethod: "HANDSHAKE",
  ShareStatus: "Shared",
  ShortName: "my-ads",
  Size: "Small",
  SsoEnabled: false,
  Stage: "Active",
  StageLastUpdatedDateTime: 2021-02-19 08:57:42.949 +0000 UTC,
  Type: "SharedMicrosoftAD"
}: timestamp=2021-02-19T12:24:40.191+0100
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: panic: runtime error: invalid memory address or nil pointer dereference
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x4d63075]
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: 
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: goroutine 1172 [running]:
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.dataSourceAwsDirectoryServiceDirectoryRead(0xc000db7b80, 0x674f300, 0xc001ed6000, 0xb33adb0, 0x75801a0)
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-aws/aws/data_source_aws_directory_service_directory.go:188 +0xdb5
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc00017fc80, 0x7eb2260, 0xc001b1ca40, 0xc000db7b80, 0x674f300, 0xc001ed6000, 0x0, 0x0, 0x0)
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.4.3/helper/schema/resource.go:290 +0x88
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).ReadDataApply(0xc00017fc80, 0x7eb2260, 0xc001b1ca40, 0xc0017ed600, 0x674f300, 0xc001ed6000, 0xc001ed6000, 0xc0017ed600, 0x0, 0x0)
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.4.3/helper/schema/resource.go:498 +0xfd
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadDataSource(0xc00092e440, 0x7eb2260, 0xc001b1ca40, 0xc0017ed340, 0xc001b1ca40, 0x6fa60a0, 0x7245420)
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.4.3/helper/schema/grpc_provider.go:1105 +0x4d4
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).ReadDataSource(0xc00189d420, 0x7eb2260, 0xc001b1ca40, 0xc001d378b0, 0xc00189d420, 0xc001a5fbf0, 0xc001604ba0)
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.2.1/tfprotov5/server/server.go:247 +0xe5
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadDataSource_Handler(0x7245420, 0xc00189d420, 0x7eb2320, 0xc001a5fbf0, 0xc000c50ae0, 0x0, 0x7eb2320, 0xc001a5fbf0, 0xc0018c6e10, 0xe6)
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.2.1/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:416 +0x214
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001ff880, 0x7ed57c0, 0xc0009b4300, 0xc001415200, 0xc000ddac60, 0xb2f9ad0, 0x0, 0x0, 0x0)
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1194 +0x522
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: google.golang.org/grpc.(*Server).handleStream(0xc0001ff880, 0x7ed57c0, 0xc0009b4300, 0xc001415200, 0x0)
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1517 +0xd05
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000f6e000, 0xc0001ff880, 0x7ed57c0, 0xc0009b4300, 0xc001415200)
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:859 +0xa5
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: created by google.golang.org/grpc.(*Server).serveStreams.func1
2021-02-19T12:24:40.194+0100 [DEBUG] plugin.terraform-provider-aws_v3.28.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:857 +0x1fd
2021/02/19 12:24:40 [ERROR] module.deliver_event_loggroup: eval: *terraform.EvalRefresh, err: rpc error: code = Unavailable desc = transport is closing
2021/02/19 12:24:40 [ERROR] <root>: eval: *terraform.EvalReadData, err: rpc error: code = Unavailable desc = transport is closing
2021/02/19 12:24:40 [ERROR] <root>: eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2021/02/19 12:24:40 [TRACE] [walkRefresh] Exiting eval tree: data.aws_directory_service_directory.bais_ads
2021/02/19 12:24:40 [TRACE] vertex "data.aws_directory_service_directory.my_ads": visit complete
2021-02-19T12:24:40.203+0100 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-02-19T12:24:40.203+0100 [DEBUG] plugin: plugin process exited: path=/myfolder/.terraform/plugins/darwin_amd64/terraform-provider-aws_v3.28.0_x5 pid=94502 error="exit status 2"
2021/02/19 12:24:40 [TRACE] <root>: eval: *terraform.EvalSequence
2021/02/19 12:24:40 [TRACE] <root>: eval: *terraform.EvalGetProvider

@ewbankkit
Copy link
Contributor

It looks like these are shared directories which do not yet have full support (not excusing the crash though).
Related:

@github-actions
Copy link

This functionality has been released in v3.46.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

@github-actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jul 18, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants