Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support VPC config for Amazon Kinesis Data Firehose #13269

Merged
merged 7 commits into from
Aug 31, 2020

Conversation

rajholla
Copy link
Contributor

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #13015

Release note for CHANGELOG:

Amazon Kinesis Data Firehose can now deliver streaming data to an Amazon Elasticsearch Service domain in an Amazon VPC.

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccXXX'

make testacc TEST=./aws TESTARGS='-run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates -timeout 120m
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (1677.81s)
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws       1677.859s

terraform-provider-aws git:(firehose-es-vpc-support) ✗ make testacc TEST=./aws TESTARGS='-run=TestAccAWSKinesisFirehoseDeliveryStream'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSKinesisFirehoseDeliveryStream -timeout 120m=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_basic=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_basic
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3basic
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3basic
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_basic
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3basic
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty (121.13s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging (123.65s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource (127.99s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty (140.90s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic (156.63s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_basic (158.61s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty (162.78s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basic (163.71s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty (166.46s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate (167.18s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration (171.60s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update (186.54s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags (193.32s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates (223.99s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled (226.55s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates (229.93s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn (139.98s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty (121.53s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update (160.25s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix (156.87s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE (302.40s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates (180.29s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates (557.15s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates (852.34s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (1830.37s)
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws       1830.450s
...

@rajholla rajholla requested a review from a team May 11, 2020 21:32
@ghost ghost added size/XL Managed by automation to categorize the size of a PR. service/firehose Issues and PRs that pertain to the firehose service. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. documentation Introduces or discusses updates to documentation. needs-triage Waiting for first response or review from a maintainer. labels May 11, 2020
Copy link
Collaborator

@DrFaust92 DrFaust92 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few minor changes.

can you also add an example in the docs for this?

aws/resource_aws_kinesis_firehose_delivery_stream.go Outdated Show resolved Hide resolved
return nil
}
vpcConfig := config[0].(map[string]interface{})
s := vpcConfig["subnet_ids"].(*schema.Set).List()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

helper func expandStringSet() can be used on vpcConfig["subnet_ids"].(*schema.Set) instead

aws/resource_aws_kinesis_firehose_delivery_stream.go Outdated Show resolved Hide resolved
@rajholla
Copy link
Contributor Author

I have addressed the PR comments.

==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates -timeout 120m
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (1605.30s)
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws       1605.359s

@ewbankkit
Copy link
Contributor

Verified acceptance tests:

$ make testacc TEST=./aws/ TESTARGS='-run=TestAccAWSKinesisFirehoseDeliveryStream_basic'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccAWSKinesisFirehoseDeliveryStream_basic -timeout 120m
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_basic
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_basic
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_basic
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_basic (110.48s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	110.514s

$ make testacc TEST=./aws/ TESTARGS='-run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates -timeout 120m
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
--- FAIL: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (2.45s)
    resource_aws_elasticsearch_domain_test.go:991: missing IAM Service Linked Role (es.amazonaws.com), please create it in the AWS account and retry
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	2.487s

$ aws iam create-service-linked-role --aws-service-name es.amazonaws.com

$ make testacc TEST=./aws/ TESTARGS='-run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates -timeout 120m
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (1444.38s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	1444.406s

@ssreechandra
Copy link

when could I expect this PR to be merged ?

@bramalingam81
Copy link

@ewbankkit , Could you please let us know by when the PR would be merged ?

@rajholla
Copy link
Contributor Author

rajholla commented Jun 2, 2020

Hello @ewbankkit @DrFaust92
Can we get some eyes on this PR ? Please let me know if there is anything pending.

@jeffmachado
Copy link

Anxiously waiting for the merge too! :D

@ewbankkit ewbankkit removed the needs-triage Waiting for first response or review from a maintainer. label Jun 9, 2020
@rajholla
Copy link
Contributor Author

Hello @breathingdust
Can you please let me know the pending items here? It has been several weeks since the initial review and looking forward to getting this released.

@breathingdust
Copy link
Member

breathingdust commented Jun 24, 2020

Hi @rajholla 👋

This is in our backlog of items to review and we are hoping to get to it soon. We are currently working through roadmap items for this quarter but once that has been completed we'll be in a better position to give feedback and hopefully merge.

At the moment we can't give you a timeline, but stay tuned for an update. We appreciate your contributions and your patience!

@walterd1969
Copy link

Hello,
Hopefully this is the right place...but I noticed that trying to modify the vpc_arn seems NOT to work.

vpc_config {
  role_arn           = "${module.FlowlogsFirehoseRole.arn}"
  security_group_ids = flatten(["${aws_security_group.firehose_es_sg.id}"])
  subnet_ids         = flatten(["${data.aws_subnet_ids.app1_subnet.ids}"])
}  

change to:

vpc_config {
  role_arn           = "${module.CloudTrailsFirehoseRole.arn}"
  security_group_ids = flatten(["${aws_security_group.firehose_es_sg.id}"])
  subnet_ids         = flatten(["${data.aws_subnet_ids.app1_subnet.ids}"])
}  

The "plan" shows that it will be modified, but after "apply", the change doesn't happen.

@rajholla
Copy link
Contributor Author

Hello,
Hopefully this is the right place...but I noticed that trying to modify the vpc_arn seems NOT to work.

vpc_config {
  role_arn           = "${module.FlowlogsFirehoseRole.arn}"
  security_group_ids = flatten(["${aws_security_group.firehose_es_sg.id}"])
  subnet_ids         = flatten(["${data.aws_subnet_ids.app1_subnet.ids}"])
}  

change to:

vpc_config {
  role_arn           = "${module.CloudTrailsFirehoseRole.arn}"
  security_group_ids = flatten(["${aws_security_group.firehose_es_sg.id}"])
  subnet_ids         = flatten(["${data.aws_subnet_ids.app1_subnet.ids}"])
}  

The "plan" shows that it will be modified, but after "apply", the change doesn't happen.

Hello @walterd1969
I believe you meant changing role_arn is not working.
Unfortunately, AWS API's doesn't allow updating destination VPC connectivity with in ES configuration.

image

@dmccaffery
Copy link

dmccaffery commented Jul 1, 2020

Per documentation:
image

The resource should handle this appropriately with a taint (delete/create) when the vpc_config changes.

According to the code, this should happen:

https://github.com/terraform-providers/terraform-provider-aws/pull/13269/files#diff-6bd9774171ef581867b57e81348d87d8R1297

Not sure why it didn't work as expected for @walterd1969

@lucaschain
Copy link

Hey folks, any updates on this? We've been using the compiled version of this branch and it seems to be working just fine, but we didn't test the resource recreation handling pointed by @dmccaffery

@rajholla
Copy link
Contributor Author

@walterd1969 Thanks for reporting the issue. It should be fixed now. Can you rebuild and test again?
cc @dmccaffery @lucaschain

Copy link
Collaborator

@DrFaust92 DrFaust92 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some changes to tests

aws/resource_aws_kinesis_firehose_delivery_stream_test.go Outdated Show resolved Hide resolved
aws/resource_aws_kinesis_firehose_delivery_stream_test.go Outdated Show resolved Hide resolved
aws/resource_aws_kinesis_firehose_delivery_stream.go Outdated Show resolved Hide resolved
aws/resource_aws_kinesis_firehose_delivery_stream.go Outdated Show resolved Hide resolved
aws/resource_aws_kinesis_firehose_delivery_stream_test.go Outdated Show resolved Hide resolved
aws/resource_aws_kinesis_firehose_delivery_stream_test.go Outdated Show resolved Hide resolved
@rajholla
Copy link
Contributor Author

@DrFaust92 thank you for the review. All comments are addressed.

@valorl
Copy link

valorl commented Jul 21, 2020

@breathingdust Since the roadmap you linked is for August - October and does not include this MR, is it safe to assume that this is not coming before end of October ?

@breathingdust
Copy link
Member

Hey @valorl 👋. The roadmap doesn't represent all that we will do in a quarter, but does represent what we are able to commit to. That said, this issue is on our radar and we hope to have an update for you soon.

@brunorochadasilva
Copy link

Hey guys, any updates about this?

@@ -1275,6 +1291,39 @@ func resourceAwsKinesisFirehoseDeliveryStream() *schema.Resource {
},
},

"vpc_config": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rajholla Could you rename this to vpc_configuration to match the AWS API? Thanks.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ewbankkit vpc_config is consistent with other resources like eks and lambda
Is there any advantage changing this ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For Lambda the AWS API names the field VpcConfig but for EKS it's named ResourceVpcConfig in the API so we are not consistently sticking to the API naming 😄.
Staying with vpc_config is fine.

Comment on lines 235 to 236
"subnet_ids": flattenStringList(description.SubnetIds),
"security_group_ids": flattenStringList(description.SecurityGroupIds),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can use flattenStringSet here for consistency with the expandStringSet below.

@ghost ghost added size/XXL Managed by automation to categorize the size of a PR. and removed size/XL Managed by automation to categorize the size of a PR. labels Aug 25, 2020
@ghost ghost added size/XL Managed by automation to categorize the size of a PR. and removed size/XXL Managed by automation to categorize the size of a PR. labels Aug 25, 2020
@rajholla
Copy link
Contributor Author

@ewbankkit requested changes are in now.

make testacc TEST=./aws TESTARGS='-run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates -timeout 120m
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (2073.09s)
PASS

@ewbankkit
Copy link
Contributor

@DrFaust92 Could you try verifying the TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates acceptance test?
It times out on creation for me in us-west-2.
Thanks.

@ewbankkit
Copy link
Contributor

Verified in us-east-1:

$ AWS_DEFAULT_REGION=us-east-1 make testacc TEST=./aws/ TESTARGS='-run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates -timeout 120m
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (1548.08s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	1548.124s

Copy link
Contributor

@ewbankkit ewbankkit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@DrFaust92
Copy link
Collaborator

ah, im actually getting the following:

    resource_aws_elasticsearch_domain_test.go:1203: missing IAM Service Linked Role (es.amazonaws.com), please create it in the AWS account and retry
--- FAIL: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (3.61s)

does that even make sense?

@ewbankkit
Copy link
Contributor

@DrFaust92 Yes, I got that originally.

aws iam create-service-linked-role --aws-service-name es.amazonaws.com

will fix.

@DrFaust92
Copy link
Collaborator

--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (1609.14s)

us-west-2

Copy link
Collaborator

@DrFaust92 DrFaust92 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bflad bflad added the enhancement Requests to existing resources that expand the functionality or scope. label Aug 28, 2020
@bflad bflad added this to the v3.5.0 milestone Aug 28, 2020
Copy link
Contributor

@bflad bflad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good 🚀

Output from acceptance testing:

--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource (102.19s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags (106.75s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty (118.90s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty (123.17s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basic (133.80s)
--- FAIL: TestAccAWSKinesisFirehoseDeliveryStream_basic (146.83s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty (148.34s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_disappears (149.37s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty (149.70s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update (152.19s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate (163.95s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging (167.63s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic (168.05s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn (173.94s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates (175.01s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty (176.11s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix (180.04s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update (187.17s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled (197.56s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_KinesisStreamSource (98.68s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE (253.21s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration (107.01s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates (139.39s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates (163.46s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates (340.62s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates (991.76s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (1574.83s)

@bflad bflad merged commit 4d3799e into hashicorp:master Aug 31, 2020
bflad added a commit that referenced this pull request Aug 31, 2020
@ghost
Copy link

ghost commented Sep 3, 2020

This has been released in version 3.5.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

@ghost
Copy link

ghost commented Sep 30, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked as resolved and limited conversation to collaborators Sep 30, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
documentation Introduces or discusses updates to documentation. enhancement Requests to existing resources that expand the functionality or scope. service/firehose Issues and PRs that pertain to the firehose service. size/XL Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

r/aws_kinesis_firehose_delivery_stream: Delivery to Amazon Elasticsearch Service domain in VPC