Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

F msk-vpc-connection #32775

Merged
merged 50 commits into from
Aug 10, 2023
Merged

Conversation

markoskandylis
Copy link
Contributor

Description

Adds support for MSK VPC-Connection

Relations

Relates #31062

References

go reference: https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/kafka
api reference: https://docs.aws.amazon.com/msk/1.0/apireference/vpc-connection.html

Output from Acceptance Testing

make testacc TESTARGS='-run=TestAccKafkaVPCConnection_' PKG=kafka  ACCTEST_PARALLELISM=2
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 2  -run=TestAccKafkaVPCConnection_ -timeout 180m
=== RUN   TestAccKafkaVPCConnection_basic
=== PAUSE TestAccKafkaVPCConnection_basic
=== RUN   TestAccKafkaVPCConnection_disappears
=== PAUSE TestAccKafkaVPCConnection_disappears
=== CONT  TestAccKafkaVPCConnection_basic
=== CONT  TestAccKafkaVPCConnection_disappears
--- PASS: TestAccKafkaVPCConnection_basic (474.67s)
--- PASS: TestAccKafkaVPCConnection_disappears (494.92s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/kafka	498.030s
...

@github-actions
Copy link

Community Note

Voting for Prioritization

  • Please vote on this pull request by adding a 👍 reaction to the original post to help the community and maintainers prioritize this pull request.
  • Please see our prioritization guide for information on how we prioritize.
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

For Submitters

  • Review the contribution guide relating to the type of change you are making to ensure all of the necessary steps have been taken.
  • For new resources and data sources, use skaff to generate scaffolding with comments detailing common expectations.
  • Whether or not the branch has been rebased will not impact prioritization, but doing so is always a welcome surprise.

@github-actions github-actions bot added size/XL Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. service/kafka Issues and PRs that pertain to the kafka service. client-connections Pertains to the AWS Client and service connections. generators Relates to code generators. needs-triage Waiting for first response or review from a maintainer. partner Contribution from a partner. labels Jul 31, 2023
@github-actions
Copy link

Thank you for your contribution! 🚀

Please note that typically Go dependency changes are handled in this repository by dependabot or the maintainers. This is to prevent pull request merge conflicts and further delay reviews of contributions. Remove any changes to the go.mod or go.sum files and commit them into this pull request.

Additional details:

  • Check open pull requests with the dependencies label to view other dependency updates.
  • If this pull request includes an update the AWS Go SDK (or any other dependency) version, only updates submitted via dependabot will be merged. This pull request will need to remove these changes and will need to be rebased after the existing dependency update via dependabot has been merged for this pull request to be reviewed.
  • If this pull request is for supporting a new AWS service:
    • Ensure the new AWS service changes are following the Contributing Guide section on new services, in particular that the dependency addition and initial provider support are in a separate pull request from other changes (e.g. new resources). Contributions not following this item will not be reviewed until the changes are split.
    • If this pull request is already a separate pull request from the above item, you can ignore this message.

@markoskandylis markoskandylis changed the title tests run succesfully F msk-vpc-connection Jul 31, 2023
@justinretzolk justinretzolk added new-resource Introduces a new resource. and removed needs-triage Waiting for first response or review from a maintainer. labels Jul 31, 2023
@ewbankkit ewbankkit self-assigned this Aug 1, 2023
@github-actions github-actions bot added the prioritized Part of the maintainer teams immediate focus. To be addressed within the current quarter. label Aug 1, 2023
@ewbankkit
Copy link
Contributor

ewbankkit commented Aug 1, 2023

@markoskandylis Thanks for the contribution 👏.

It looks like we'll have to add broker_node_group_info.connectivity_info.vpc_connectivity to the aws_msk_broker resource to get the tests working. Currently:

% make testacc TESTARGS='-run=TestAccKafkaVPCConnection_' PKG=kafka ACCTEST_PARALLELISM=2
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 2  -run=TestAccKafkaVPCConnection_ -timeout 180m
=== RUN   TestAccKafkaVPCConnection_basic
=== PAUSE TestAccKafkaVPCConnection_basic
=== RUN   TestAccKafkaVPCConnection_disappears
=== PAUSE TestAccKafkaVPCConnection_disappears
=== CONT  TestAccKafkaVPCConnection_basic
=== CONT  TestAccKafkaVPCConnection_disappears
    vpc_connection_test.go:59: Step 1/1 error: Error running apply: exit status 1
        
        Error: creating Amazon Managed Streaming for Kafka VPC Connection (): operation error Kafka: CreateVpcConnection, https response error StatusCode: 400, RequestID: 7ec182fa-505a-422c-b1e3-2bc9e207b467, BadRequestException: SASL/IAM client authentication must be enabled for the cluster.
        
          with aws_msk_vpc_connection.test,
          on terraform_plugin_test.tf line 137, in resource "aws_msk_vpc_connection" "test":
         137: resource "aws_msk_vpc_connection" "test" {
        
=== NAME  TestAccKafkaVPCConnection_basic
    vpc_connection_test.go:28: Step 1/2 error: Error running apply: exit status 1
        
        Error: creating Amazon Managed Streaming for Kafka VPC Connection (): operation error Kafka: CreateVpcConnection, https response error StatusCode: 400, RequestID: 7e52bb5b-0039-4484-854f-365076a32429, BadRequestException: SASL/IAM client authentication must be enabled for the cluster.
        
          with aws_msk_vpc_connection.test,
          on terraform_plugin_test.tf line 137, in resource "aws_msk_vpc_connection" "test":
         137: resource "aws_msk_vpc_connection" "test" {
        
--- FAIL: TestAccKafkaVPCConnection_basic (2026.07s)
--- FAIL: TestAccKafkaVPCConnection_disappears (2033.02s)
FAIL
FAIL	github.com/hashicorp/terraform-provider-aws/internal/service/kafka	2038.193s
FAIL
make: *** [testacc] Error 1

TODO

  • aws_msk_broker.broker_node_group_info.connectivity_info.vpc_connectivity
  • aws_msk_vpc_connection.tags

@markoskandylis
Copy link
Contributor Author

Hi @ewbankkit i will check this, i thought it was the policy resource that had to add to make this work :)

@ewbankkit
Copy link
Contributor

I think policy is only required for cross-account connections.

@markoskandylis
Copy link
Contributor Author

markoskandylis commented Aug 1, 2023

Hmm ye probably you are right i will check and fix it tomorrow :)

@ewbankkit
Copy link
Contributor

% ACCTEST_TIMEOUT=720m make testacc TESTARGS='-run=TestAccKafkaCluster_' PKG=kafka ACCTEST_PARALLELISM=3
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 3  -run=TestAccKafkaCluster_ -timeout 720m
=== RUN   TestAccKafkaCluster_basic
=== PAUSE TestAccKafkaCluster_basic
=== RUN   TestAccKafkaCluster_disappears
=== PAUSE TestAccKafkaCluster_disappears
=== RUN   TestAccKafkaCluster_tags
=== PAUSE TestAccKafkaCluster_tags
=== RUN   TestAccKafkaCluster_BrokerNodeGroupInfo_storageInfo
=== PAUSE TestAccKafkaCluster_BrokerNodeGroupInfo_storageInfo
=== RUN   TestAccKafkaCluster_BrokerNodeGroupInfo_instanceType
=== PAUSE TestAccKafkaCluster_BrokerNodeGroupInfo_instanceType
=== RUN   TestAccKafkaCluster_BrokerNodeGroupInfo_publicAccessSASLIAM
=== PAUSE TestAccKafkaCluster_BrokerNodeGroupInfo_publicAccessSASLIAM
=== RUN   TestAccKafkaCluster_BrokerNodeGroupInfo_vpcConnectivity
=== PAUSE TestAccKafkaCluster_BrokerNodeGroupInfo_vpcConnectivity
=== RUN   TestAccKafkaCluster_ClientAuthenticationSASL_scram
=== PAUSE TestAccKafkaCluster_ClientAuthenticationSASL_scram
=== RUN   TestAccKafkaCluster_ClientAuthenticationSASL_iam
=== PAUSE TestAccKafkaCluster_ClientAuthenticationSASL_iam
=== RUN   TestAccKafkaCluster_ClientAuthenticationTLS_certificateAuthorityARNs
=== PAUSE TestAccKafkaCluster_ClientAuthenticationTLS_certificateAuthorityARNs
=== RUN   TestAccKafkaCluster_ClientAuthenticationTLS_initiallyNoAuthentication
=== PAUSE TestAccKafkaCluster_ClientAuthenticationTLS_initiallyNoAuthentication
=== RUN   TestAccKafkaCluster_Info_revision
=== PAUSE TestAccKafkaCluster_Info_revision
=== RUN   TestAccKafkaCluster_EncryptionInfo_encryptionAtRestKMSKeyARN
=== PAUSE TestAccKafkaCluster_EncryptionInfo_encryptionAtRestKMSKeyARN
=== RUN   TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_clientBroker
=== PAUSE TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_clientBroker
=== RUN   TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_inCluster
=== PAUSE TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_inCluster
=== RUN   TestAccKafkaCluster_enhancedMonitoring
=== PAUSE TestAccKafkaCluster_enhancedMonitoring
=== RUN   TestAccKafkaCluster_numberOfBrokerNodes
=== PAUSE TestAccKafkaCluster_numberOfBrokerNodes
=== RUN   TestAccKafkaCluster_openMonitoring
=== PAUSE TestAccKafkaCluster_openMonitoring
=== RUN   TestAccKafkaCluster_storageMode
=== PAUSE TestAccKafkaCluster_storageMode
=== RUN   TestAccKafkaCluster_loggingInfo
=== PAUSE TestAccKafkaCluster_loggingInfo
=== RUN   TestAccKafkaCluster_kafkaVersionUpgrade
=== PAUSE TestAccKafkaCluster_kafkaVersionUpgrade
=== RUN   TestAccKafkaCluster_kafkaVersionDowngrade
=== PAUSE TestAccKafkaCluster_kafkaVersionDowngrade
=== RUN   TestAccKafkaCluster_kafkaVersionUpgradeWithInfo
=== PAUSE TestAccKafkaCluster_kafkaVersionUpgradeWithInfo
=== CONT  TestAccKafkaCluster_basic
=== CONT  TestAccKafkaCluster_EncryptionInfo_encryptionAtRestKMSKeyARN
=== CONT  TestAccKafkaCluster_BrokerNodeGroupInfo_vpcConnectivity
--- PASS: TestAccKafkaCluster_basic (2070.60s)
=== CONT  TestAccKafkaCluster_storageMode
--- PASS: TestAccKafkaCluster_EncryptionInfo_encryptionAtRestKMSKeyARN (2072.89s)
=== CONT  TestAccKafkaCluster_BrokerNodeGroupInfo_storageInfo
--- PASS: TestAccKafkaCluster_storageMode (3035.92s)
=== CONT  TestAccKafkaCluster_kafkaVersionUpgradeWithInfo
--- PASS: TestAccKafkaCluster_BrokerNodeGroupInfo_storageInfo (2247.99s)
=== CONT  TestAccKafkaCluster_BrokerNodeGroupInfo_publicAccessSASLIAM
--- PASS: TestAccKafkaCluster_BrokerNodeGroupInfo_vpcConnectivity (6260.84s)
=== CONT  TestAccKafkaCluster_kafkaVersionDowngrade
--- PASS: TestAccKafkaCluster_kafkaVersionUpgradeWithInfo (6874.66s)
=== CONT  TestAccKafkaCluster_kafkaVersionUpgrade
--- PASS: TestAccKafkaCluster_kafkaVersionDowngrade (4318.38s)
=== CONT  TestAccKafkaCluster_ClientAuthenticationTLS_certificateAuthorityARNs
--- PASS: TestAccKafkaCluster_BrokerNodeGroupInfo_publicAccessSASLIAM (6293.44s)
=== CONT  TestAccKafkaCluster_BrokerNodeGroupInfo_instanceType
--- PASS: TestAccKafkaCluster_ClientAuthenticationTLS_certificateAuthorityARNs (2231.34s)
=== CONT  TestAccKafkaCluster_loggingInfo
--- PASS: TestAccKafkaCluster_loggingInfo (2770.44s)
=== CONT  TestAccKafkaCluster_Info_revision
--- PASS: TestAccKafkaCluster_kafkaVersionUpgrade (6817.77s)
=== CONT  TestAccKafkaCluster_ClientAuthenticationTLS_initiallyNoAuthentication
--- PASS: TestAccKafkaCluster_BrokerNodeGroupInfo_instanceType (6869.62s)
=== CONT  TestAccKafkaCluster_enhancedMonitoring
--- PASS: TestAccKafkaCluster_Info_revision (3254.13s)
=== CONT  TestAccKafkaCluster_openMonitoring
--- PASS: TestAccKafkaCluster_enhancedMonitoring (2407.59s)
=== CONT  TestAccKafkaCluster_numberOfBrokerNodes
--- PASS: TestAccKafkaCluster_ClientAuthenticationTLS_initiallyNoAuthentication (4826.71s)
=== CONT  TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_inCluster
--- PASS: TestAccKafkaCluster_openMonitoring (3031.32s)
=== CONT  TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_clientBroker
--- PASS: TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_inCluster (2075.66s)
=== CONT  TestAccKafkaCluster_ClientAuthenticationSASL_iam
--- PASS: TestAccKafkaCluster_numberOfBrokerNodes (3280.01s)
=== CONT  TestAccKafkaCluster_ClientAuthenticationSASL_scram
--- PASS: TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_clientBroker (2429.11s)
=== CONT  TestAccKafkaCluster_disappears
--- PASS: TestAccKafkaCluster_disappears (1913.35s)
=== CONT  TestAccKafkaCluster_tags
--- PASS: TestAccKafkaCluster_ClientAuthenticationSASL_scram (3742.33s)
--- PASS: TestAccKafkaCluster_ClientAuthenticationSASL_iam (4224.93s)
--- PASS: TestAccKafkaCluster_tags (2230.81s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/kafka	28445.227s
% ACCTEST_TIMEOUT=720m make testacc TESTARGS='-run=TestAccKafkaVPCConnectionDataSource_\|TestAccKafkaVPCConnection_' PKG=kafka ACCTEST_PARALLELISM=1
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 1  -run=TestAccKafkaVPCConnectionDataSource_\|TestAccKafkaVPCConnection_ -timeout 720m
=== RUN   TestAccKafkaVPCConnectionDataSource_basic
=== PAUSE TestAccKafkaVPCConnectionDataSource_basic
=== RUN   TestAccKafkaVPCConnection_basic
=== PAUSE TestAccKafkaVPCConnection_basic
=== RUN   TestAccKafkaVPCConnection_disappears
=== PAUSE TestAccKafkaVPCConnection_disappears
=== CONT  TestAccKafkaVPCConnectionDataSource_basic
=== CONT  TestAccKafkaVPCConnection_disappears
--- PASS: TestAccKafkaVPCConnectionDataSource_basic (6975.36s)
--- PASS: TestAccKafkaVPCConnection_disappears (6379.56s)
=== CONT  TestAccKafkaVPCConnection_basic
--- PASS: TestAccKafkaVPCConnection_basic (7065.81s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/kafka	20426.234s

Copy link
Contributor

@ewbankkit ewbankkit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀.

% ACCTEST_TIMEOUT=720m make testacc TESTARGS='-run=TestAccKafkaVPCConnection_tags' PKG=kafka ACCTEST_PARALLELISM=1
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 1  -run=TestAccKafkaVPCConnection_tags -timeout 720m
=== RUN   TestAccKafkaVPCConnection_tags
=== PAUSE TestAccKafkaVPCConnection_tags
=== CONT  TestAccKafkaVPCConnection_tags
--- PASS: TestAccKafkaVPCConnection_tags (7365.78s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/kafka	7366.695s

@ewbankkit
Copy link
Contributor

@markoskandylis Thanks for the contribution 🎉 👏.

@ewbankkit ewbankkit merged commit b8ef387 into hashicorp:main Aug 10, 2023
48 checks passed
@github-actions github-actions bot added this to the v5.12.0 milestone Aug 10, 2023
@github-actions
Copy link

This functionality has been released in v5.12.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

@github-actions
Copy link

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 10, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
client-connections Pertains to the AWS Client and service connections. generators Relates to code generators. new-data-source Introduces a new data source. new-resource Introduces a new resource. partner Contribution from a partner. prioritized Part of the maintainer teams immediate focus. To be addressed within the current quarter. service/kafka Issues and PRs that pertain to the kafka service. size/XL Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants