r/aws_elasticsearch_domain: Add custom endpoint support #16192

matiaszilli · 2020-11-14T08:48:30Z

Add support to define a custom endpoint for your Elasticsearch domain and associate an SSL certificate from AWS ACM.
AWS announcement

The following attributes under domain_endpoint_options were added:

custom_endpoint_enabled
custom_endpoint
custom_endpoint_certificate_arn

resource "aws_elasticsearch_domain" "example" {
  domain_endpoint_options {
    custom_endpoint_enabled         = true
    custom_endpoint                 = "example.com"
    custom_endpoint_certificate_arn = aws_acm_certificate.example.arn
  }
}

Community Note

Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #16059

Release note for CHANGELOG:

resource/aws_elasticsearch_domain: Add custom endpoint support

Output from acceptance testing:

$ make testacc TEST=./aws TESTARGS='-run=TestAccAWSElasticSearchDomain_CustomEndpoint'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSElasticSearchDomain_CustomEndpoint -timeout 120m
=== RUN   TestAccAWSElasticSearchDomain_CustomEndpoint
=== PAUSE TestAccAWSElasticSearchDomain_CustomEndpoint
=== CONT  TestAccAWSElasticSearchDomain_CustomEndpoint
--- PASS: TestAccAWSElasticSearchDomain_CustomEndpoint (1134.57s)
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws       1134.713s

…ult to True

…g_CustomEndpoint

matiaszilli · 2020-11-25T06:09:58Z

Hi @bflad @DrFaust92 , please can anyone from your team revise this PR?. Thanks in advance.

aviau · 2020-11-30T23:53:34Z

Looking forward to see this in the terraform provider! Thank you for working on this :)

nathanwebsterdotme · 2020-12-03T18:06:57Z

Eagerly awaiting this!

kristjankullerkann · 2020-12-07T09:18:36Z

Looking forward for this feature.

sc250024 · 2020-12-07T10:35:37Z

Thank you @matiaszilli

davikawasaki · 2020-12-13T18:00:22Z

Waiting for this feature as well :)

wassertim · 2020-12-23T10:40:19Z

Looking forward for this to be merged

infosec-shinobi · 2021-01-14T13:39:36Z

Similar to other people's comments... Any update on this? Would love to move away from the pre-generated urls for kibana to a custom one via TF.

bm1549 · 2021-01-14T16:40:49Z

Does anyone know if there's someone from AWS, hashicorp, or otherwise that we can @ mention to get some eyes on this?

matiaszilli · 2021-01-14T16:50:20Z

Does anyone know if there's someone from AWS, hashicorp, or otherwise that we can @ mention to get some eyes on this?

I have mentioned three guys above but they did not answer.

smozhevskiiEPAM · 2021-01-27T11:05:52Z

Any news?

sc250024 · 2021-01-27T11:10:39Z

I just checked the most recently merged PRs, and other than @bflad , @anGie44 was also active for approving and merging requests.

@anGie44 Would you mind helping us out please?

DrFaust92 · 2021-01-27T11:58:29Z

Hey, I dont work for Hashi so i dont comment on their behalf but please see https://github.com/hashicorp/terraform-provider-aws/blob/main/docs/FAQ.md and https://github.com/hashicorp/terraform-provider-aws/blob/main/ROADMAP.md for currently planned activities.

Nudging maintainers is not helpful and just makes their(And Mine) notifications noisy.

And ofcourse see the Issue template itself:

Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

mrtnfchs · 2021-03-25T12:22:58Z

Are there any plans to have this change released soon ?

Gowiem · 2021-03-25T13:57:26Z

@mrtnfchs no way of knowing unless it's in the roadmap, which this is not. That means it's not a priority for the coming few months unfortunately.

One thing that can help is sharing this issue with your team / community and getting folks to 👍 it. I believe the hashi team is more likely to evaluate PRs when they're high up in the 👍 reaction list. This one already is, so I'm sure it wouldn't be hard to push it further towards the top.

tomelliff

Bit of a drive by review as I was curious but in general looks good to me (as a non maintainer) at a glance with a couple of nits/questions.

tomelliff · 2021-03-29T14:43:59Z

aws/resource_aws_elasticsearch_domain.go

+							Optional: true,
+							Default:  true,


What was the reasoning for this change? I don't see any explanation in 5217ecb and it looks like an unnecessary change at a glance.

Hi, because it doesn't has to be a necessary attribute to set. For example, you might want to enable a custom endpoint, so you would have to just set custom_endpoint_enabled and custom_endpoint attributes and no more that that.

tomelliff · 2021-03-29T14:52:00Z

aws/resource_aws_elasticsearch_domain.go

+							StateFunc: func(v interface{}) string {
+								// AWS Provider aws_acm_certification.domain_validation_options.resource_record_name
+								// references (and perhaps others) contain a trailing period, requiring a custom StateFunc
+								// to trim the string to prevent Route53 API error
+								value := strings.TrimSuffix(v.(string), ".")
+								return strings.ToLower(value)


This looks like it has been lifted directly from aws/resource_aws_route53_record.go. Is it strictly needed here because of a similar API error on the ES API? Sadly https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-configuration-api.html#es-configuration-api-datatypes-domainendpointoptions leaves any validation undocumented.

https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-customendpoint.html suggests you can leave a trailing dot at the end but the console may just strip that.

I was also wondering if there was a common validate function that checks it's a valid FQDN (max length in total and per label etc) but I don't see one elsewhere in the code base or in the plugin SDK.

At the least the comment here probably wants updating.

You're right, I have just lifted that code from aws/resource_aws_route53_record.go. I was trying to find a common validation function but it doesn't exist, it might be becose the different AWS API endpoints require different validations to be done. I agree with you about the comment update, I will do that. Thank you!

It is generally preferable to leave out these types of StateFunc where possible. If the API has issues with trailing periods, we can introduce validation to try and catch that early.

Perfect @bflad , I will get rid of that validation.

aws/resource_aws_elasticsearch_domain_test.go

website/docs/r/elasticsearch_domain.html.markdown

bflad · 2021-03-31T15:38:18Z

aws/resource_aws_elasticsearch_domain.go

+							StateFunc: func(v interface{}) string {
+								// AWS Provider aws_acm_certification.domain_validation_options.resource_record_name
+								// references (and perhaps others) contain a trailing period, requiring a custom StateFunc
+								// to trim the string to prevent Route53 API error
+								value := strings.TrimSuffix(v.(string), ".")
+								return strings.ToLower(value)


It is generally preferable to leave out these types of StateFunc where possible. If the API has issues with trailing periods, we can introduce validation to try and catch that early.

bflad

Overall this is looking good, just running the full resource acceptance testing now. Leaving these minor items here in case you are updating the pull request already. Thank you, @matiaszilli

Co-authored-by: Brian Flad <bflad417@gmail.com>

aws/resource_aws_elasticsearch_domain.go

bflad

Thanks again, @matiaszilli 🚀

Output from acceptance testing:

--- PASS: TestAccAWSElasticSearchDomain_AdvancedSecurityOptions_Disabled (1110.37s)
--- PASS: TestAccAWSElasticSearchDomain_AdvancedSecurityOptions_IAM (910.89s)
--- PASS: TestAccAWSElasticSearchDomain_AdvancedSecurityOptions_UserDB (2315.72s)
--- PASS: TestAccAWSElasticSearchDomain_basic (828.97s)
--- PASS: TestAccAWSElasticSearchDomain_ClusterConfig_ZoneAwarenessConfig (6476.66s)
--- PASS: TestAccAWSElasticSearchDomain_CognitoOptionsCreateAndRemove (1678.06s)
--- PASS: TestAccAWSElasticSearchDomain_CognitoOptionsUpdate (2566.28s)
--- PASS: TestAccAWSElasticSearchDomain_complex (1058.50s)
--- PASS: TestAccAWSElasticSearchDomain_CustomEndpoint (2500.69s)
--- PASS: TestAccAWSElasticSearchDomain_duplicate (712.87s)
--- PASS: TestAccAWSElasticSearchDomain_encrypt_at_rest_default_key (993.05s)
--- PASS: TestAccAWSElasticSearchDomain_encrypt_at_rest_specify_key (1464.85s)
--- PASS: TestAccAWSElasticSearchDomain_internetToVpcEndpoint (2329.21s)
--- PASS: TestAccAWSElasticSearchDomain_LogPublishingOptions_AuditLogs (1158.98s)
--- PASS: TestAccAWSElasticSearchDomain_LogPublishingOptions_EsApplicationLogs (1519.00s)
--- PASS: TestAccAWSElasticSearchDomain_LogPublishingOptions_IndexSlowLogs (858.37s)
--- PASS: TestAccAWSElasticSearchDomain_LogPublishingOptions_SearchSlowLogs (1081.40s)
--- PASS: TestAccAWSElasticSearchDomain_NodeToNodeEncryption (938.28s)
--- PASS: TestAccAWSElasticSearchDomain_policy (1332.27s)
--- PASS: TestAccAWSElasticSearchDomain_RequireHTTPS (1653.56s)
--- PASS: TestAccAWSElasticSearchDomain_tags (1714.35s)
--- PASS: TestAccAWSElasticSearchDomain_update (2329.18s)
--- PASS: TestAccAWSElasticSearchDomain_update_version (4438.77s)
--- PASS: TestAccAWSElasticSearchDomain_update_volume_type (3179.57s)
--- PASS: TestAccAWSElasticSearchDomain_v23 (1237.71s)
--- PASS: TestAccAWSElasticSearchDomain_vpc (930.36s)
--- PASS: TestAccAWSElasticSearchDomain_vpc_update (3231.96s)
--- PASS: TestAccAWSElasticSearchDomain_warm (6666.88s)
--- PASS: TestAccAWSElasticSearchDomain_withDedicatedMaster (3033.83s)
--- PASS: TestAccAWSElasticSearchDomain_WithVolumeType_Missing (840.98s)

ghost · 2021-04-01T22:56:26Z

This has been released in version 3.35.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

ghost · 2021-05-01T17:10:53Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

r/aws_elasticsearch_domain: Change schema for domain_endpoint_options

34e2ebf

ghost added size/S Managed by automation to categorize the size of a PR. service/elasticsearch Issues and PRs that pertain to the elasticsearch service. labels Nov 14, 2020

github-actions bot added the needs-triage Waiting for first response or review from a maintainer. label Nov 14, 2020

matiaszilli added 4 commits November 18, 2020 12:28

r/aws_elasticsearch_domain: Change structure to include ESCustomEndpoint

8250597

r/aws_elasticsearch_domain: Add custom_endpoint validators

5e7841b

r/aws_elasticsearch_domain: Change enforce_https to Optional and Defa…

5217ecb

…ult to True

r/aws_elasticsearch_domain: Add testAccESDomainConfig_CustomEndpoint

6522655

ghost added size/L Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. and removed size/S Managed by automation to categorize the size of a PR. labels Nov 18, 2020

matiaszilli added 2 commits November 18, 2020 14:33

r/aws_elasticsearch_domain: Fix tfformat on test testAccESDomainConfi…

f0aceec

…g_CustomEndpoint

r/aws_elasticsearch_domain: Update docs under domain_endpoint_options

cffafc4

matiaszilli changed the title ~~[WIP] r/aws_elasticsearch_domain: Add custom endpoint support~~ r/aws_elasticsearch_domain: Add custom endpoint support Nov 18, 2020

matiaszilli marked this pull request as ready for review November 18, 2020 17:19

matiaszilli requested a review from a team as a code owner November 18, 2020 17:19

Base automatically changed from master to main January 23, 2021 00:59

tomelliff reviewed Mar 29, 2021

View reviewed changes

bflad added enhancement Requests to existing resources that expand the functionality or scope. and removed needs-triage Waiting for first response or review from a maintainer. labels Mar 31, 2021

bflad self-assigned this Mar 31, 2021

bflad reviewed Mar 31, 2021

View reviewed changes

r/aws_elasticsearch_domain: Add newer linter check compatibility

02f9366

Co-authored-by: Brian Flad <bflad417@gmail.com>

ghost added the documentation Introduces or discusses updates to documentation. label Mar 31, 2021

matiaszilli and others added 2 commits March 31, 2021 19:53

r/aws_elasticsearch_domain: add variable default value documentation

9cb98eb

Co-authored-by: Brian Flad <bflad417@gmail.com>

r/aws_elasticsearch_domain: Delete custom_endpoint validation

5e686ec

bflad reviewed Mar 31, 2021

View reviewed changes

aws/resource_aws_elasticsearch_domain.go Outdated Show resolved Hide resolved

Update aws/resource_aws_elasticsearch_domain.go

b3102a5

bflad approved these changes Mar 31, 2021

View reviewed changes

bflad merged commit 4b1eeff into hashicorp:main Mar 31, 2021

github-actions bot added this to the v3.35.0 milestone Mar 31, 2021

bflad added a commit that referenced this pull request Mar 31, 2021

Update CHANGELOG for #16192

8a7665f

This was referenced Apr 12, 2021

feat: support custom endpoint configuration gberenice/terraform-aws-elasticsearch#1

Merged

feat: support custom enpoint configuration cloudposse/terraform-aws-elasticsearch#105

Merged

ghost locked as resolved and limited conversation to collaborators May 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

r/aws_elasticsearch_domain: Add custom endpoint support #16192

r/aws_elasticsearch_domain: Add custom endpoint support #16192

matiaszilli commented Nov 14, 2020 •

edited

Loading

matiaszilli commented Nov 25, 2020 •

edited

Loading

aviau commented Nov 30, 2020

nathanwebsterdotme commented Dec 3, 2020

kristjankullerkann commented Dec 7, 2020

sc250024 commented Dec 7, 2020

davikawasaki commented Dec 13, 2020

wassertim commented Dec 23, 2020

infosec-shinobi commented Jan 14, 2021

bm1549 commented Jan 14, 2021

matiaszilli commented Jan 14, 2021

smozhevskiiEPAM commented Jan 27, 2021

sc250024 commented Jan 27, 2021

DrFaust92 commented Jan 27, 2021 •

edited

Loading

mrtnfchs commented Mar 25, 2021

Gowiem commented Mar 25, 2021 •

edited

Loading

tomelliff left a comment

tomelliff Mar 29, 2021

matiaszilli Mar 31, 2021

tomelliff Mar 29, 2021

matiaszilli Mar 31, 2021

bflad Mar 31, 2021

matiaszilli Mar 31, 2021

bflad Mar 31, 2021

bflad left a comment

bflad left a comment

ghost commented Apr 1, 2021

ghost commented May 1, 2021

r/aws_elasticsearch_domain: Add custom endpoint support #16192

r/aws_elasticsearch_domain: Add custom endpoint support #16192

Conversation

matiaszilli commented Nov 14, 2020 • edited Loading

Community Note

matiaszilli commented Nov 25, 2020 • edited Loading

aviau commented Nov 30, 2020

nathanwebsterdotme commented Dec 3, 2020

kristjankullerkann commented Dec 7, 2020

sc250024 commented Dec 7, 2020

davikawasaki commented Dec 13, 2020

wassertim commented Dec 23, 2020

infosec-shinobi commented Jan 14, 2021

bm1549 commented Jan 14, 2021

matiaszilli commented Jan 14, 2021

smozhevskiiEPAM commented Jan 27, 2021

sc250024 commented Jan 27, 2021

DrFaust92 commented Jan 27, 2021 • edited Loading

mrtnfchs commented Mar 25, 2021

Gowiem commented Mar 25, 2021 • edited Loading

tomelliff left a comment

Choose a reason for hiding this comment

tomelliff Mar 29, 2021

Choose a reason for hiding this comment

matiaszilli Mar 31, 2021

Choose a reason for hiding this comment

tomelliff Mar 29, 2021

Choose a reason for hiding this comment

matiaszilli Mar 31, 2021

Choose a reason for hiding this comment

bflad Mar 31, 2021

Choose a reason for hiding this comment

matiaszilli Mar 31, 2021

Choose a reason for hiding this comment

bflad Mar 31, 2021

Choose a reason for hiding this comment

bflad left a comment

Choose a reason for hiding this comment

bflad left a comment

Choose a reason for hiding this comment

ghost commented Apr 1, 2021

ghost commented May 1, 2021

matiaszilli commented Nov 14, 2020 •

edited

Loading

matiaszilli commented Nov 25, 2020 •

edited

Loading

DrFaust92 commented Jan 27, 2021 •

edited

Loading

Gowiem commented Mar 25, 2021 •

edited

Loading