Initial implementation #1

aknysh · 2018-07-23T16:21:45Z

what

Terraform module to provision an Elasticsearch cluster in a VPC with build-in Kibana and Logstash

why

Easily provision Elasticsearch cluster with the specified node count in the provided subnets in a VPC
Provision Elasticsearch domain policy that accepts a list of IAM role ARNs from which to permit management traffic to the cluster
Provision Security Group to control access to the Elasticsearch domain (inputs to the Security Group are other Security Groups or CIDRs blocks to be allowed to connect to the cluster)
Provision DNS hostname record for Elasticsearch cluster
DNS hostname record for Kibana

osterman · 2018-07-23T17:23:39Z

README.yaml

+  The subnets must be in different Availability Zones in the same region.
+  If you don't enable zone awareness, Amazon ES places an endpoint into only one subnet
+
+  Further reading:


Move this to references: section

osterman · 2018-07-23T17:24:21Z

main.tf

+  name        = "${module.label.id}"
+  description = "Allow inbound traffic from Security Groups and CIDRs"
+
+  ingress {


Always use aws_security_group_rule for idempotency

osterman · 2018-07-23T17:27:04Z

README.yaml

+description: |-
+  Terraform module to provision an [`Elasticsearch`](https://aws.amazon.com/elasticsearch-service/) cluster with built-in integrations with [Kibana](https://aws.amazon.com/elasticsearch-service/kibana/) and [Logstash](https://aws.amazon.com/elasticsearch-service/logstash/).
+
+  This module will create:


Move this to introduction

osterman · 2018-07-23T17:28:14Z

README.md

+[![Cloud Posse](https://cloudposse.com/logo-300x69.svg)](https://cloudposse.com)
+
+# terraform-aws-elasticsearch [![Build Status](https://travis-ci.org/cloudposse/terraform-aws-elasticsearch.svg?branch=master)](https://travis-ci.org/cloudposse/terraform-aws-elasticsearch) [![Latest Release](https://img.shields.io/github/release/cloudposse/terraform-aws-elasticsearch.svg)](https://github.com/cloudposse/terraform-aws-elasticsearch/releases/latest) [![Slack Community](https://slack.cloudposse.com/badge.svg)](https://slack.cloudposse.com)


Enable CI/CD

osterman · 2018-07-23T18:15:04Z

main.tf

+  enabled   = "${var.enabled == "true" && length(var.dns_zone_id) > 0 ? "true" : "false"}"
+  namespace = "${var.namespace}"
+  stage     = "${var.stage}"
+  name      = "${format("kibana%s%s", var.delimiter, var.name)}"


let's make kibana a variable that defaults to kibana. e.g. it could be overridden to ui

Actually, let's make kibana%s%s a variable

osterman

See inline comments

osterman · 2018-07-23T20:27:28Z

README.yaml

+  - name: "es-createupdatedomains"
+    description: "Creating and Configuring Amazon Elasticsearch Service Domains"
+    url: "https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-createupdatedomains.html"
+  - name: "es-kibana"


Name doesn't need to be codified. It's the name (or title) of the link. We use "codified" names when referring to repositories because that's the easiest way to refer to them. In this case, I would make "Name" the title of the document and a description about why you think it's worth referencing.

osterman · 2018-07-23T20:27:37Z

README.yaml

+  - "docs/terraform.md"
+
+references:
+  - name: "what-is-amazon-elasticsearch-service"


Don't codify

osterman · 2018-07-23T20:27:49Z

README.yaml

+  - name: "what-is-amazon-elasticsearch-service"
+    description: "What is Amazon Elasticsearch Service"
+    url: "https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/what-is-amazon-elasticsearch-service.html"
+  - name: "es-ac"


Don't codify.... etc...

osterman · 2018-07-23T20:28:54Z

main.tf

+  tags        = "${module.label.tags}"
+}
+
+resource "aws_security_group_rule" "ingress_security_groups" {


Seems like we should distinguish between Elastic Search and Kibana.

good pint.
they are separate resources and having different roles to access them would be preferable.

actually, Kibana does not support IAM.
https://forums.aws.amazon.com/thread.jspa?threadID=217149
It's little bit complicated to set up access to Kibana.
The solution is to use a proxy for IP-based control, and Amazon Cognito for use-based control.

https://aws.amazon.com/about-aws/whats-new/2018/04/amazon-elasticsearch-service-simplifies-user-authentication-and-access-for-kibana-with-amazon-cognito/

let's move the decision of who can access what out of the module.
Top-level modules will provide:

variable "iam_role_arns" { type = "list" default = [] description = "List of IAM role ARNs to permit access to the Elasticsearch domain" } variable "iam_actions" { type = "list" default = [] description = "List of actions to allow for the IAM roles, _e.g._ `es:ESHttpGet`, `es:ESHttpPut`, `es:ESHttpPost`" }

osterman · 2018-07-23T20:29:24Z

main.tf

+  tags = "${module.label.tags}"
+}
+
+resource "aws_elasticsearch_domain_policy" "es_vpc_management_access" {


from es_ prefix as we do not need the disambiguation.

osterman

See inline comments

osterman · 2018-07-23T21:08:35Z

README.md

+- [Kibana and Logstash](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-kibana.html) - Describes some considerations for using Kibana and Logstash with Amazon Elasticsearch Service
+- [Amazon Cognito Authentication for Kibana](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-cognito-auth.html) - Amazon Elasticsearch Service uses Amazon Cognito to offer user name and password protection for Kibana
+- [elasticsearch_domain](https://www.terraform.io/docs/providers/aws/r/elasticsearch_domain.html) - Configuration for `elasticsearch_domain` Terraform resource


Terraform reference documentation for the elasticsearch_domain resource

osterman · 2018-07-23T21:08:53Z

README.md

+- [Amazon Cognito Authentication for Kibana](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-cognito-auth.html) - Amazon Elasticsearch Service uses Amazon Cognito to offer user name and password protection for Kibana
+- [elasticsearch_domain](https://www.terraform.io/docs/providers/aws/r/elasticsearch_domain.html) - Configuration for `elasticsearch_domain` Terraform resource
+- [elasticsearch_domain_policy](https://www.terraform.io/docs/providers/aws/r/elasticsearch_domain_policy.html) - Configuration for `elasticsearch_domain_policy` Terraform resource


Terraform reference documentation for the elasticsearch_domain_policy resource

osterman · 2018-07-23T21:44:34Z

Travis is failing

Custom/removebuiltinusers

Initial implementation

0ba2c50

aknysh self-assigned this Jul 23, 2018

aknysh requested review from sarkis, osterman and goruha July 23, 2018 16:21

osterman reviewed Jul 23, 2018

View reviewed changes

goruha previously approved these changes Jul 23, 2018

View reviewed changes

osterman requested changes Jul 23, 2018

View reviewed changes

Address CR comments

b396549

aknysh dismissed goruha’s stale review via b396549 July 23, 2018 20:19

aknysh requested review from osterman and goruha July 23, 2018 20:24

osterman reviewed Jul 23, 2018

View reviewed changes

README.yaml Outdated

- "docs/terraform.md"

references:

- name: "what-is-amazon-elasticsearch-service"

Copy link

Member

osterman Jul 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't codify

osterman reviewed Jul 23, 2018

View reviewed changes

osterman requested changes Jul 23, 2018

View reviewed changes

Address CR comments

c140cff

osterman reviewed Jul 23, 2018

View reviewed changes

aknysh added 3 commits July 23, 2018 17:34

Address CR comments

79be192

Update README

97d1e51

Address CR comments

defa66b

aknysh requested a review from osterman July 23, 2018 21:41

Bump terraform-aws-route53-cluster-hostname version

5851526

osterman approved these changes Jul 24, 2018

View reviewed changes

aknysh merged commit 141a17b into master Jul 24, 2018

aknysh deleted the init branch July 24, 2018 03:12

osterman mentioned this pull request Oct 2, 2018

Only one log type can be chosen #7

Closed

tuxtek referenced this pull request in tuxtek/terraform-aws-elasticsearch Mar 15, 2022

Merge pull request #1 from tuxtek/custom/removebuiltinusers

05c010b

Custom/removebuiltinusers

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial implementation #1

Initial implementation #1

aknysh commented Jul 23, 2018

osterman Jul 23, 2018

osterman Jul 23, 2018

osterman Jul 23, 2018

osterman Jul 23, 2018

osterman Jul 23, 2018

osterman Jul 23, 2018

osterman left a comment

osterman Jul 23, 2018

osterman Jul 23, 2018

osterman Jul 23, 2018

osterman Jul 23, 2018

aknysh Jul 23, 2018

aknysh Jul 23, 2018

aknysh Jul 23, 2018

aknysh Jul 23, 2018

osterman Jul 23, 2018

osterman left a comment

osterman Jul 23, 2018

osterman Jul 23, 2018

osterman commented Jul 23, 2018

		[![Cloud Posse](https://cloudposse.com/logo-300x69.svg)](https://cloudposse.com)

		# terraform-aws-elasticsearch [![Build Status](https://travis-ci.org/cloudposse/terraform-aws-elasticsearch.svg?branch=master)](https://travis-ci.org/cloudposse/terraform-aws-elasticsearch) [![Latest Release](https://img.shields.io/github/release/cloudposse/terraform-aws-elasticsearch.svg)](https://github.com/cloudposse/terraform-aws-elasticsearch/releases/latest) [![Slack Community](https://slack.cloudposse.com/badge.svg)](https://slack.cloudposse.com)

Initial implementation #1

Initial implementation #1

Conversation

aknysh commented Jul 23, 2018

what

why

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

osterman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

osterman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

osterman commented Jul 23, 2018