Skip to content

Terraform module to provision an Elasticsearch cluster with built-in integrations with Kibana and Logstash.

License

Notifications You must be signed in to change notification settings

Shogan/terraform-aws-elasticsearch

 
 

Repository files navigation

README Header

Cloud Posse

terraform-aws-elasticsearch Codefresh Build Status Latest Release Slack Community

Fork Modifications

** Important Fork modifications **

This Terraform module was forked from the CloudPosse terraform-aws-elasticsearch repository.

The main goal of this fork is to have a module that allows for the creation of AWS ElasticSearch resources in AWS accounts/environments where the Security Group and/or IAM resources have been locked down, or denied access to via corporate / company policy.

Some organisations have teams that manage security and might prevent other teams from modifying or creating Security Groups and/or IAM resources, instead providing them with an interface to create these resources up-front.

This module allows these pre-created resources for IAM / Security Groups to be fed in, instead of having the module create these resources dynamically (where their creation would otherwise fail with those IAM permissions being denied).

These are the modifications that I have made (so far):

  • Allow for passing in pre-defined, pre-created Security Groups for the ElasticSearch domain that is created with this module. Doing this will disable the auto-generated security group creation that the original module does based on hooking up existing SGs to have access to ES via a newly terraformed SG.
  • Allow for passing in of a custom ES domain access policy by data template, instead of relying on IAM roles and IAM resources being dynamically created by the module. If you pass in a custom policy, then the module will not use it's own default version.

Terraform module to provision an Elasticsearch cluster with built-in integrations with Kibana and Logstash.


This project is part of our comprehensive "SweetOps" approach towards DevOps.

Terraform Open Source Modules

It's 100% Open Source and licensed under the APACHE2.

We literally have hundreds of terraform modules that are Open Source and well-maintained. Check them out!

Introduction

This module will create:

  • Elasticsearch cluster with the specified node count in the provided subnets in a VPC
  • Elasticsearch domain policy that accepts a list of IAM role ARNs from which to permit management traffic to the cluster
  • Security Group to control access to the Elasticsearch domain (inputs to the Security Group are other Security Groups or CIDRs blocks to be allowed to connect to the cluster)
  • DNS hostname record for Elasticsearch cluster (if DNS Zone ID is provided)
  • DNS hostname record for Kibana (if DNS Zone ID is provided)

NOTE: To enable zone awareness to deploy Elasticsearch nodes into two different Availability Zones, you need to set zone_awareness_enabled to true and provide two different subnets in subnet_ids. If you enable zone awareness for your domain, Amazon ES places an endpoint into two subnets. The subnets must be in different Availability Zones in the same region. If you don't enable zone awareness, Amazon ES places an endpoint into only one subnet.

Usage

IMPORTANT: The master branch is used in source just as an example. In your code, do not pin to master because there may be breaking changes between releases. Instead pin to the release tag (e.g. ?ref=tags/x.y.z) of one of our latest releases.

Basic example

module "elasticsearch" {
  source                  = "git::https://github.com/cloudposse/terraform-aws-elasticsearch.git?ref=master"
  namespace               = "eg"
  stage                   = "dev"
  name                    = "es"
  dns_zone_id             = "Z14EN2YD427LRQ"
  security_groups         = ["sg-XXXXXXXXX", "sg-YYYYYYYY"]
  vpc_id                  = "vpc-XXXXXXXXX"
  subnet_ids              = ["subnet-XXXXXXXXX", "subnet-YYYYYYYY"]
  zone_awareness_enabled  = "true"
  elasticsearch_version   = "6.5"
  instance_type           = "t2.small.elasticsearch"
  instance_count          = 4
  iam_role_arns           = ["arn:aws:iam::XXXXXXXXX:role/ops", "arn:aws:iam::XXXXXXXXX:role/dev"]
  iam_actions             = ["es:ESHttpGet", "es:ESHttpPut", "es:ESHttpPost"]
  encrypt_at_rest_enabled = true
  kibana_subdomain_name   = "kibana-es"

  advanced_options {
    "rest.action.multi.allow_explicit_index" = "true"
  }
}

Makefile Targets

Available targets:

  help                                Help screen
  help/all                            Display help for all targets
  help/short                          This help short screen
  lint                                Lint terraform code

Inputs

Name Description Type Default Required
advanced_options Key-value string pairs to specify advanced configuration options map(string) <map> no
allowed_cidr_blocks List of CIDR blocks to be allowed to connect to the cluster list(string) <list> no
attributes Additional attributes (e.g. 1) list(string) <list> no
automated_snapshot_start_hour Hour at which automated snapshots are taken, in UTC number 0 no
availability_zone_count Number of Availability Zones for the domain to use. number 2 no
create_iam_service_linked_role Whether to create AWSServiceRoleForAmazonElasticsearchService service-linked role. Set it to false if you already have an ElasticSearch cluster created in the AWS account and AWSServiceRoleForAmazonElasticsearchService already exists. See hashicorp/terraform-provider-aws#5218 for more info bool true no
dedicated_master_count Number of dedicated master nodes in the cluster number 0 no
dedicated_master_enabled Indicates whether dedicated master nodes are enabled for the cluster bool false no
dedicated_master_type Instance type of the dedicated master nodes in the cluster string t2.small.elasticsearch no
delimiter Delimiter to be used between namespace, stage, name and attributes string - no
dns_zone_id Route53 DNS Zone ID to add hostname records for Elasticsearch domain and Kibana string `` no
ebs_iops The baseline input/output (I/O) performance of EBS volumes attached to data nodes. Applicable only for the Provisioned IOPS EBS volume type number 0 no
ebs_volume_size EBS volumes for data storage in GB number 0 no
ebs_volume_type Storage type of EBS volumes string gp2 no
elasticsearch_version Version of Elasticsearch to deploy string 6.5 no
enabled Set to false to prevent the module from creating any resources bool true no
encrypt_at_rest_enabled Whether to enable encryption at rest bool true no
encrypt_at_rest_kms_key_id The KMS key ID to encrypt the Elasticsearch domain with. If not specified, then it defaults to using the AWS/Elasticsearch service KMS key string `` no
iam_actions List of actions to allow for the IAM roles, e.g. es:ESHttpGet, es:ESHttpPut, es:ESHttpPost list(string) <list> no
iam_authorizing_role_arns List of IAM role ARNs to permit to assume the Elasticsearch user role list(string) <list> no
iam_role_arns List of IAM role ARNs to permit access to the Elasticsearch domain list(string) <list> no
instance_count Number of data nodes in the cluster number 4 no
instance_type Elasticsearch instance type for data nodes in the cluster string t2.small.elasticsearch no
kibana_subdomain_name The name of the subdomain for Kibana in the DNS zone (e.g. kibana, ui, ui-es, search-ui, kibana.elasticsearch) string kibana no
log_publishing_application_cloudwatch_log_group_arn ARN of the CloudWatch log group to which log for ES_APPLICATION_LOGS needs to be published string `` no
log_publishing_application_enabled Specifies whether log publishing option for ES_APPLICATION_LOGS is enabled or not bool false no
log_publishing_index_cloudwatch_log_group_arn ARN of the CloudWatch log group to which log for INDEX_SLOW_LOGS needs to be published string `` no
log_publishing_index_enabled Specifies whether log publishing option for INDEX_SLOW_LOGS is enabled or not bool false no
log_publishing_search_cloudwatch_log_group_arn ARN of the CloudWatch log group to which log for SEARCH_SLOW_LOGS needs to be published string `` no
log_publishing_search_enabled Specifies whether log publishing option for SEARCH_SLOW_LOGS is enabled or not bool false no
name Name of the application string - yes
namespace Namespace (e.g. eg or cp) string `` no
node_to_node_encryption_enabled Whether to enable node-to-node encryption bool false no
security_groups List of security group IDs to be allowed to connect to the cluster list(string) <list> no
stage Stage (e.g. prod, dev, staging) string `` no
subnet_ids Subnet IDs list(string) - yes
tags Additional tags (e.g. map(BusinessUnit,XYZ) map(string) <map> no
vpc_id VPC ID string - yes
zone_awareness_enabled Enable zone awareness for Elasticsearch cluster bool true no

Outputs

Name Description
domain_arn ARN of the Elasticsearch domain
domain_endpoint Domain-specific endpoint used to submit index, search, and data upload requests
domain_hostname Elasticsearch domain hostname to submit index, search, and data upload requests
domain_id Unique identifier for the Elasticsearch domain
elasticsearch_user_iam_role_arn The ARN of the IAM role to allow access to Elasticsearch cluster
elasticsearch_user_iam_role_name The name of the IAM role to allow access to Elasticsearch cluster
kibana_endpoint Domain-specific endpoint for Kibana without https scheme
kibana_hostname Kibana hostname
security_group_id Security Group ID to control access to the Elasticsearch domain

References

For additional context, refer to some of these links.

Help

Got a question?

File a GitHub issue, send us an email or join our Slack Community.

README Commercial Support

Commercial Support

Work directly with our team of DevOps experts via email, slack, and video conferencing.

We provide commercial support for all of our Open Source projects. As a Dedicated Support customer, you have access to our team of subject matter experts at a fraction of the cost of a full-time engineer.

E-Mail

  • Questions. We'll use a Shared Slack channel between your team and ours.
  • Troubleshooting. We'll help you triage why things aren't working.
  • Code Reviews. We'll review your Pull Requests and provide constructive feedback.
  • Bug Fixes. We'll rapidly work to fix any bugs in our projects.
  • Build New Terraform Modules. We'll develop original modules to provision infrastructure.
  • Cloud Architecture. We'll assist with your cloud strategy and design.
  • Implementation. We'll provide hands-on support to implement our reference architectures.

Terraform Module Development

Are you interested in custom Terraform module development? Submit your inquiry using our form today and we'll get back to you ASAP.

Slack Community

Join our Open Source Community on Slack. It's FREE for everyone! Our "SweetOps" community is where you get to talk with others who share a similar vision for how to rollout and manage infrastructure. This is the best place to talk shop, ask questions, solicit feedback, and work together as a community to build totally sweet infrastructure.

Newsletter

Signup for our newsletter that covers everything on our technology radar. Receive updates on what we're up to on GitHub as well as awesome new projects we discover.

Contributing

Bug Reports & Feature Requests

Please use the issue tracker to report any bugs or file feature requests.

Developing

If you are interested in being a contributor and want to get involved in developing this project or help out with our other projects, we would love to hear from you! Shoot us an email.

In general, PRs are welcome. We follow the typical "fork-and-pull" Git workflow.

  1. Fork the repo on GitHub
  2. Clone the project to your own machine
  3. Commit changes to your own branch
  4. Push your work back up to your fork
  5. Submit a Pull Request so that we can review your changes

NOTE: Be sure to merge the latest changes from "upstream" before making a pull request!

Copyright

Copyright © 2017-2019 Cloud Posse, LLC

License

License

See LICENSE for full details.

Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.

Trademarks

All other trademarks referenced herein are the property of their respective owners.

About

This project is maintained and funded by Cloud Posse, LLC. Like it? Please let us know by leaving a testimonial!

Cloud Posse

We're a DevOps Professional Services company based in Los Angeles, CA. We ❤️ Open Source Software.

We offer paid support on all of our projects.

Check out our other projects, follow us on twitter, apply for a job, or hire us to help with your cloud strategy and implementation.

Contributors

Erik Osterman
Erik Osterman
Andriy Knysh
Andriy Knysh
Igor Rodionov
Igor Rodionov
Sarkis Varozian
Sarkis Varozian

README Footer Beacon

About

Terraform module to provision an Elasticsearch cluster with built-in integrations with Kibana and Logstash.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • HCL 75.2%
  • Go 12.8%
  • Makefile 12.0%