Elasticsearch, Cloudformation and ZoneAwareness, the struggle is real

[irvingpop]

AWS recently added 3-AZ zone awareness to Elasticsearch.

This is awesome because previously ZoneAwarenessEnabled forced you into some bad decisions:

1. Run with this value false and you only get good availability at the ES software level, because that requires 3 instances, but they are all in a single zone
   • This is what we do today, and if AWS loses an AZ then you'll have a very bad time
2. Run with this value true and you're forced to specify an even number of ES instances
   • either 2 (which gives you poor availability)
   • or 4 which is super expensive and overkill for us

Amazon provides a good document on the availability trade-offs: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains.html#es-managedomains-multiaz - particularly this table:

Unfortunately, Cloudformation doesn't yet support 3-AZ awareness, so we are forced to wait until it does 😞- keep an eye on this page

[irvingpop]

Once Cloudformation adds support for this, we need to think about the UX for aws_native users. In no particular order:

1. Ensuring (or better documenting) that at least 3 subnets must be passed in
2. Probably hard-coding ChefElasticSearchShardCount to 3, to prevent people from footgunning themselves. An alternative is that we could set AllowedValues to only be multiples of 3 (3,6,9)
3. What happens if someone applies these changes to a live cluster?

• Need to test so we know if the whole cluster is replaced (an outage event requiring manual intervention) or if it does the right thing behind the scenes

[irvingpop]

FWIW, Terraform is also still working on this feature. This will be an interesting test to see who can ship it faster 😄
hashicorp/terraform-provider-aws#8144

[irvingpop]

Adding another point of tracking, there's been an open request for this in the AWS forums since March. Please add your voices here! https://forums.aws.amazon.com/thread.jspa?messageID=905336&#905336

[irvingpop]

An update: the AWS Cloudformation team has opened up their roadmap to public viewing.

As of today, the Cloudformation feature we need is in the "Coming Soon" column: aws-cloudformation/cloudformation-coverage-roadmap#8

[pdohertybcov]

It looks like the change has finally landed.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-elasticsearch-domain-zoneawarenessconfig.html