Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploy elasticsearch with terraform #1229

Merged
merged 1 commit into from
May 14, 2018
Merged

Conversation

xbrianh
Copy link
Member

@xbrianh xbrianh commented May 10, 2018

No description provided.

@xbrianh xbrianh self-assigned this May 10, 2018
Copy link
Contributor

@hannes-ucsc hannes-ucsc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the list of source IPs in the policy should be a secret or even of concern here. I'd be ok with having Terraform resetting the IP list in the policy to empty on every deployment. I'd rather re-add my IP in the IAM policy than dig up this secret, modify it and then redeploy. If people modify the policy directly instead of the secret, their changes will be clobbered anyhow by me following the procedure. So what people will end up doing is to always modify the policy instead of the secret and the secret becomes obsolete. Maybe we're ok with that.

Is there a way to tell Terraform to not touch the policy on redeployments?

BTW: You can use scripts/kibana_proxy.py to access Kibana without making any modifications to the ES instance in AWS.

@xbrianh
Copy link
Member Author

xbrianh commented May 11, 2018

Infrastructure managed by Terraform should be updated with Terraform, not with the console.

If we don't want ES access IPs to appear in the repo, Secrets Manager seems like a decent solution. The procedure to update access IPs (update the secret and run make deploy-infra), is more awkward than before, unfortunately.

@hannes-ucsc
Copy link
Contributor

Infrastructure managed by Terraform should be updated with Terraform, not with the console.

Can you explain why we need to be dogmatic about this? The approach I propose seems to be more practical.

If we don't want ES access IPs to appear in the repo

I agree that we don't. Not because they are secret, but because they change all the time.

Also, let me ask again: Is there a way to tell Terraform to not touch the policy on redeployments?

@xbrianh
Copy link
Member Author

xbrianh commented May 11, 2018

If changes are made outside of Terraform, the state file becomes out-of-date.

IMO if we want to "deploy and forget", or "deploy and manage elsewhere", we should not use Terraform.

@hannes-ucsc
Copy link
Contributor

If changes are made outside of Terraform, the state file becomes out-of-date.

And why is that a problem?

Also, let me ask again: Is there a way to tell Terraform to not touch the policy on redeployments?

@hannes-ucsc
Copy link
Contributor

hannes-ucsc commented May 11, 2018

To formulate the question differently: I'v noticed that the policy is optional in the TF module for ES. If you omit the policy from the TF config, will the state file contain a value for the policy? If it doesn't, then the state file would not become stale by users modifying the policy through the console or the CLI.

From a birds eye perspective, I would be surprised if TF had this black or white approach to managed resources: either a resource is managed by TF and you can't configure any aspect of it through any other means than TF. That seems improbable but I don't know TF at all.

@mweiden
Copy link
Contributor

mweiden commented May 14, 2018

@hannes-ucsc Just want to add a datapoint in weighing the alternatives here: to my understanding, you don't have to redeploy to change the access policy on a terraform-managed ES cluster. Doing a terraform apply will simply change the access policy if that is the only difference in configuration.

Copy link
Contributor

@mweiden mweiden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm agnostic to the IP discussion. The rest here LGTM.

@xbrianh
Copy link
Member Author

xbrianh commented May 14, 2018

The ES access policy may be deployable externally to TF, but it seems unclear that a hybrid deployment is better than managing a list IPs through secrets manager.

Moving forward for now.

@xbrianh xbrianh merged commit 562a3ca into master May 14, 2018
@xbrianh
Copy link
Member Author

xbrianh commented May 14, 2018

Connects to #856

@xbrianh xbrianh deleted the bhannafi-infra-elasticsearch branch May 14, 2018 21:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants