elasticsearch-keystore requirement makes it needlesly obtuse to automate

[XANi]

Before (using Puppet to automate):

adding:

• create a template file
• deploy a template to the machine
• make sure only ES can read a file

changing:

• secret key changed ? template will take care of that, trigger any hook if necessary

all of that takes miliseconds for the Configuration Management software to do

After keystore

adding:

• make sure elasticsearch have WRITE PERMISSIONS TO ETC WHICH IT DIDN'T NEED BEFORE
• run a command to add it (~second, coz java)
• run another command just to refresh it, not on the node but on cluster, so additional synchronization is needed

changing:

• ????? not possible to check whether password I want to set is the same or different
• only option is to run add on every CM run, which adds precious seconds to runtime which would be completely unnecessary.
• multiply that by 20 nodes

There is no added security with file that just lays there on the hard drive next to the config and defaults to no password, it's a security theatre. It's same terrible idea Java had for the cert keystore. Don't make it harder for people that can chmod a file on a hard drive. Also, keystore doesn't even create a file with proper permissions in the first place so just to set settings sudo is needed (and hey, it won't even tell you when you accidentally run it as root).

Or take inspiration from old unix daemons that loaded config to memory and then chrooted themselves away from the config dir so in case of security issue the process didn't even had permission to access its own config in the first place

[elasticmachine]

Pinging @elastic/es-security (:Security/Authentication)

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Settings)

[rjernst]

@XANi We appreciate the feedback. Software is hard, and appeasing every user and use case is a balancing act.

We added the keystore intending to make storage of sensitive data more secure, protecting against other potential readers of the data on the same system. Configuration files like elasticsearch.yml are normally world readable, and additionally are in plaintext. While the initial implementation only obfuscated with a null passphrase, that at least protected the sensitive information from being read with something like strings. Earlier this year we added full encryption support, and while users can still use a keystore without a passphrase, the bifurcation of these sensitive and non-sensitive settings is important. It allows us as developers to cleanly separate the constructs used for reading the data, as well as users to not potentially be confused with the same setting existing in two places.

In regards to the troubles you have had with using the keystore within configuration management systems, we understand how frustrating slow processes can be. While editing the keystore is slower than editing a plaintext file, we have taken some recent steps to improve throughput, for example by allowing multiple keys to be set within the same elasticsearch-keystore invocation. We have considered additional features like outputting a hash of the values when listing, or allowing a compare-then-set operation, but overall this isn't going to be much faster than just overwriting all values at once.

Given the above, we don't have any plans at this time to change the keystore behavior, as it is working as intended. As such, I hope you don't mind that I close this issue. We value your feedback, so please continue to open enhancement requests if the issues you have had change or become worse.