Security index and auto expand replicas

[elasticmachine]

Original comment by @jaymode:

When developing the users and roles API, a decision was made to use auto-expand replicas for the .security index so that users/roles were local to every data node ("auto_expand_replicas" : "0-all").

Security also caches users for 20 minutes since the last successful authentication on each node and roles are cached until they have been modified. I feel like we are duplicating efforts here by doing this and should instead use a sane limit for auto-expand replicas. Expanding to all nodes also doesn't really provide much if users are using dedicated client nodes as the user/role still needs to be retrieved from a data node if it is not cached.

So far we have not seen negative side effects from this, but if we decide to use this index for something with a much higher volume such as sessions, then we are adding a lot of load to the cluster since we have to replicate the operation to every data node.

@jasontedor @bleskes curious on your thoughts about this

[elasticmachine]

Original comment by @bleskes:

it sounds reasonable to me - if you we have local node caches, it is redundant to also have local physical copy. Better keep things simple and decide on a comfortable number of replicas (1?) to allow for acceptable data loss. I presume the caches will carry most of the read load and we don't need to worry about that.

[elasticmachine]

Original comment by @clintongormley:

Probably the only downside of not having the .security index on all nodes is that the cluster may not become available until all nodes have been started, which I think is OK

[elasticmachine]

Original comment by @jaymode:

An interesting issue that seems to have cropped up in a customer request is that in a hot/warm architecture, overloaded warm nodes can cause the whole cluster to slow down including security operations. Filtering the allocation of the .security index resolves the poor performance.