Fix security index auto-create and state recovery race #39582

albertzaharovits · 2019-03-01T19:30:04Z

EDITED:

The security index can be wrongfully recreated, because it was interpreted as missing (as is the case of a fresh install), but the index existed in the state which was not yet recovered.

This fix delays all requests (reads/writes) for the security index until the state has been recovered, as there is no other response we could return, apart from "try-again-later". A cluster with a "state-not-recovered" block is not really usable therefore this state should be transient, hence the option to delay the security requests instead of rejecting them.

Excerpt from https://kibana-ci.elastic.co/job/elastic+kibana+pull-request/3156/JOB=x-pack-ciGroup5,node=immutable/consoleFull

15:55:39      │ info [o.e.x.s.s.SecurityIndexManager] [kibana-ci-immutable-ubuntu-1551446199417305232] security index does not exist. Creating [.security-7] with alias [.security]
15:55:39      │ info [o.e.l.LicenseService] [kibana-ci-immutable-ubuntu-1551446199417305232] license [894371dc-9t49-4997-93cb-802e3d7fa4a8] mode [platinum] - valid
15:55:39      │ info [o.e.g.GatewayService] [kibana-ci-immutable-ubuntu-1551446199417305232] recovered [4] indices into cluster_state
15:55:39      │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-1551446199417305232] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]
15:55:39      │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-1551446199417305232] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]
15:55:39      │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-1551446199417305232] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]
15:55:39      │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-1551446199417305232] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]
15:55:39      │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-1551446199417305232] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]
15:55:39      │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-ubuntu-1551446199417305232] [.security-7] creating index, cause [api], templates [], shards [1]/[0], mappings [doc]
15:55:39      │ info [o.e.c.s.ClusterApplierService] [kibana-ci-immutable-ubuntu-1551446199417305232] failed to notify ClusterStateListener
15:55:39      │      java.lang.IllegalStateException: Alias [.security] points to more than one index: [.security-6, .security-7]
15:55:39      │      	at org.elasticsearch.xpack.security.support.SecurityIndexManager.resolveConcreteIndex(SecurityIndexManager.java:251) ~[?:?]
15:55:39      │      	at org.elasticsearch.xpack.security.support.SecurityIndexManager.clusterChanged(SecurityIndexManager.java:155) ~[?:?]
15:55:39      │      	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateListeners$6(ClusterApplierService.java:480) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
15:55:39      │      	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) [?:?]
15:55:39      │      	at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) [?:?]
15:55:39      │      	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) [?:?]
15:55:39      │      	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:477) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
15:55:39      │      	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:466) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
15:55:39      │      	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:413) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
15:55:39      │      	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:164) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
15:55:39      │      	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
15:55:39      │      	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
15:55:39      │      	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
15:55:39      │      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
15:55:39      │      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
15:55:39      │      	at java.lang.Thread.run(Thread.java:834) [?:?]
15:55:39      │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-1551446199417305232] adding index lifecycle policy [watch-history-ilm-policy]

cc @jbudz

elasticmachine · 2019-03-01T19:30:07Z

Pinging @elastic/es-security

jkakavas

This looks good to me, the build failures are because of an ambitious line that wanted to grow to 141 chars. I'd prefer to revisit this Monday morning with a clear head though.

This can possibly happen on rolling upgrades from 6.7 to 7.x so it's important this makes it to 7.0. I'll add it as a blocker in https://github.com/elastic/dev/issues/1141 but given the schedule I think this can wait for more and fresher 👀 on Monday.

jkakavas · 2019-03-01T21:14:13Z

...in/security/src/main/java/org/elasticsearch/xpack/security/support/SecurityIndexManager.java

@@ -281,7 +290,10 @@ private static Version readMappingVersion(String indexName, MappingMetaData mapp
     */
    public void checkIndexVersionThenExecute(final Consumer<Exception> consumer, final Runnable andThen) {
        final State indexState = this.indexState; // use a local copy so all checks execute against the same state!
-        if (indexState.indexExists && indexState.isIndexUpToDate == false) {
+        if (indexState.concreteIndexName == null) {


use stateRecovered() == false ?

I went with your suggestion, and then reverted. I have kept on checking indexState because this is the "frozen" index state that is used in all if-tests.

jkakavas · 2019-03-01T21:14:25Z

...in/security/src/main/java/org/elasticsearch/xpack/security/support/SecurityIndexManager.java

@@ -297,14 +309,17 @@ public void checkIndexVersionThenExecute(final Consumer<Exception> consumer, fin
    public void prepareIndexIfNeededThenExecute(final Consumer<Exception> consumer, final Runnable andThen) {
        final State indexState = this.indexState; // use a local copy so all checks execute against the same state!
        // TODO we should improve this so we don't fire off a bunch of requests to do the same thing (create or update mappings)
-        if (indexState.indexExists && indexState.isIndexUpToDate == false) {
+        if (indexState.concreteIndexName == null) {


use stateRecovered() == false ?

ditto.

I went with your suggestion, and then reverted. I have kept on checking indexState because this is the "frozen" index state that is used in all if-tests.

…#32340) * [kbn/es] pin 7.x snapshot until #39582 is merged * snapshot needs version number * need to use an older snapshot * who checks todos?

albertzaharovits · 2019-03-03T21:12:37Z

I'll add it as a blocker in elastic/dev#1141

Thanks for the review and care for this one @jkakavas !

albertzaharovits · 2019-03-03T21:18:35Z

...in/security/src/main/java/org/elasticsearch/xpack/security/support/SecurityIndexManager.java

@@ -161,14 +168,17 @@ public void clusterChanged(ClusterChangedEvent event) {
        final Version mappingVersion = oldestIndexMappingVersion(event.state());
        final ClusterHealthStatus indexStatus = indexMetaData == null ? null :
            new ClusterIndexHealth(indexMetaData, event.state().getRoutingTable().index(indexMetaData.getIndex())).getStatus();
+        // index name non-null iff state recovered
        final String concreteIndexName = indexMetaData == null ? INTERNAL_SECURITY_INDEX : indexMetaData.getIndex().getName();


Normally, I would've opted for a new state flag isStateRecovered, but given the amount of flags already existing I think concreteIndexName is a fitting substitute because the index name is also semantically (not only practically) linked to the state recovery status - it's easy to reason that we can't name the security index until the state containing all the index names has been recovered.

I would prefer a new flag.
I think it's too easy to miss that the concreteIndexName variable has additional, non-obvious semantics, and change the default constructor to assign INTERNAL_SECURITY_INDEX to that field.
It's really random that I didn't do that when I introduced the index name field.

As the code stands, you hanging this protection off the fact that we

happen to set that variable to null at construction,

don't (currently) assign a value to it until we get past the STATE_NOT_RECOVERED_BLOCK

always set it to a non-null value once we've recovered.

I don't think you can be sure that we will never change this implementation to behave differently and break one of those pre-conditions.
If we're worried about the number of fields here, we can switch all the booleans to a BitSet, but I don't think it's actually necessary to re-use fields for a purpose other than what they were intended.

tvernum · 2019-03-04T12:50:25Z

I'll add it as a blocker in elastic/dev#1141

If you add a blocker to the description on a release issue, please also make a comment on the issue explaining that you added it so that everyone gets notified.

tvernum

This PR seems to me to be a big change that looks like a small change.
Based on the fact that we're past beta and feature freeze, my inclinaton is that we should simply fix the race condition and continue to let requests fail if they're made before recovery completes.

I could be convinced that it's worth doing if we've taken the necessary steps to be sure this is safe, but I don't think we have.

tvernum · 2019-03-04T13:02:28Z

...in/security/src/main/java/org/elasticsearch/xpack/security/support/SecurityIndexManager.java

@@ -161,14 +168,17 @@ public void clusterChanged(ClusterChangedEvent event) {
        final Version mappingVersion = oldestIndexMappingVersion(event.state());
        final ClusterHealthStatus indexStatus = indexMetaData == null ? null :
            new ClusterIndexHealth(indexMetaData, event.state().getRoutingTable().index(indexMetaData.getIndex())).getStatus();
+        // index name non-null iff state recovered
        final String concreteIndexName = indexMetaData == null ? INTERNAL_SECURITY_INDEX : indexMetaData.getIndex().getName();


I would prefer a new flag.
I think it's too easy to miss that the concreteIndexName variable has additional, non-obvious semantics, and change the default constructor to assign INTERNAL_SECURITY_INDEX to that field.
It's really random that I didn't do that when I introduced the index name field.

As the code stands, you hanging this protection off the fact that we

happen to set that variable to null at construction,

don't (currently) assign a value to it until we get past the STATE_NOT_RECOVERED_BLOCK

always set it to a non-null value once we've recovered.

I don't think you can be sure that we will never change this implementation to behave differently and break one of those pre-conditions.
If we're worried about the number of fields here, we can switch all the booleans to a BitSet, but I don't think it's actually necessary to re-use fields for a purpose other than what they were intended.

tvernum · 2019-03-04T13:09:07Z

...in/security/src/main/java/org/elasticsearch/xpack/security/support/SecurityIndexManager.java

+            // point in time iterator
+            final Iterator<BiConsumer<State, State>> stateListenerIterator = stateChangeListeners.iterator();
+            while (stateListenerIterator.hasNext()) {
+                stateListenerIterator.next().accept(previousState, newState);


As far as I can tell this change doesn't actually do anything.
A for loop is just syntactic sugar over iterator() and next(), you haven't change the semantics. The stateListenerIterator is still backed by the same List and is at risk of ConcurrentModificationException issues if a listener is added/removed.

Assuming that's the problem you're trying to solve, then the options come down to:

synchronize whenever we use the list

switch the list implementation to CopyOnWriteArrayList

make a copy of the list before iterating over it

(2) is probably the best option. I don't think we add listeners very often, so making a list copy each time is probably OK, but you'd need to audit the places we modify the list.

As far as I can tell this change doesn't actually do anything.

It is a rewrite that unwraps the syntactic sugar because I find it confusing to use that when the list is concurrently modified.

The list is already of the CopyOnWriteArrayList type.

tvernum · 2019-03-04T13:21:13Z

...in/security/src/main/java/org/elasticsearch/xpack/security/support/SecurityIndexManager.java

-        if (indexState.indexExists && indexState.isIndexUpToDate == false) {
+        if (indexState.concreteIndexName == null) {
+            // state not yet recovered
+            delayUntilStateRecovered(consumer, () -> checkIndexVersionThenExecute(consumer, andThen));


This worries me.
We're suddenly adding in a queue of pending tasks to run when the recovery block is removed, but I don't see any analysis on how big that queue could get, and what the impact is on the thread pool when it happens.

We don't do that now. We don't queue if the security index is red, we currently fail if the index is not recovered.
There's not enough information in this PR to know whether or not it's a sensible thing to do, and how we'd protect ourselves from getting into trouble with it (e.g. too many queued tasks, or delayed read + writes because recovery was slow).

From a usability point of view it's a nice idea, but it feels like we're jumping in without thinking it through.

Okay, understood! I just wonder if there is some alternative for "delaying requests" that would be acceptable beyond feature freeze?

albertzaharovits · 2019-03-04T14:10:17Z

This PR seems to me to be a big change that looks like a small change.
Based on the fact that we're past beta and feature freeze, my inclinaton is that we should simply fix the race condition and continue to let requests fail if they're made before recovery completes.

Alright, I will refit it to simply reject any request before state recovered. I understand how this might seem like a big change in the end.

I could be convinced that it's worth doing if we've taken the necessary steps to be sure this is safe, but I don't think we have.

What if the implementation would've been throwing listeners over to the generic threadpool. We don't have any queuing rules about limits for this practice, and this implementation is not so very different from the said one. I am not being pushy, but I obviously pondered the alternatives, and I wonder what would be a sketch implementation for the delay-requests option that would be LGTMd after feature freeze, if any. Again the goal is for me to adjust, so that next time I propose the alternative with the most chances of being approved.

Thanks for the feedback Tim!

albertzaharovits · 2019-03-04T17:25:09Z

@jkakavas @tvernum I redid it, so that requests will return HTTP SERVICE_UNAVAILABLE if they try to write to the .security index while the state is not yet recovered.

Please cast your inquisitive eyes anew 🎩

tvernum · 2019-03-05T05:35:36Z

I wonder what would be a sketch implementation for the delay-requests option that would be LGTMd after feature freeze, if any

I'll walk through my thinking. Bear in mind it was about midnight for me, and we're trying to get this sorted quickly so I didn't have time to do any experiments, and I defaulted to the conservative approach. If we had more time to throw around ideas and test things we might be able to find something we were confident in.

Concern 1 - unbounded task queue
Observation: The set of listeners is unbounded, and could grow infinitely large. When the cluster state is recovered all of those tasks will be executed more-or-less simultaneously on the generic thread pool.

Problem A: Is that a reasonable use of the thread pool? Will it starve resources off of other tasks?
Problem B: What if the number of listeners exceeds the capacity of the thread pool? How are rejections handled? What happens to the task that was queued?
Problem C: The set of things these queue tasks will be trying to do are similar. They start with prepareIndexIfNeededThenExecute or checkIndexVersionThenExecute, so we could potentially have a swarm of threads all trying to create/upgrade the index and then read/write to it. How well do we handle that? Will it show up more race conditions in the code?
Problem D: Does using the listener list for this have the potential to cause problems for the other listeners (which are typically trying to clear caches when the index state changes)? Are they going to get delayed, or dropped? Are there risks of mixing those 2 types of listeners into a single list?

Possible Solution: I think this would be better if it had its own queue rather than reusing the listener list. That queue could be bounded, and we could reject new tasks if that queue got too long. It would also mean that we could pull the prepareIndexIfNeededThenExecute / checkIndexVersionThenExecute calls up a level and only execute them once, and then run all the queued tasks.
We could also look at rate limiting the tasks as we execute them, but I'm not sure how that would work.
But, while that seems like a better approach, it doesn't automatically solve all of the issues so it would need more testing and analysis, and it's also a big enough change that I don't think we should rush it in for 7.0

Concern 2 - unbounded delay
Observation: I assume (but I may be wrong) that if you have a multi-master setup and don't have a quorum, that the STATE_NOT_RECOVERED_BLOCK exists until you reach a quorum.
Consequently, a task that is added to the listener list may go unexecuted for an indeterminite length of
time. Quite easily in the 10s of seconds, but potentially 10s of minutes.

Problem A: Is it reasonable to hang on to a task for that long? Is it appropriate to still execute it if it's been queued for a minute or longer? How do we know it's still relevant & safe?
Problem B: What happens to the requests that initiated those reads/writes? Are they also queued? Does the caller get a response, or does their HTTP request just timeout?
Problem C: Given that a very common source of these tasks will be authentication attempts during cluster startup, how will the Rest layer cope if we just suspend every incoming rest request until the cluster state is recovered? Will we eventually exhaust a pool/queue/something there, and block all HTTP access?
Problem D: If we do stop responding to rest requests then a common user behaviour is to just try again. That will cause an ever increasing size in the task queue and in suspended rest requests. That may include multiple requests to do the same thing. When the cluster state recovers they'll all execute at the same time and we will flood other components with a deluge of identical requests.
Problem E: We don't seem to guarantee any order of how these queued requests will exceute. That feels like it will come and bite someone down the track.

Possible Solution: The queue of tasks should have some time limit attached. If we decide to queue them up while the cluster is recovering, they need to have some expiry that eventually responds with SERVICE UNAVAILABLE. I think that would work, but it raises the question of whether it's really worth it.

Since the PR doesn't indicate that those concerns have been considered, and it doesn't really include tests for any of them (the 1 new test covers a very simplistic scenario), and it's fixing an issue that we only found a few days ago and haven't had time to discuss and really think about, it just seemed too risky and a bit rushed.

I think we could come up with a solution that's stable and safe enough. But I think it would need to be more complex than the one proposed here, and at that point I'd question whether that's where we want to spend our engineering efforts & whether the ongoing maintenance cost is justified.
A well written client can handle SERVICE_UNAVAILABLE and try again. If we want to push a custom header that helps them with that we can (I'm not sure what it would be, but if we need it we could).
I feel as though everything that this task queue tries to do can be done client side, so I'm unconvinced that it's something we want to take responsibility for.

Certainly it's not something that I think makes sense to rush in just because there's a deadline coming up, when we can fix the blocking bug much more simply.

...in/security/src/main/java/org/elasticsearch/xpack/security/support/SecurityIndexManager.java

…ecurity/support/SecurityIndexManager.java Co-Authored-By: albertzaharovits <albert.zaharovits@gmail.com>

albertzaharovits · 2019-03-05T07:50:00Z

Thank you for the write-up @tvernum ! I appreciate it!
Note to self: Next time, think of reasons to keep it simple, rather than reasons to make it fancy.

jkakavas

LGTM

Previously, the security index could be wrongfully recreated. This might happen if the index was interpreted as missing, as in the case of a fresh install, but the index existed and the state did not yet recover. This fix will return HTTP SERVICE_UNAVAILABLE (503) for requests that try to write to the security index before the state has not been recovered yet.

* 6.7: (39 commits) Remove beta label from CCR (elastic#39722) Rename retention lease setting (elastic#39719) Add Docker build type (elastic#39378) Use any index specified by .watches for Watcher (elastic#39541) (elastic#39706) Add documentation on remote recovery (elastic#39483) fix typo in synonym graph filter docs Removed incorrect ML YAML tests (elastic#39400) Improved Terms Aggregation documentation (elastic#38892) Fix Fuzziness#asDistance(String) (elastic#39643) Revert "unmute EvilLoggerTests#testDeprecatedSettings (elastic#38743)" Mute TokenAuthIntegTests.testExpiredTokensDeletedAfterExpiration (elastic#39690) Fix security index auto-create and state recovery race (elastic#39582) [DOCS] Sorts security APIs Check for .watches that wasn't upgraded properly (elastic#39609) Assert recovery done in testDoNotWaitForPendingSeqNo (elastic#39595) [DOCS] Updates API in Watcher transform context (elastic#39540) Fixing the custom object serialization bug in diffable utils. (elastic#39544) mute test SQL: Don't allow inexact fields for MIN/MAX (elastic#39563) Update release notes for 6.7.0 ...

…s merged (elastic#32340)" This reverts commit 5285e8b.

#32580) * Revert "[kbn/es] pin 7.x snapshot until elastic/elasticsearch#39582 is merged (#32340)" This reverts commit 5285e8b. * Update config.js

albertzaharovits added 4 commits March 1, 2019 20:24

Main

9c2e1de

test 1

4db7e68

Tests

99a1a05

Merge branch 'master' into fix_security_index_recovery_race

ac82dd4

albertzaharovits added >bug v7.0.0 :Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) v6.7.0 v8.0.0 v7.2.0 labels Mar 1, 2019

albertzaharovits self-assigned this Mar 1, 2019

albertzaharovits requested review from tvernum, jaymode and jkakavas March 1, 2019 19:30

spalger mentioned this pull request Mar 1, 2019

[kbn/es] pin 7.x snapshot until elastic/elasticsearch#39582 is merged elastic/kibana#32340

Merged

jkakavas reviewed Mar 1, 2019

View reviewed changes

albertzaharovits added 3 commits March 3, 2019 22:49

Iter

f734038

Iter2

1cd74ec

back to the frozen state

6363365

albertzaharovits commented Mar 3, 2019

View reviewed changes

tvernum requested changes Mar 4, 2019

View reviewed changes

albertzaharovits added 2 commits March 4, 2019 17:42

Revert

b358996

Done

ea90cc1

Checkstyle

0851725

tvernum approved these changes Mar 5, 2019

View reviewed changes

...in/security/src/main/java/org/elasticsearch/xpack/security/support/SecurityIndexManager.java Outdated Show resolved Hide resolved

tvernum and others added 2 commits March 5, 2019 09:26

Update x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/s…

9146fdb

…ecurity/support/SecurityIndexManager.java Co-Authored-By: albertzaharovits <albert.zaharovits@gmail.com>

Checkstyle

974fc47

jkakavas approved these changes Mar 5, 2019

View reviewed changes

albertzaharovits merged commit fd22c80 into elastic:master Mar 5, 2019

albertzaharovits deleted the fix_security_index_recovery_race branch March 5, 2019 10:13

jbudz added a commit to jbudz/kibana that referenced this pull request Mar 6, 2019

Revert "[kbn/es] pin 7.x snapshot until elastic/elasticsearch#39582 i…

62f90b5

…s merged (elastic#32340)" This reverts commit 5285e8b.

jakelandis added v7.0.0-rc2 and removed v7.0.0 labels Apr 3, 2019

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix security index auto-create and state recovery race #39582

Fix security index auto-create and state recovery race #39582

albertzaharovits commented Mar 1, 2019 •

edited

Loading

elasticmachine commented Mar 1, 2019

jkakavas left a comment

jkakavas Mar 1, 2019

albertzaharovits Mar 3, 2019 •

edited

Loading

jkakavas Mar 1, 2019

albertzaharovits Mar 3, 2019 •

edited

Loading

albertzaharovits commented Mar 3, 2019

albertzaharovits Mar 3, 2019

tvernum Mar 4, 2019

tvernum commented Mar 4, 2019

tvernum left a comment

tvernum Mar 4, 2019

tvernum Mar 4, 2019

albertzaharovits Mar 4, 2019

tvernum Mar 4, 2019

albertzaharovits Mar 4, 2019

albertzaharovits commented Mar 4, 2019

albertzaharovits commented Mar 4, 2019

tvernum commented Mar 5, 2019

albertzaharovits commented Mar 5, 2019

jkakavas left a comment

Fix security index auto-create and state recovery race #39582

Fix security index auto-create and state recovery race #39582

Conversation

albertzaharovits commented Mar 1, 2019 • edited Loading

elasticmachine commented Mar 1, 2019

jkakavas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

albertzaharovits Mar 3, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

albertzaharovits Mar 3, 2019 • edited Loading

Choose a reason for hiding this comment

albertzaharovits commented Mar 3, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tvernum commented Mar 4, 2019

tvernum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

albertzaharovits commented Mar 4, 2019

albertzaharovits commented Mar 4, 2019

tvernum commented Mar 5, 2019

albertzaharovits commented Mar 5, 2019

jkakavas left a comment

Choose a reason for hiding this comment

albertzaharovits commented Mar 1, 2019 •

edited

Loading

albertzaharovits Mar 3, 2019 •

edited

Loading

albertzaharovits Mar 3, 2019 •

edited

Loading