[tests] Security Index unavailable

[Mpdreamz]

Sometimes the security index has unavailable shards

Raised this before: elastic/elasticsearch#45250

Would like to understand if we're doing something particularly destructive in our integration tests this happens

[2019-12-17T08:49:27,039][INFO ][o.e.x.w.WatcherService   ] [xpack-node-d9e48e9200] reloading watcher, reason [new local watcher shard allocation ids], cancelled [0] queued tasks
  X Tests.XPack.Enrich.ExecutePolicy.ExecutePolicyApiTests.ReturnsExpectedStatusCode [3s 159ms]
  Error Message:
   System.Exception : Failure setting up integration test: Invalid NEST response built from a unsuccessful (503) low level call on PUT: /_enrich/policy/nest-fluentasync-c48b1eeb?pretty=true&error_trace=true
# Audit trail of this API call:
 - [1] BadResponse: Node: https://localhost:9200/ Took: 00:00:00.0159654
# OriginalException: Elasticsearch.Net.ElasticsearchClientException: The remote server returned an error: (503) Server Unavailable.. Call: Status code 503 from: PUT /_enrich/policy/nest-fluentasync-c48b1eeb?pretty=true&error_trace=true. ServerError: Type: unavailable_shards_exception Reason: "at least one primary shard for the index [.security-7] is unavailable" ---> System.Net.WebException: The remote server returned an error: (503) Server Unavailable.
   at System.Net.HttpWebRequest.GetResponse() 
   at Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData) in D:\a\1\s\src\Elasticsearch.Net\Connection\HttpWebRequestConnection.cs:line 59
   --- End of inner exception stack trace ---
# Request:
{"match":{"indices":["project"],"match_field":"name","enrich_fields":["description","tags"]}}
# Response:
{
  "error" : {
    "root_cause" : [
      {
        "type" : "unavailable_shards_exception",
        "reason" : "at least one primary shard for the index [.security-7] is unavailable",
        "stack_trace" : "UnavailableShardsException[at least one primary shard for the index [.security-7] is unavailable]\r\n\tat org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:182)\r\n\tat org.elasticsearch.xpack.security.authz.store.NativePrivilegeStore.getPrivileges(NativePrivilegeStore.java:102)\r\n\tat org.elasticsearch.xpack.security.action.user.TransportHasPrivilegesAction.resolveApplicationPrivileges(TransportHasPrivilegesAction.java:89)\r\n\tat org.elasticsearch.xpack.security.action.user.TransportHasPrivilegesAction.doExecute(TransportHasPrivilegesAction.java:81)\r\n\tat org.elasticsearch.xpack.security.action.user.TransportHasPrivilegesAction.doExecute(TransportHasPrivilegesAction.java:37)\r\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:153)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:86)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:172)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$4(AuthorizationService.java:241)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:591)\r\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\r\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:144)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:243)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:227)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:193)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\r\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:117)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roles(CompositeRolesStore.java:152)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:231)\r\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:123)\r\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:111)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:195)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:172)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:158)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:310)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:321)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:245)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:107)\r\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:151)\r\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:129)\r\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64)\r\n\tat org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:94)\r\n\tat org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:83)\r\n\tat org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:396)\r\n\tat org.elasticsearch.xpack.enrich.action.TransportPutEnrichPolicyAction.masterOperation(TransportPutEnrichPolicyAction.java:118)\r\n\tat org.elasticsearch.xpack.enrich.action.TransportPutEnrichPolicyAction.masterOperation(TransportPutEnrichPolicyAction.java:37)\r\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:98)\r\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:169)\r\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)\r\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:169)\r\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:132)\r\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:109)\r\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:58)\r\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:153)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:86)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:172)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$4(AuthorizationService.java:241)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:591)\r\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\r\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:144)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:243)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:227)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:193)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\r\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:117)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$buildThenMaybeCacheRole$10(CompositeRolesStore.java:273)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromDescriptors(CompositeRolesStore.java:396)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildThenMaybeCacheRole(CompositeRolesStore.java:254)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$roles$4(CompositeRolesStore.java:175)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$loadRoleDescriptorsAsync$14(CompositeRolesStore.java:308)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\r\n\tat org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:120)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$loadRoleDescriptorsAsync$17(CompositeRolesStore.java:328)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authz.store.FileRolesStore.accept(FileRolesStore.java:95)\r\n\tat org.elasticsearch.xpack.security.authz.store.FileRolesStore.accept(FileRolesStore.java:57)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$loadRoleDescriptorsAsync$18(CompositeRolesStore.java:314)\r\n\tat org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:117)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$loadRoleDescriptorsAsync$17(CompositeRolesStore.java:328)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.core.security.authz.store.ReservedRolesStore.accept(ReservedRolesStore.java:293)\r\n\tat org.elasticsearch.xpack.core.security.authz.store.ReservedRolesStore.accept(ReservedRolesStore.java:34)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$loadRoleDescriptorsAsync$18(CompositeRolesStore.java:314)\r\n\tat org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:102)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.loadRoleDescriptorsAsync(CompositeRolesStore.java:330)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roleDescriptors(CompositeRolesStore.java:297)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roles(CompositeRolesStore.java:155)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:231)\r\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:123)\r\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:111)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:195)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:172)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:158)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:310)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:321)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:245)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:107)\r\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:151)\r\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:129)\r\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64)\r\n\tat org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:94)\r\n\tat org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:83)\r\n\tat org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:396)\r\n\tat org.elasticsearch.xpack.enrich.rest.RestPutEnrichPolicyAction.lambda$prepareRequest$0(RestPutEnrichPolicyAction.java:32)\r\n\tat org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:108)\r\n\tat org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:63)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$writeAuthToContext$24(AuthenticationService.java:596)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.writeAuthToContext(AuthenticationService.java:610)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.finishAuthentication(AuthenticationService.java:584)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeUser(AuthenticationService.java:531)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$16(AuthenticationService.java:415)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\r\n\tat org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:120)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$13(AuthenticationService.java:385)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$1(CachingUsernamePasswordRealm.java:145)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.handleCachedAuthentication(CachingUsernamePasswordRealm.java:196)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$2(CachingUsernamePasswordRealm.java:137)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)\r\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)\r\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:68)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:132)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:103)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$15(AuthenticationService.java:374)\r\n\tat org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:117)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$13(AuthenticationService.java:399)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$3(CachingUsernamePasswordRealm.java:175)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:91)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:166)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:103)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$15(AuthenticationService.java:374)\r\n\tat org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:102)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:419)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$11(AuthenticationService.java:341)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:351)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:292)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:327)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:272)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:254)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:381)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:250)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:310)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:321)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:245)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:122)\r\n\tat org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:55)\r\n\tat org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:227)\r\n\tat org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:309)\r\n\tat org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:167)\r\n\tat org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:322)\r\n\tat org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:372)\r\n\tat org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:301)\r\n\tat org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:54)\r\n\tat org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:29)\r\n\tat io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\r\n\tat org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\r\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\r\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:326)\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:300)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\r\n\tat io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\r\n\tat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1478)\r\n\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227)\r\n\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503)\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\r\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\r\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)\r\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:600)\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:554)\r\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)\r\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)\r\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\r\n\tat java.base/java.lang.Thread.run(Thread.java:830)\r\n"
      }
    ],
    "type" : "unavailable_shards_exception",
    "reason" : "at least one primary shard for the index [.security-7] is unavailable",
    "stack_trace" : "UnavailableShardsException[at least one primary shard for the index [.security-7] is unavailable]\r\n\tat org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:182)\r\n\tat org.elasticsearch.xpack.security.authz.store.NativePrivilegeStore.getPrivileges(NativePrivilegeStore.java:102)\r\n\tat org.elasticsearch.xpack.security.action.user.TransportHasPrivilegesAction.resolveApplicationPrivileges(TransportHasPrivilegesAction.java:89)\r\n\tat org.elasticsearch.xpack.security.action.user.TransportHasPrivilegesAction.doExecute(TransportHasPrivilegesAction.java:81)\r\n\tat org.elasticsearch.xpack.security.action.user.TransportHasPrivilegesAction.doExecute(TransportHasPrivilegesAction.java:37)\r\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:153)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:86)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:172)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$4(AuthorizationService.java:241)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:591)\r\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\r\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:144)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:243)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:227)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:193)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\r\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:117)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roles(CompositeRolesStore.java:152)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:231)\r\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:123)\r\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:111)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:195)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:172)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:158)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:310)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:321)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:245)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:107)\r\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:151)\r\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:129)\r\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64)\r\n\tat org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:94)\r\n\tat org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:83)\r\n\tat org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:396)\r\n\tat org.elasticsearch.xpack.enrich.action.TransportPutEnrichPolicyAction.masterOperation(TransportPutEnrichPolicyAction.java:118)\r\n\tat org.elasticsearch.xpack.enrich.action.TransportPutEnrichPolicyAction.masterOperation(TransportPutEnrichPolicyAction.java:37)\r\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:98)\r\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:169)\r\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)\r\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:169)\r\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:132)\r\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:109)\r\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:58)\r\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:153)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:86)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:172)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$4(AuthorizationService.java:241)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:591)\r\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\r\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:144)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:243)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:227)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:193)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\r\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:117)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$buildThenMaybeCacheRole$10(CompositeRolesStore.java:273)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromDescriptors(CompositeRolesStore.java:396)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildThenMaybeCacheRole(CompositeRolesStore.java:254)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$roles$4(CompositeRolesStore.java:175)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$loadRoleDescriptorsAsync$14(CompositeRolesStore.java:308)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\r\n\tat org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:120)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$loadRoleDescriptorsAsync$17(CompositeRolesStore.java:328)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authz.store.FileRolesStore.accept(FileRolesStore.java:95)\r\n\tat org.elasticsearch.xpack.security.authz.store.FileRolesStore.accept(FileRolesStore.java:57)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$loadRoleDescriptorsAsync$18(CompositeRolesStore.java:314)\r\n\tat org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:117)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$loadRoleDescriptorsAsync$17(CompositeRolesStore.java:328)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.core.security.authz.store.ReservedRolesStore.accept(ReservedRolesStore.java:293)\r\n\tat org.elasticsearch.xpack.core.security.authz.store.ReservedRolesStore.accept(ReservedRolesStore.java:34)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$loadRoleDescriptorsAsync$18(CompositeRolesStore.java:314)\r\n\tat org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:102)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.loadRoleDescriptorsAsync(CompositeRolesStore.java:330)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roleDescriptors(CompositeRolesStore.java:297)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roles(CompositeRolesStore.java:155)\r\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:231)\r\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:123)\r\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:111)\r\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:195)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:172)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:158)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:310)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:321)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:245)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155)\r\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:107)\r\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:151)\r\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:129)\r\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64)\r\n\tat org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:94)\r\n\tat org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:83)\r\n\tat org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:396)\r\n\tat org.elasticsearch.xpack.enrich.rest.RestPutEnrichPolicyAction.lambda$prepareRequest$0(RestPutEnrichPolicyAction.java:32)\r\n\tat org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:108)\r\n\tat org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:63)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$writeAuthToContext$24(AuthenticationService.java:596)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.writeAuthToContext(AuthenticationService.java:610)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.finishAuthentication(AuthenticationService.java:584)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeUser(AuthenticationService.java:531)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$16(AuthenticationService.java:415)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\r\n\tat org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:120)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$13(AuthenticationService.java:385)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$1(CachingUsernamePasswordRealm.java:145)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.handleCachedAuthentication(CachingUsernamePasswordRealm.java:196)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$2(CachingUsernamePasswordRealm.java:137)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)\r\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)\r\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:68)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:132)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:103)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$15(AuthenticationService.java:374)\r\n\tat org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:117)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$13(AuthenticationService.java:399)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$3(CachingUsernamePasswordRealm.java:175)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:91)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:166)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:103)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$15(AuthenticationService.java:374)\r\n\tat org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:102)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:419)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$11(AuthenticationService.java:341)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:351)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:292)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:327)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:272)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:254)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\r\n\tat org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:381)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:250)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:310)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:321)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:245)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196)\r\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:122)\r\n\tat org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:55)\r\n\tat org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:227)\r\n\tat org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:309)\r\n\tat org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:167)\r\n\tat org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:322)\r\n\tat org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:372)\r\n\tat org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:301)\r\n\tat org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:54)\r\n\tat org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:29)\r\n\tat io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\r\n\tat org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\r\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\r\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:326)\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:300)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\r\n\tat io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\r\n\tat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1478)\r\n\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227)\r\n\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503)\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\r\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\r\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)\r\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:600)\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:554)\r\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)\r\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)\r\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\r\n\tat java.base/java.lang.Thread.run(Thread.java:830)\r\n"
  },
  "status" : 503
}

[Mpdreamz]

Relates to: elastic/elasticsearch#36078

[russcam]

I don't recall seeing these CI test failures anymore, since WaitForSecurityIndices() was introduced in d624044. I think we can close this one for now, @Mpdreamz?

[russcam]

I believe this issue is now resolved with WaitForSecurityIndices() as I haven't seen this issue since. Closing

[LinTechSo]

Hi any Updates ?

at least one primary shard for the index [.security-7] is unavailable
Using the latest version of the elastic search bitnami chart

[stevejgordon]

@LinTechSo This is a closed issue. Not sure what update you are expecting?

Following the linked issues, this is expected behaviour from the server if the security index is not ready on a cluster. Our tests work around this by waiting until the index is green before performing security operations which require the presence of that index.

client.Cluster.Health(new ClusterHealthRequest(".security-*") { WaitForStatus = WaitForStatus.Green })

If you are running into a different issue, please can you raise a new issue with the details and steps to reproduce it?