[CI] XDocsClientYamlTestSuiteIT {yaml=en/rest-api/security/clear-cache/line_39} failed

[cbuescher]

Build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=amazon/20/console

At first I couldn't reproduce with:

./gradlew :x-pack:docs:integTestRunner \
  -Dtests.seed=70AE4CEFD4CE334D \
  -Dtests.class=org.elasticsearch.smoketest.XDocsClientYamlTestSuiteIT \
  -Dtests.method="test {yaml=en/rest-api/security/clear-cache/line_39}" \
  -Dtests.security.manager=true \
  -Dtests.locale=hi-IN \
  -Dtests.timezone=Asia/Hebron \
  -Dcompiler.java=10 \
  -Druntime.java=8

Then after a few runs I once saw the following thread dump and a failure locally, but couldn't repeat this so far:

[2018-09-24T20:18:06,313][INFO ][o.e.t.TransportService   ] [node-0] publish_address {127.0.0.1:49512}, bound_addresses {[::1]:49511}, {127.0.0.1:49512}
|    [2018-09-24T20:18:06,373][WARN ][o.e.b.BootstrapChecks    ] [node-0] HTTPS is required in order to use the token service; please enable HTTPS using the [xpack.security.http.ssl.enabled] setting or disable the token service using the [xpack.security.authc.token.enabled] setting
|
|  [jstack]
|    2018-09-24 20:18:09
|    Full thread dump Java HotSpot(TM) 64-Bit Server VM (10.0.2+13 mixed mode):
|
|    Threads class SMR info:
|    _java_thread_list=0x00007f8f32e0bfd0, length=34, elements={
|    0x00007f8f2b809000, 0x00007f8f2b02b800, 0x00007f8f2b02e800, 0x00007f8f2a840800,
|    0x00007f8f2b034000, 0x00007f8f2c0d9000, 0x00007f8f2b87f800, 0x00007f8f2d800800,
|    0x00007f8f2c0da000, 0x00007f8f2b032000, 0x00007f8f2a835000, 0x00007f8f2b2b2800,
|    0x00007f8f2f338000, 0x00007f8f2b481800, 0x00007f8f2da0c800, 0x00007f8f31059000,
|    0x00007f8f3016d800, 0x00007f8f31887000, 0x00007f8f30c0d800, 0x00007f8f31f26800,
|    0x00007f8f2b8ff000, 0x00007f8f2fb25800, 0x00007f8f2fb2a800, 0x00007f8f32077000,
|    0x00007f8f30961800, 0x00007f8f31058800, 0x00007f8f2faba000, 0x00007f8f2bc23800,
|    0x00007f8f2df08800, 0x00007f8f301a7000, 0x00007f8f2a8ac000, 0x00007f8f2f879000,
|    0x00007f8f309c2000, 0x00007f8f321e5000
|    }
|
|    "main" #1 prio=5 os_prio=31 tid=0x00007f8f2b809000 nid=0x2503 waiting on condition  [0x0000000109bf8000]
|       java.lang.Thread.State: WAITING (parking)
|       at jdk.internal.misc.Unsafe.park(java.base@10.0.2/Native Method)
|       - parking to wait for  <0x00000007a8acf5c0> (a java.util.concurrent.CountDownLatch$Sync)
|       at java.util.concurrent.locks.LockSupport.park(java.base@10.0.2/LockSupport.java:194)
|       at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@10.0.2/AbstractQueuedSynchronizer.java:883)
|       at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@10.0.2/AbstractQueuedSynchronizer.java:1037)
|       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@10.0.2/AbstractQueuedSynchronizer.java:1343)
|       at java.util.concurrent.CountDownLatch.await(java.base@10.0.2/CountDownLatch.java:232)
|       at org.elasticsearch.node.Node.start(Node.java:727)
|       at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:264)
|       at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:335)
|       at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136)
|       at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127)
|       at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
|       at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
|       at org.elasticsearch.cli.Command.main(Command.java:90)
|       at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
|       at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86)
|
|    "Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007f8f2b02b800 nid=0x3903 waiting on condition  [0x0000700003ff0000]
|       java.lang.Thread.State: RUNNABLE
|       at java.lang.ref.Reference.waitForReferencePendingList(java.base@10.0.2/Native Method)
|       at java.lang.ref.Reference.processPendingReferences(java.base@10.0.2/Reference.java:174)
|       at java.lang.ref.Reference.access$000(java.base@10.0.2/Reference.java:44)
|       at java.lang.ref.Reference$ReferenceHandler.run(java.base@10.0.2/Reference.java:138)
|
|    "Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007f8f2b02e800 nid=0x3c03 in Object.wait()  [0x00007000040f3000]
|       java.lang.Thread.State: WAITING (on object monitor)
|       at java.lang.Object.wait(java.base@10.0.2/Native Method)
|       - waiting on <0x00000007aaac3810> (a java.lang.ref.ReferenceQueue$Lock)
|       at java.lang.ref.ReferenceQueue.remove(java.base@10.0.2/ReferenceQueue.java:151)
|       - waiting to re-lock in wait() <0x00000007aaac3810> (a java.lang.ref.ReferenceQueue$Lock)
|       at java.lang.ref.ReferenceQueue.remove(java.base@10.0.2/ReferenceQueue.java:172)
|       at java.lang.ref.Finalizer$FinalizerThread.run(java.base@10.0.2/Finalizer.java:216)
|
|    "Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007f8f2a840800 nid=0xa803 runnable  [0x0000000000000000]
|       java.lang.Thread.State: RUNNABLE
|
|    "C2 CompilerThread0" #5 daemon prio=9 os_prio=31 tid=0x00007f8f2b034000 nid=0x5a03 runnable  [0x0000000000000000]
|       java.lang.Thread.State: RUNNABLE
|       Compiling: 5203 %     4       org.apache.lucene.util.automaton.MinimizationOperations::minimize @ 777 (1439 bytes)
|
|    "C2 CompilerThread1" #6 daemon prio=9 os_prio=31 tid=0x00007f8f2c0d9000 nid=0xa703 waiting on condition  [0x0000000000000000]
|       java.lang.Thread.State: RUNNABLE
|       No compile task
|
|    "C2 CompilerThread2" #7 daemon prio=9 os_prio=31 tid=0x00007f8f2b87f800 nid=0x5d03 waiting on condition  [0x0000000000000000]
|       java.lang.Thread.State: RUNNABLE
|       No compile task
|
|    "C1 CompilerThread3" #8 daemon prio=9 os_prio=31 tid=0x00007f8f2d800800 nid=0xa303 waiting on condition  [0x0000000000000000]
|       java.lang.Thread.State: RUNNABLE
|       No compile task
|
|    "Sweeper thread" #9 daemon prio=9 os_prio=31 tid=0x00007f8f2c0da000 nid=0xa003 runnable  [0x0000000000000000]
|       java.lang.Thread.State: RUNNABLE
|
|    "Service Thread" #10 daemon prio=9 os_prio=31 tid=0x00007f8f2b032000 nid=0x9f03 runnable  [0x0000000000000000]
|       java.lang.Thread.State: RUNNABLE
|
|    "Common-Cleaner" #11 daemon prio=8 os_prio=31 tid=0x00007f8f2a835000 nid=0x6103 in Object.wait()  [0x000070000490b000]
|       java.lang.Thread.State: TIMED_WAITING (on object monitor)
|       at java.lang.Object.wait(java.base@10.0.2/Native Method)
|       - waiting on <0x00000007aaaca998> (a java.lang.ref.ReferenceQueue$Lock)
|       at java.lang.ref.ReferenceQueue.remove(java.base@10.0.2/ReferenceQueue.java:151)
|       - waiting to re-lock in wait() <0x00000007aaaca998> (a java.lang.ref.ReferenceQueue$Lock)
|       at jdk.internal.ref.CleanerImpl.run(java.base@10.0.2/CleanerImpl.java:148)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|       at jdk.internal.misc.InnocuousThread.run(java.base@10.0.2/InnocuousThread.java:134)
|
|    "process reaper" #16 daemon prio=10 os_prio=31 tid=0x00007f8f2b2b2800 nid=0x6303 runnable  [0x0000700004a35000]
|       java.lang.Thread.State: RUNNABLE
|       at java.lang.ProcessHandleImpl.waitForProcessExit0(java.base@10.0.2/Native Method)
|       at java.lang.ProcessHandleImpl.access$000(java.base@10.0.2/ProcessHandleImpl.java:50)
|       at java.lang.ProcessHandleImpl$1.run(java.base@10.0.2/ProcessHandleImpl.java:138)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@10.0.2/ThreadPoolExecutor.java:1135)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@10.0.2/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "elasticsearch[node-0][[timer]]" #19 daemon prio=5 os_prio=31 tid=0x00007f8f2f338000 nid=0x1007 waiting on condition  [0x0000700003243000]
|       java.lang.Thread.State: TIMED_WAITING (sleeping)
|       at java.lang.Thread.sleep(java.base@10.0.2/Native Method)
|       at org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:549)
|
|    "elasticsearch[node-0][scheduler][T#1]" #20 daemon prio=5 os_prio=31 tid=0x00007f8f2b481800 nid=0x6503 waiting on condition  [0x0000700004b38000]
|       java.lang.Thread.State: TIMED_WAITING (parking)
|       at jdk.internal.misc.Unsafe.park(java.base@10.0.2/Native Method)
|       - parking to wait for  <0x00000007abc067b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
|       at java.util.concurrent.locks.LockSupport.parkNanos(java.base@10.0.2/LockSupport.java:234)
|       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@10.0.2/AbstractQueuedSynchronizer.java:2117)
|       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@10.0.2/ScheduledThreadPoolExecutor.java:1182)
|       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@10.0.2/ScheduledThreadPoolExecutor.java:899)
|       at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@10.0.2/ThreadPoolExecutor.java:1061)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@10.0.2/ThreadPoolExecutor.java:1121)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@10.0.2/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "Connection evictor" #21 daemon prio=5 os_prio=31 tid=0x00007f8f2da0c800 nid=0x9803 waiting on condition  [0x0000700004c3b000]
|       java.lang.Thread.State: TIMED_WAITING (sleeping)
|       at java.lang.Thread.sleep(java.base@10.0.2/Native Method)
|       at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "ticker-schedule-trigger-engine" #22 daemon prio=5 os_prio=31 tid=0x00007f8f31059000 nid=0x9603 waiting on condition  [0x0000700004d3e000]
|       java.lang.Thread.State: TIMED_WAITING (sleeping)
|       at java.lang.Thread.sleep(java.base@10.0.2/Native Method)
|       at org.elasticsearch.xpack.watcher.trigger.schedule.engine.TickerScheduleTriggerEngine$Ticker.run(TickerScheduleTriggerEngine.java:171)
|
|    "ml-cpp-log-tail-thread" #23 daemon prio=5 os_prio=31 tid=0x00007f8f3016d800 nid=0x6a03 runnable  [0x0000700004e41000]
|       java.lang.Thread.State: RUNNABLE
|       at java.io.FileInputStream.readBytes(java.base@10.0.2/Native Method)
|       at java.io.FileInputStream.read(java.base@10.0.2/FileInputStream.java:258)
|       at org.elasticsearch.xpack.ml.job.process.logging.CppLogMessageHandler.tailStream(CppLogMessageHandler.java:105)
|       at org.elasticsearch.xpack.ml.job.process.NativeController.lambda$tailLogsInThread$0(NativeController.java:69)
|       at org.elasticsearch.xpack.ml.job.process.NativeController$$Lambda$1831/1890190272.run(Unknown Source)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "elasticsearch[node-0][generic][T#1]" #24 daemon prio=5 os_prio=31 tid=0x00007f8f31887000 nid=0x6d03 waiting on condition  [0x0000700004f44000]
|       java.lang.Thread.State: WAITING (parking)
|       at jdk.internal.misc.Unsafe.park(java.base@10.0.2/Native Method)
|       - parking to wait for  <0x00000007abd01438> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
|       at java.util.concurrent.locks.LockSupport.park(java.base@10.0.2/LockSupport.java:194)
|       at java.util.concurrent.LinkedTransferQueue.awaitMatch(java.base@10.0.2/LinkedTransferQueue.java:743)
|       at java.util.concurrent.LinkedTransferQueue.xfer(java.base@10.0.2/LinkedTransferQueue.java:684)
|       at java.util.concurrent.LinkedTransferQueue.take(java.base@10.0.2/LinkedTransferQueue.java:1367)
|       at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@10.0.2/ThreadPoolExecutor.java:1061)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@10.0.2/ThreadPoolExecutor.java:1121)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@10.0.2/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "elasticsearch[node-0][generic][T#2]" #25 daemon prio=5 os_prio=31 tid=0x00007f8f30c0d800 nid=0x6f03 waiting on condition  [0x0000700005047000]
|       java.lang.Thread.State: WAITING (parking)
|       at jdk.internal.misc.Unsafe.park(java.base@10.0.2/Native Method)
|       - parking to wait for  <0x00000007a8a3bbc8> (a java.util.concurrent.CompletableFuture$Signaller)
|       at java.util.concurrent.locks.LockSupport.park(java.base@10.0.2/LockSupport.java:194)
|       at java.util.concurrent.CompletableFuture$Signaller.block(java.base@10.0.2/CompletableFuture.java:1796)
|       at java.util.concurrent.ForkJoinPool.managedBlock(java.base@10.0.2/ForkJoinPool.java:3156)
|       at java.util.concurrent.CompletableFuture.waitingGet(java.base@10.0.2/CompletableFuture.java:1823)
|       at java.util.concurrent.CompletableFuture.get(java.base@10.0.2/CompletableFuture.java:1998)
|       at org.elasticsearch.discovery.zen.ZenDiscovery.pingAndWait(ZenDiscovery.java:1044)
|       at org.elasticsearch.discovery.zen.ZenDiscovery.findMaster(ZenDiscovery.java:887)
|       at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:441)
|       at org.elasticsearch.discovery.zen.ZenDiscovery.access$2500(ZenDiscovery.java:89)
|       at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1246)
|       at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@10.0.2/ThreadPoolExecutor.java:1135)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@10.0.2/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "elasticsearch[node-0][generic][T#3]" #26 daemon prio=5 os_prio=31 tid=0x00007f8f31f26800 nid=0x7003 waiting on condition  [0x000070000514a000]
|       java.lang.Thread.State: WAITING (parking)
|       at jdk.internal.misc.Unsafe.park(java.base@10.0.2/Native Method)
|       - parking to wait for  <0x00000007abd01438> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
|       at java.util.concurrent.locks.LockSupport.park(java.base@10.0.2/LockSupport.java:194)
|       at java.util.concurrent.LinkedTransferQueue.awaitMatch(java.base@10.0.2/LinkedTransferQueue.java:743)
|       at java.util.concurrent.LinkedTransferQueue.xfer(java.base@10.0.2/LinkedTransferQueue.java:684)
|       at java.util.concurrent.LinkedTransferQueue.take(java.base@10.0.2/LinkedTransferQueue.java:1367)
|       at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@10.0.2/ThreadPoolExecutor.java:1061)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@10.0.2/ThreadPoolExecutor.java:1121)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@10.0.2/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "elasticsearch[node-0][generic][T#4]" #27 daemon prio=5 os_prio=31 tid=0x00007f8f2b8ff000 nid=0x9003 waiting on condition  [0x000070000524d000]
|       java.lang.Thread.State: WAITING (parking)
|       at jdk.internal.misc.Unsafe.park(java.base@10.0.2/Native Method)
|       - parking to wait for  <0x00000007abd01438> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
|       at java.util.concurrent.locks.LockSupport.park(java.base@10.0.2/LockSupport.java:194)
|       at java.util.concurrent.LinkedTransferQueue.awaitMatch(java.base@10.0.2/LinkedTransferQueue.java:743)
|       at java.util.concurrent.LinkedTransferQueue.xfer(java.base@10.0.2/LinkedTransferQueue.java:684)
|       at java.util.concurrent.LinkedTransferQueue.take(java.base@10.0.2/LinkedTransferQueue.java:1367)
|       at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@10.0.2/ThreadPoolExecutor.java:1061)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@10.0.2/ThreadPoolExecutor.java:1121)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@10.0.2/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "elasticsearch[node-0][[transport_server_worker.default]][T#1]" #28 daemon prio=5 os_prio=31 tid=0x00007f8f2fb25800 nid=0x8e03 runnable  [0x0000700005350000]
|       java.lang.Thread.State: RUNNABLE
|       at sun.nio.ch.KQueueArrayWrapper.kevent0(java.base@10.0.2/Native Method)
|       at sun.nio.ch.KQueueArrayWrapper.poll(java.base@10.0.2/KQueueArrayWrapper.java:196)
|       at sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@10.0.2/KQueueSelectorImpl.java:116)
|       at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@10.0.2/SelectorImpl.java:89)
|       - locked <0x00000007a8c8b0f0> (a sun.nio.ch.Util$2)
|       - locked <0x00000007a8c8b100> (a java.util.Collections$UnmodifiableSet)
|       - locked <0x00000007a8c8b0a0> (a sun.nio.ch.KQueueSelectorImpl)
|       at sun.nio.ch.SelectorImpl.select(java.base@10.0.2/SelectorImpl.java:100)
|       at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:755)
|       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
|       at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "elasticsearch[node-0][[transport_server_worker.default]][T#2]" #29 daemon prio=5 os_prio=31 tid=0x00007f8f2fb2a800 nid=0x7403 runnable  [0x0000700005453000]
|       java.lang.Thread.State: RUNNABLE
|       at sun.nio.ch.KQueueArrayWrapper.kevent0(java.base@10.0.2/Native Method)
|       at sun.nio.ch.KQueueArrayWrapper.poll(java.base@10.0.2/KQueueArrayWrapper.java:196)
|       at sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@10.0.2/KQueueSelectorImpl.java:116)
|       at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@10.0.2/SelectorImpl.java:89)
|       - locked <0x00000007a8914a60> (a sun.nio.ch.Util$2)
|       - locked <0x00000007a8914a70> (a java.util.Collections$UnmodifiableSet)
|       - locked <0x00000007a8914a10> (a sun.nio.ch.KQueueSelectorImpl)
|       at sun.nio.ch.SelectorImpl.select(java.base@10.0.2/SelectorImpl.java:100)
|       at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:755)
|       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
|       at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "elasticsearch[node-0][[unicast_connect]][T#1]" #30 daemon prio=5 os_prio=31 tid=0x00007f8f32077000 nid=0x8c03 waiting on condition  [0x0000700005556000]
|       java.lang.Thread.State: TIMED_WAITING (parking)
|       at jdk.internal.misc.Unsafe.park(java.base@10.0.2/Native Method)
|       - parking to wait for  <0x00000007a9397eb8> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
|       at java.util.concurrent.locks.LockSupport.parkNanos(java.base@10.0.2/LockSupport.java:234)
|       at java.util.concurrent.LinkedTransferQueue.awaitMatch(java.base@10.0.2/LinkedTransferQueue.java:740)
|       at java.util.concurrent.LinkedTransferQueue.xfer(java.base@10.0.2/LinkedTransferQueue.java:684)
|       at java.util.concurrent.LinkedTransferQueue.poll(java.base@10.0.2/LinkedTransferQueue.java:1375)
|       at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@10.0.2/ThreadPoolExecutor.java:1060)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@10.0.2/ThreadPoolExecutor.java:1121)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@10.0.2/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "elasticsearch[node-0][[unicast_connect]][T#2]" #31 daemon prio=5 os_prio=31 tid=0x00007f8f30961800 nid=0x7903 waiting on condition  [0x0000700005659000]
|       java.lang.Thread.State: TIMED_WAITING (parking)
|       at jdk.internal.misc.Unsafe.park(java.base@10.0.2/Native Method)
|       - parking to wait for  <0x00000007a9397eb8> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
|       at java.util.concurrent.locks.LockSupport.parkNanos(java.base@10.0.2/LockSupport.java:234)
|       at java.util.concurrent.LinkedTransferQueue.awaitMatch(java.base@10.0.2/LinkedTransferQueue.java:740)
|       at java.util.concurrent.LinkedTransferQueue.xfer(java.base@10.0.2/LinkedTransferQueue.java:684)
|       at java.util.concurrent.LinkedTransferQueue.poll(java.base@10.0.2/LinkedTransferQueue.java:1375)
|       at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@10.0.2/ThreadPoolExecutor.java:1060)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@10.0.2/ThreadPoolExecutor.java:1121)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@10.0.2/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "elasticsearch[node-0][clusterApplierService#updateTask][T#1]" #32 daemon prio=5 os_prio=31 tid=0x00007f8f31058800 nid=0x7b03 waiting on condition  [0x000070000575c000]
|       java.lang.Thread.State: WAITING (parking)
|       at jdk.internal.misc.Unsafe.park(java.base@10.0.2/Native Method)
|       - parking to wait for  <0x00000007a8bf7400> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
|       at java.util.concurrent.locks.LockSupport.park(java.base@10.0.2/LockSupport.java:194)
|       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@10.0.2/AbstractQueuedSynchronizer.java:2075)
|       at java.util.concurrent.PriorityBlockingQueue.take(java.base@10.0.2/PriorityBlockingQueue.java:550)
|       at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@10.0.2/ThreadPoolExecutor.java:1061)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@10.0.2/ThreadPoolExecutor.java:1121)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@10.0.2/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "elasticsearch[node-0][[unicast_connect]][T#3]" #33 daemon prio=5 os_prio=31 tid=0x00007f8f2faba000 nid=0x7d03 waiting on condition  [0x000070000585f000]
|       java.lang.Thread.State: TIMED_WAITING (parking)
|       at jdk.internal.misc.Unsafe.park(java.base@10.0.2/Native Method)
|       - parking to wait for  <0x00000007a9397eb8> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
|       at java.util.concurrent.locks.LockSupport.parkNanos(java.base@10.0.2/LockSupport.java:234)
|       at java.util.concurrent.LinkedTransferQueue.awaitMatch(java.base@10.0.2/LinkedTransferQueue.java:740)
|       at java.util.concurrent.LinkedTransferQueue.xfer(java.base@10.0.2/LinkedTransferQueue.java:684)
|       at java.util.concurrent.LinkedTransferQueue.poll(java.base@10.0.2/LinkedTransferQueue.java:1375)
|       at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@10.0.2/ThreadPoolExecutor.java:1060)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@10.0.2/ThreadPoolExecutor.java:1121)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@10.0.2/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "elasticsearch[node-0][transport_client_boss][T#2]" #35 daemon prio=5 os_prio=31 tid=0x00007f8f2bc23800 nid=0x8903 runnable  [0x0000700005962000]
|       java.lang.Thread.State: RUNNABLE
|       at sun.nio.ch.KQueueArrayWrapper.kevent0(java.base@10.0.2/Native Method)
|       at sun.nio.ch.KQueueArrayWrapper.poll(java.base@10.0.2/KQueueArrayWrapper.java:196)
|       at sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@10.0.2/KQueueSelectorImpl.java:116)
|       at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@10.0.2/SelectorImpl.java:89)
|       - locked <0x00000007a9414280> (a sun.nio.ch.Util$2)
|       - locked <0x00000007a9414270> (a java.util.Collections$UnmodifiableSet)
|       - locked <0x00000007a94141d0> (a sun.nio.ch.KQueueSelectorImpl)
|       at sun.nio.ch.SelectorImpl.select(java.base@10.0.2/SelectorImpl.java:100)
|       at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:755)
|       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
|       at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "elasticsearch[node-0][transport_client_boss][T#1]" #34 daemon prio=5 os_prio=31 tid=0x00007f8f2df08800 nid=0x8103 runnable  [0x0000700005a65000]
|       java.lang.Thread.State: RUNNABLE
|       at sun.nio.ch.KQueueArrayWrapper.kevent0(java.base@10.0.2/Native Method)
|       at sun.nio.ch.KQueueArrayWrapper.poll(java.base@10.0.2/KQueueArrayWrapper.java:196)
|       at sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@10.0.2/KQueueSelectorImpl.java:116)
|       at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@10.0.2/SelectorImpl.java:89)
|       - locked <0x00000007a919f1a0> (a sun.nio.ch.Util$2)
|       - locked <0x00000007a919f190> (a java.util.Collections$UnmodifiableSet)
|       - locked <0x00000007a919f0f0> (a sun.nio.ch.KQueueSelectorImpl)
|       at sun.nio.ch.SelectorImpl.select(java.base@10.0.2/SelectorImpl.java:100)
|       at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:755)
|       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
|       at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "elasticsearch[node-0][transport_client_boss][T#4]" #36 daemon prio=5 os_prio=31 tid=0x00007f8f301a7000 nid=0x8603 runnable  [0x0000700005b68000]
|       java.lang.Thread.State: RUNNABLE
|       at sun.nio.ch.KQueueArrayWrapper.kevent0(java.base@10.0.2/Native Method)
|       at sun.nio.ch.KQueueArrayWrapper.poll(java.base@10.0.2/KQueueArrayWrapper.java:196)
|       at sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@10.0.2/KQueueSelectorImpl.java:116)
|       at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@10.0.2/SelectorImpl.java:89)
|       - locked <0x00000007a91a4130> (a sun.nio.ch.Util$2)
|       - locked <0x00000007a91a4120> (a java.util.Collections$UnmodifiableSet)
|       - locked <0x00000007a91a4080> (a sun.nio.ch.KQueueSelectorImpl)
|       at sun.nio.ch.SelectorImpl.select(java.base@10.0.2/SelectorImpl.java:100)
|       at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:755)
|       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
|       at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "elasticsearch[node-0][transport_client_boss][T#3]" #37 daemon prio=5 os_prio=31 tid=0x00007f8f2a8ac000 nid=0x8403 runnable  [0x0000700005c6b000]
|       java.lang.Thread.State: RUNNABLE
|       at sun.nio.ch.KQueueArrayWrapper.kevent0(java.base@10.0.2/Native Method)
|       at sun.nio.ch.KQueueArrayWrapper.poll(java.base@10.0.2/KQueueArrayWrapper.java:196)
|       at sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@10.0.2/KQueueSelectorImpl.java:116)
|       at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@10.0.2/SelectorImpl.java:89)
|       - locked <0x00000007a91a19b0> (a sun.nio.ch.Util$2)
|       - locked <0x00000007a91a19a0> (a java.util.Collections$UnmodifiableSet)
|       - locked <0x00000007a91a1900> (a sun.nio.ch.KQueueSelectorImpl)
|       at sun.nio.ch.SelectorImpl.select(java.base@10.0.2/SelectorImpl.java:100)
|       at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:755)
|       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
|       at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "elasticsearch[node-0][transport_client_boss][T#5]" #38 daemon prio=5 os_prio=31 tid=0x00007f8f2f879000 nid=0xac03 runnable  [0x0000700005d6e000]
|       java.lang.Thread.State: RUNNABLE
|       at sun.nio.ch.KQueueArrayWrapper.kevent0(java.base@10.0.2/Native Method)
|       at sun.nio.ch.KQueueArrayWrapper.poll(java.base@10.0.2/KQueueArrayWrapper.java:196)
|       at sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@10.0.2/KQueueSelectorImpl.java:116)
|       at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@10.0.2/SelectorImpl.java:89)
|       - locked <0x00000007a91a68b0> (a sun.nio.ch.Util$2)
|       - locked <0x00000007a91a68a0> (a java.util.Collections$UnmodifiableSet)
|       - locked <0x00000007a91a6800> (a sun.nio.ch.KQueueSelectorImpl)
|       at sun.nio.ch.SelectorImpl.select(java.base@10.0.2/SelectorImpl.java:100)
|       at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:755)
|       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
|       at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "elasticsearch[node-0][transport_client_boss][T#6]" #39 daemon prio=5 os_prio=31 tid=0x00007f8f309c2000 nid=0x15303 runnable  [0x0000700005e71000]
|       java.lang.Thread.State: RUNNABLE
|       at sun.nio.ch.KQueueArrayWrapper.kevent0(java.base@10.0.2/Native Method)
|       at sun.nio.ch.KQueueArrayWrapper.poll(java.base@10.0.2/KQueueArrayWrapper.java:196)
|       at sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@10.0.2/KQueueSelectorImpl.java:116)
|       at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@10.0.2/SelectorImpl.java:89)
|       - locked <0x00000007a91a9030> (a sun.nio.ch.Util$2)
|       - locked <0x00000007a91a9020> (a java.util.Collections$UnmodifiableSet)
|       - locked <0x00000007a91a8f80> (a sun.nio.ch.KQueueSelectorImpl)
|       at sun.nio.ch.SelectorImpl.select(java.base@10.0.2/SelectorImpl.java:100)
|       at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:755)
|       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
|       at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
|       at java.lang.Thread.run(java.base@10.0.2/Thread.java:844)
|
|    "Attach Listener" #40 daemon prio=9 os_prio=31 tid=0x00007f8f321e5000 nid=0x15103 waiting on condition  [0x0000000000000000]
|       java.lang.Thread.State: RUNNABLE
|
|    "VM Thread" os_prio=31 tid=0x00007f8f2b039000 nid=0x3703 runnable
|
|    "GC Thread#0" os_prio=31 tid=0x00007f8f2b000000 nid=0x5103 runnable
|
|    "GC Thread#1" os_prio=31 tid=0x00007f8f2c00c000 nid=0x2d03 runnable
|
|    "GC Thread#2" os_prio=31 tid=0x00007f8f2c00d000 nid=0x4f03 runnable
|
|    "GC Thread#3" os_prio=31 tid=0x00007f8f2c00d800 nid=0x4d03 runnable
|
|    "GC Thread#4" os_prio=31 tid=0x00007f8f2c00e800 nid=0x2f03 runnable
|
|    "GC Thread#5" os_prio=31 tid=0x00007f8f2c00f000 nid=0x3103 runnable
|
|    "GC Thread#6" os_prio=31 tid=0x00007f8f2c010000 nid=0x4c03 runnable
|
|    "GC Thread#7" os_prio=31 tid=0x00007f8f2b812000 nid=0x3403 runnable
|
|    "CMS Main Thread" os_prio=31 tid=0x00007f8f2b81d800 nid=0x4803 runnable
|
|    "CMS Thread#0" os_prio=31 tid=0x00007f8f2c010800 nid=0x4b03 runnable
|
|    "CMS Thread#1" os_prio=31 tid=0x00007f8f2a802800 nid=0x3603 runnable
|
|    "VM Periodic Task Thread" os_prio=31 tid=0x00007f8f2a801000 nid=0x9b03 waiting on condition
|
|    JNI global references: 74
|
|-----------------------------------------

> Task :x-pack:docs:integTestCluster#wait FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':x-pack:docs:integTestCluster#wait'.
> Failed to start elasticsearch: timed out after 30 seconds

I'm currently not able to log into the machine (ssh timeouts) to look if there are similar things in the cluster logs, the failure I see in the test log is:

18:01:10   1> [2018-09-24T19:00:39,963][ERROR][o.e.s.XDocsClientYamlTestSuiteIT] [test] This failing test was generated by documentation starting at en/rest-api/security/clear-cache.asciidoc:line_39. It may include many snippets. See Elasticsearch's docs/README.asciidoc for an explanation of test generation.
18:01:10 ERROR   0.45s | XDocsClientYamlTestSuiteIT.test {yaml=en/rest-api/security/clear-cache/line_39} <<< FAILURES!
18:01:10    > Throwable #1: org.elasticsearch.test.rest.yaml.ClientYamlTestResponseException: org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:46301], URI [/_xpack/security/user?error_trace=true], status line [HTTP/1.1 503 Service Unavailable]
18:01:10    > {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[],"stack_trace":"Failed to execute phase [query], all shards failed\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254)\n\tat org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:106)\n\tat org.elasticsearch.action.search.InitialSearchPhase.access$200(InitialSearchPhase.java:50)\n\tat org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:277)\n\tat org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73)\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)\n\tat org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:442)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1121)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1218)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1202)\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66)\n\tat org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:103)\n\tat org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:346)\n\tat org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:340)\n\tat org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:334)\n\tat org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1073)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:844)\n"},"status":503}

[elasticmachine]

Pinging @elastic/es-security

[droberts195]

Another test in the same docs file failed in the same way in 6.4 and I suspect the root cause is the same.

The repro command is this, although it didn't reproduce locally for me:

./gradlew :x-pack:docs:integTestRunner \
  -Dtests.seed=6668AB67F4819461 \
  -Dtests.class=org.elasticsearch.smoketest.XDocsClientYamlTestSuiteIT \
  -Dtests.method="test {yaml=en/rest-api/security/clear-cache/line_47}" \
  -Dtests.security.manager=true \
  -Dtests.locale=th \
  -Dtests.timezone=Pacific/Galapagos

The error in the test log is this:

   > Throwable #1: org.elasticsearch.test.rest.yaml.ClientYamlTestResponseException: org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:40403], URI [/_xpack/security/user?error_trace=true], status line [HTTP/1.1 503 Service Unavailable]
   > {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[],"stack_trace":"Failed to execute phase [query], all shards failed\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254)\n\tat org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101)\n\tat org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48)\n\tat org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:222)\n\tat org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:176)\n\tat org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48)\n\tat org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:222)\n\tat org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73)\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)\n\tat org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:526)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1068)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1165)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1149)\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66)\n\tat org.elasticsearch.action.search.SearchTransportService$6$1.onFailure(SearchTransportService.java:384)\n\tat org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:341)\n\tat org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335)\n\tat org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:329)\n\tat org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1019)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:844)\n"},"status":503}

These are the log messages in the server log around the time the error occurred:

[2018-09-26T06:05:37,760][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node-0] [.security-6/-VvwEk-qQyCVyi-EtdpUTg] deleting index
[2018-09-26T06:05:37,917][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-0] [.security-6] creating index, cause [api], templates [security-index-template], shards [1]/[0], mappings [doc]
[2018-09-26T06:05:37,946][DEBUG][o.e.a.s.TransportSearchAction] [node-0] All shards failed for phase: [query]
[2018-09-26T06:05:37,974][INFO ][o.e.c.r.a.AllocationService] [node-0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.security-6][0]] ...]).
[2018-09-26T06:05:38,094][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node-0] [.security-6/WAbCr-IhRKqL_sRxyHbn6A] deleting index
[2018-09-26T06:05:38,385][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-0] [.security-6] creating index, cause [api], templates [security-index-template], shards [1]/[0], mappings [doc]
[2018-09-26T06:05:38,486][INFO ][o.e.c.r.a.AllocationService] [node-0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.security-6][0]] ...]).

The timestamp on the error itself is 2018-09-26T06:05:38,234, so during the period between when the .security-6 was deleted and when it was recreated. So I suspect the problem here is that the docs tests are repeatedly deleting and recreating the .security-6 index without making sure that the test code runs while the index exists.

[nik9000]

We believe this is caused because the tests clean out the security index after each test but don't wait for it to be recreated before the test. I've started working on cleaning up some of the other x-pack test cleanup code and eventually I'll get to security. And when I do, this issue is waiting for me.

[andrershov]

Reproduced again https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+periodic/171/console

REPRODUCE WITH: ./gradlew :x-pack:docs:integTestRunner \
  -Dtests.seed=CA88C7EBB4D1EE93 \
  -Dtests.class=org.elasticsearch.smoketest.XDocsClientYamlTestSuiteIT \
  -Dtests.method="test {yaml=en/rest-api/security/clear-cache/line_39}" \
  -Dtests.security.manager=true \
  -Dtests.locale=ar-QA \
  -Dtests.timezone=Greenwich \
  -Dcompiler.java=11 \
  -Druntime.java=8

[andyb-elastic]

I ran the tests for this doc page in a loop a couple hundred times and couldn't reproduce. Here's the failures for security docs pages with stack traces that look like the ones here. It seems like there were a handful of them in fall 2018 and none since then

[jkakavas]

As these haven't failed since late October last year, it looks related to #33864. Could it be that the failures here were also caused/affected by the watcher issue that was resolved in #35271 ?

I'm going to close this for now, we can reopen if any such failure re-emerges.