Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SnapshotIT testCreateSnapshot fails possibly due to simultaneous deletions #37581

Closed
jaymode opened this issue Jan 17, 2019 · 6 comments · Fixed by #37612 or #49142
Closed

SnapshotIT testCreateSnapshot fails possibly due to simultaneous deletions #37581

jaymode opened this issue Jan 17, 2019 · 6 comments · Fixed by #37612 or #49142
Assignees
@original-brownbear
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI

Comments

@jaymode
Copy link
Member

jaymode commented Jan 17, 2019

I hit this failure locally multiple times but it does not reproduce.

Reproduce line:

./gradlew :client:rest-high-level:integTestRunner -Dtests.seed=DEF00E9C2189727D -Dtests.class=org.elasticsearch.client.SnapshotIT -Dtests.method="testCreateSnapshot" -Dtests.security.manager=true -Dtests.locale=sr-Cyrl-XK -Dtests.timezone=Asia/Yekaterinburg -Dcompiler.java=11 -Druntime.java=11

ERROR   0.12s | SnapshotIT.testCreateSnapshot <<< FAILURES!
   > Throwable #1: ElasticsearchStatusException[Elasticsearch exception [type=repository_exception, reason=[test_repository] concurrent modification of the index-N file, expected current generation [13], actual current generation [14] - possibly due to simultaneous snapshot deletion requests]]
   > 	at __randomizedtesting.SeedInfo.seed([DEF00E9C2189727D:20969B2F173A8064]:0)
   > 	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
   > 	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1703)
   > 	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1679)
   > 	at org.elasticsearch.client.RestHighLevelClient$1.onFailure(RestHighLevelClient.java:1598)
   > 	at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:614)
   > 	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:317)
   > 	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:294)
   > 	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
   > 	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
   > 	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
   > 	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)
   > 	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
   > 	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
   > 	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
   > 	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
   > 	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
   > 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
   > 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
   > 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
   > 	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
   > 	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
   > 	at java.base/java.lang.Thread.run(Thread.java:834)
   > 	Suppressed: org.elasticsearch.client.ResponseException: method [DELETE], host [http://127.0.0.1:35141], URI [/_snapshot/test_repository/test_snapshot?master_timeout=30s], status line [HTTP/1.1 500 Internal Server Error]
   > {"error":{"root_cause":[{"type":"repository_exception","reason":"[test_repository] concurrent modification of the index-N file, expected current generation [13], actual current generation [14] - possibly due to simultaneous snapshot deletion requests"}],"type":"repository_exception","reason":"[test_repository] concurrent modification of the index-N file, expected current generation [13], actual current generation [14] - possibly due to simultaneous snapshot deletion requests"},"status":500}
   > 		at org.elasticsearch.client.RestClient$1.completed(RestClient.java:309)
   > 		... 16 more

Test runner output with test ordering:

Suite: org.elasticsearch.client.SnapshotIT
  1> [2019-01-17T20:42:16,972][INFO ][o.e.c.SnapshotIT         ] [testSnapshotGetRepositoriesUsingParams] before test
  1> [2019-01-17T20:42:16,974][INFO ][o.e.c.SnapshotIT         ] [testSnapshotGetRepositoriesUsingParams] initializing REST clients against [http://127.0.0.1:35141]
  1> [2019-01-17T20:42:17,074][INFO ][o.e.c.SnapshotIT         ] [testSnapshotGetRepositoriesUsingParams] after test
  1> [2019-01-17T20:42:17,087][INFO ][o.e.c.SnapshotIT         ] [testSnapshotDeleteRepository] before test
  1> [2019-01-17T20:42:17,148][INFO ][o.e.c.SnapshotIT         ] [testSnapshotDeleteRepository] after test
  1> [2019-01-17T20:42:17,160][INFO ][o.e.c.SnapshotIT         ] [testSnapshotsStatus] before test
  1> [2019-01-17T20:42:17,316][INFO ][o.e.c.SnapshotIT         ] [testSnapshotsStatus] after test
  1> [2019-01-17T20:42:17,328][INFO ][o.e.c.SnapshotIT         ] [testRestoreSnapshot] before test
  1> [2019-01-17T20:42:17,553][INFO ][o.e.c.SnapshotIT         ] [testRestoreSnapshot] after test
  1> [2019-01-17T20:42:17,568][INFO ][o.e.c.SnapshotIT         ] [testSnapshotGetRepositoriesNonExistent] before test
  1> [2019-01-17T20:42:17,600][INFO ][o.e.c.SnapshotIT         ] [testSnapshotGetRepositoriesNonExistent] after test
  1> [2019-01-17T20:42:17,615][INFO ][o.e.c.SnapshotIT         ] [testCreateSnapshot] before test
  1> [2019-01-17T20:42:17,710][INFO ][o.e.c.SnapshotIT         ] [testCreateSnapshot] after test
  2> REPRODUCE WITH: ./gradlew :client:rest-high-level:integTestRunner -Dtests.seed=DEF00E9C2189727D -Dtests.class=org.elasticsearch.client.SnapshotIT -Dtests.method="testCreateSnapshot" -Dtests.security.manager=true -Dtests.locale=sr-Cyrl-XK -Dtests.timezone=Asia/Yekaterinburg -Dcompiler.java=11 -Druntime.java=11
ERROR   0.12s | SnapshotIT.testCreateSnapshot <<< FAILURES!
   > Throwable #1: ElasticsearchStatusException[Elasticsearch exception [type=repository_exception, reason=[test_repository] concurrent modification of the index-N file, expected current generation [13], actual current generation [14] - possibly due to simultaneous snapshot deletion requests]]
   > 	at __randomizedtesting.SeedInfo.seed([DEF00E9C2189727D:20969B2F173A8064]:0)
   > 	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
   > 	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1703)
   > 	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1679)
   > 	at org.elasticsearch.client.RestHighLevelClient$1.onFailure(RestHighLevelClient.java:1598)
   > 	at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:614)
   > 	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:317)
   > 	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:294)
   > 	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
   > 	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
   > 	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
   > 	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)
   > 	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
   > 	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
   > 	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
   > 	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
   > 	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
   > 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
   > 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
   > 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
   > 	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
   > 	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
   > 	at java.base/java.lang.Thread.run(Thread.java:834)
   > 	Suppressed: org.elasticsearch.client.ResponseException: method [DELETE], host [http://127.0.0.1:35141], URI [/_snapshot/test_repository/test_snapshot?master_timeout=30s], status line [HTTP/1.1 500 Internal Server Error]
   > {"error":{"root_cause":[{"type":"repository_exception","reason":"[test_repository] concurrent modification of the index-N file, expected current generation [13], actual current generation [14] - possibly due to simultaneous snapshot deletion requests"}],"type":"repository_exception","reason":"[test_repository] concurrent modification of the index-N file, expected current generation [13], actual current generation [14] - possibly due to simultaneous snapshot deletion requests"},"status":500}
   > 		at org.elasticsearch.client.RestClient$1.completed(RestClient.java:309)
   > 		... 16 more
  1> [2019-01-17T20:42:17,729][INFO ][o.e.c.SnapshotIT         ] [testSnapshotGetDefaultRepositories] before test
  1> [2019-01-17T20:42:17,813][INFO ][o.e.c.SnapshotIT         ] [testSnapshotGetDefaultRepositories] after test
  1> [2019-01-17T20:42:17,827][INFO ][o.e.c.SnapshotIT         ] [testGetSnapshots] before test
  1> [2019-01-17T20:42:17,938][INFO ][o.e.c.SnapshotIT         ] [testGetSnapshots] after test
  1> [2019-01-17T20:42:17,951][INFO ][o.e.c.SnapshotIT         ] [testDeleteSnapshot] before test
  1> [2019-01-17T20:42:18,029][INFO ][o.e.c.SnapshotIT         ] [testDeleteSnapshot] after test
  1> [2019-01-17T20:42:18,041][INFO ][o.e.c.SnapshotIT         ] [testVerifyRepository] before test
  1> [2019-01-17T20:42:18,108][INFO ][o.e.c.SnapshotIT         ] [testVerifyRepository] after test
  1> [2019-01-17T20:42:18,121][INFO ][o.e.c.SnapshotIT         ] [testCreateRepository] before test
  1> [2019-01-17T20:42:18,175][INFO ][o.e.c.SnapshotIT         ] [testCreateRepository] after test
  2> NOTE: leaving temporary files on disk at: /var/jenkins_home/jobs/elasticsearch-pipeline/workspace/elasticsearch/client/rest-high-level/build/testrun/integTestRunner/J0/temp/org.elasticsearch.client.SnapshotIT_DEF00E9C2189727D-001
  2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=1237, maxMBSortInHeap=7.33712303577021, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@123f275b), locale=sr-Cyrl-XK, timezone=Asia/Yekaterinburg
  2> NOTE: Linux 4.18.0-13-generic amd64/Oracle Corporation 11 (64-bit)/cpus=32,threads=1,free=430176280,total=536870912
  2> NOTE: All tests run in this JVM: [CCRDocumentationIT, WatcherDocumentationIT, BulkRequestWithGlobalParametersIT, StoredScriptsDocumentationIT, SecurityIT, RankEvalIT, MigrationIT, BulkProcessorRetryIT, MigrationDocumentationIT, ClusterClientIT, ClusterClientDocumentationIT, BulkProcessorIT, ReindexIT, IngestClientIT, MachineLearningIT, IndicesClientIT, GraphIT, CrudIT, MigrationClientDocumentationIT, LicensingDocumentationIT, SnapshotClientDocumentationIT, CRUDDocumentationIT, WatcherIT, IndexLifecycleIT, SearchDocumentationIT, PingAndInfoIT, CCRIT, RollupDocumentationIT, SearchIT, MachineLearningGetResultsIT, RollupIT, MlClientDocumentationIT, ILMDocumentationIT, LicenseIT, SnapshotIT]
Completed [35/43] in 1.22s, 11 tests, 1 error <<< FAILURES!

Log from integ test cluster:

Cluster client_rest-high-level_integTestCluster - node 0 log excerpt:
(full log at /var/jenkins_home/jobs/elasticsearch-pipeline/workspace/elasticsearch/client/rest-high-level/build/cluster/integTestCluster node0/elasticsearch-7.0.0-SNAPSHOT/logs/client_rest-high-level_integTestCluster.log)
-----------------------------------------
[2019-01-17T15:38:44,034][INFO ][o.e.e.NodeEnvironment    ] [node-0] using [1] data paths, mounts [[/var/jenkins_home (/dev/nvme0n1p1)]], net usable_space [397.3gb], net total_space [468.4gb], types [ext4]
[2019-01-17T15:38:44,037][INFO ][o.e.e.NodeEnvironment    ] [node-0] heap size [494.9mb], compressed ordinary object pointers [true]
[2019-01-17T15:38:44,039][INFO ][o.e.n.Node               ] [node-0] node name [node-0], node ID [71qEwof_Rp2r8YOBlWNorw]
[2019-01-17T15:38:44,039][INFO ][o.e.n.Node               ] [node-0] version[7.0.0-SNAPSHOT], pid[49947], build[default/zip/0b849e3/2019-01-17T14:55:42.754849Z], OS[Linux/4.18.0-13-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11/11+28]
[2019-01-17T15:38:44,040][INFO ][o.e.n.Node               ] [node-0] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-1700522338155642831, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Dio.netty.allocator.type=unpooled, -Des.scripting.update.ctx_in_params=false, -Xms512m, -Xmx512m, -ea, -esa, -Des.path.home=/var/jenkins_home/jobs/elasticsearch-pipeline/workspace/elasticsearch/client/rest-high-level/build/cluster/integTestCluster node0/elasticsearch-7.0.0-SNAPSHOT, -Des.path.conf=/var/jenkins_home/jobs/elasticsearch-pipeline/workspace/elasticsearch/client/rest-high-level/build/cluster/integTestCluster node0/elasticsearch-7.0.0-SNAPSHOT/config, -Des.distribution.flavor=default, -Des.distribution.type=zip]
[2019-01-17T15:38:44,040][WARN ][o.e.n.Node               ] [node-0] version [7.0.0-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
[2019-01-17T15:38:45,628][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [aggs-matrix-stats]
[2019-01-17T15:38:45,629][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [analysis-common]
[2019-01-17T15:38:45,629][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [ingest-common]
[2019-01-17T15:38:45,629][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [ingest-geoip]
[2019-01-17T15:38:45,629][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [ingest-user-agent]
[2019-01-17T15:38:45,629][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [lang-expression]
[2019-01-17T15:38:45,629][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [lang-mustache]
[2019-01-17T15:38:45,629][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [lang-painless]
[2019-01-17T15:38:45,629][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [mapper-extras]
[2019-01-17T15:38:45,629][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [parent-join]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [percolator]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [rank-eval]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [reindex]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [repository-url]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [transport-netty4]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [x-pack-ccr]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [x-pack-core]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [x-pack-deprecation]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [x-pack-graph]
[2019-01-17T15:38:45,630][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [x-pack-ilm]
[2019-01-17T15:38:45,631][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [x-pack-logstash]
[2019-01-17T15:38:45,631][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [x-pack-ml]
[2019-01-17T15:38:45,631][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [x-pack-monitoring]
[2019-01-17T15:38:45,631][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [x-pack-rollup]
[2019-01-17T15:38:45,631][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [x-pack-security]
[2019-01-17T15:38:45,631][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [x-pack-sql]
[2019-01-17T15:38:45,631][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [x-pack-upgrade]
[2019-01-17T15:38:45,631][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [x-pack-watcher]
[2019-01-17T15:38:45,633][INFO ][o.e.p.PluginsService     ] [node-0] no plugins loaded
[2019-01-17T15:38:55,216][INFO ][o.e.x.s.a.s.FileRolesStore] [node-0] parsed [0] roles from file [/var/jenkins_home/jobs/elasticsearch-pipeline/workspace/elasticsearch/client/rest-high-level/build/cluster/integTestCluster node0/elasticsearch-7.0.0-SNAPSHOT/config/roles.yml]
[2019-01-17T15:38:58,538][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-0] [controller/50200] [Main.cc@109] controller (64 bit): Version 7.0.0-SNAPSHOT (Build cac31ab73e71c1) Copyright (c) 2019 Elasticsearch BV
[2019-01-17T15:38:59,816][DEBUG][o.e.a.ActionModule       ] [node-0] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-01-17T15:39:00,087][INFO ][o.e.d.DiscoveryModule    ] [node-0] using discovery type [zen2] and host providers [settings, file]
[2019-01-17T15:39:00,666][INFO ][o.e.n.Node               ] [node-0] initialized
[2019-01-17T15:39:00,666][INFO ][o.e.n.Node               ] [node-0] starting ...
[2019-01-17T15:39:00,811][INFO ][o.e.t.TransportService   ] [node-0] publish_address {127.0.0.1:38429}, bound_addresses {127.0.0.1:38429}
[2019-01-17T15:39:00,822][WARN ][o.e.b.BootstrapChecks    ] [node-0] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2019-01-17T15:39:00,822][WARN ][o.e.b.BootstrapChecks    ] [node-0] HTTPS is required in order to use the token service; please enable HTTPS using the [xpack.security.http.ssl.enabled] setting or disable the token service using the [xpack.security.authc.token.enabled] setting
[2019-01-17T15:39:00,832][WARN ][o.e.d.z.FileBasedUnicastHostsProvider] [node-0] expected, but did not find, a dynamic hosts list at [/var/jenkins_home/jobs/elasticsearch-pipeline/workspace/elasticsearch/client/rest-high-level/build/cluster/integTestCluster node0/elasticsearch-7.0.0-SNAPSHOT/config/unicast_hosts.txt]
[2019-01-17T15:39:00,858][INFO ][o.e.c.c.Coordinator      ] [node-0] setting initial configuration to VotingConfiguration{71qEwof_Rp2r8YOBlWNorw}
[2019-01-17T15:39:00,970][INFO ][o.e.c.s.MasterService    ] [node-0] elected-as-master ([1] nodes joined)[{node-0}{71qEwof_Rp2r8YOBlWNorw}{qPGV8i6PQsuMbWnX3a2KyQ}{127.0.0.1}{127.0.0.1:38429}{ml.machine_memory=67473756160, xpack.installed=true, testattr=test, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, reason: master node changed {previous [], current [{node-0}{71qEwof_Rp2r8YOBlWNorw}{qPGV8i6PQsuMbWnX3a2KyQ}{127.0.0.1}{127.0.0.1:38429}{ml.machine_memory=67473756160, xpack.installed=true, testattr=test, ml.max_open_jobs=20}]}
[2019-01-17T15:39:00,994][INFO ][o.e.c.s.ClusterApplierService] [node-0] master node changed {previous [], current [{node-0}{71qEwof_Rp2r8YOBlWNorw}{qPGV8i6PQsuMbWnX3a2KyQ}{127.0.0.1}{127.0.0.1:38429}{ml.machine_memory=67473756160, xpack.installed=true, testattr=test, ml.max_open_jobs=20}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2019-01-17T15:39:01,067][INFO ][o.e.h.AbstractHttpServerTransport] [node-0] publish_address {127.0.0.1:35141}, bound_addresses {127.0.0.1:35141}
[2019-01-17T15:39:01,069][INFO ][o.e.n.Node               ] [node-0] started
[2019-01-17T15:39:01,119][INFO ][o.e.g.GatewayService     ] [node-0] recovered [0] indices into cluster_state
... SKIPPED 34 LINES ...
[2019-01-17T15:40:12,719][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=yfUFRnNwTFOOltzVl94t0A, pollTimeout=1m, maxBatchSize=32mb}]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2019-01-17T15:40:12,719][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=yfUFRnNwTFOOltzVl94t0A, pollTimeout=1m, maxBatchSize=32mb}]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2019-01-17T15:40:12,719][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=yfUFRnNwTFOOltzVl94t0A, pollTimeout=1m, maxBatchSize=32mb}]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
... SKIPPED 32 LINES ...
[2019-01-17T15:40:13,328][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=0PJlSPs7RgSddZsb7_xGDQ, pollTimeout=1m, maxBatchSize=32mb}]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2019-01-17T15:40:13,328][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=0PJlSPs7RgSddZsb7_xGDQ, pollTimeout=1m, maxBatchSize=32mb}]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
... SKIPPED 38 LINES ...
[2019-01-17T15:40:14,054][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=3w2wEnMFQse4DmhDi4OnCg, pollTimeout=1m, maxBatchSize=32mb}]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2019-01-17T15:40:14,054][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=3w2wEnMFQse4DmhDi4OnCg, pollTimeout=1m, maxBatchSize=32mb}]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
... SKIPPED 71 LINES ...
[2019-01-17T15:40:14,915][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=gYfHgM-FQ2eWidG7sUFXYg, pollTimeout=1m, maxBatchSize=32mb}]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2019-01-17T15:40:14,915][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=gYfHgM-FQ2eWidG7sUFXYg, pollTimeout=1m, maxBatchSize=32mb}]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
... SKIPPED 75 LINES ...
[2019-01-17T15:40:15,976][ERROR][o.e.x.w.Watcher          ] [node-0] triggered watches could not be deleted [my_watch_id_1c2f9cc6-90a1-4168-819d-d2cf4e7c51f1-2019-01-17T15:40:15.808Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]]
... SKIPPED 14 LINES ...
[2019-01-17T15:40:16,464][ERROR][o.e.x.w.Watcher          ] [node-0] triggered watches could not be deleted [my_watch_id_3ade6bac-354c-4bbb-ac81-437da2668d91-2019-01-17T15:40:16.352Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]]
[2019-01-17T15:40:16,512][ERROR][o.e.x.w.Watcher          ] [node-0] triggered watches could not be deleted [my_watch_id_9e658efa-3913-4bf9-9689-c61d46a9ff30-2019-01-17T15:40:16.510Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]]
... SKIPPED 198 LINES ...
[2019-01-17T15:40:23,212][WARN ][r.suppressed             ] [node-0] path: /_migration/upgrade/test, params: {index=test, wait_for_completion=true}
java.lang.IllegalStateException: Index [test] cannot be upgraded
	at org.elasticsearch.xpack.upgrade.IndexUpgradeService.upgrade(IndexUpgradeService.java:115) ~[?:?]
	at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:59) ~[?:?]
	at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:25) ~[?:?]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:208) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 8 LINES ...
[2019-01-17T15:40:23,395][WARN ][o.e.t.LoggingTaskListener] [node-0] 1869 failed with exception
java.lang.IllegalStateException: Index [test] cannot be upgraded
	at org.elasticsearch.xpack.upgrade.IndexUpgradeService.upgrade(IndexUpgradeService.java:115) ~[?:?]
	at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:59) ~[?:?]
	at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:25) ~[?:?]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:208) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 560 LINES ...
[2019-01-17T15:40:42,592][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [node-0] [test-start-datafeed] [autodetect/56095] [CBucketGatherer.cc@466] No counts available at 1547739580, current bucket = [1547739585,1547739590)
... SKIPPED 6 LINES ...
[2019-01-17T15:40:42,717][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [node-0] [test-start-datafeed] [autodetect/56095] [CBucketGatherer.cc@466] No counts available at 1547739580, current bucket = [1547739585,1547739590)
... SKIPPED 1422 LINES ...
[2019-01-17T15:41:14,572][WARN ][r.suppressed             ] [node-0] path: /_migration/upgrade/test, params: {index=test, wait_for_completion=true}
java.lang.IllegalStateException: Index [test] cannot be upgraded
	at org.elasticsearch.xpack.upgrade.IndexUpgradeService.upgrade(IndexUpgradeService.java:115) ~[?:?]
	at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:59) ~[?:?]
	at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:25) ~[?:?]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:208) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 17 LINES ...
[2019-01-17T15:41:14,783][WARN ][o.e.t.LoggingTaskListener] [node-0] 17987 failed with exception
java.lang.IllegalStateException: Index [test] cannot be upgraded
	at org.elasticsearch.xpack.upgrade.IndexUpgradeService.upgrade(IndexUpgradeService.java:115) ~[?:?]
	at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:59) ~[?:?]
	at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:25) ~[?:?]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:208) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 9 LINES ...
[2019-01-17T15:41:14,989][WARN ][r.suppressed             ] [node-0] path: /_migration/upgrade/test, params: {index=test, wait_for_completion=true}
java.lang.IllegalStateException: Index [test] cannot be upgraded
	at org.elasticsearch.xpack.upgrade.IndexUpgradeService.upgrade(IndexUpgradeService.java:115) ~[?:?]
	at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:59) ~[?:?]
	at org.elasticsearch.xpack.upgrade.actions.TransportIndexUpgradeAction.masterOperation(TransportIndexUpgradeAction.java:25) ~[?:?]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:208) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 76 LINES ...
[2019-01-17T15:41:16,323][WARN ][o.e.r.b.FileRestoreContext] [node-0] [[restored_index][0]] [test_snapshot/Ti2vomt7QSqQK1p3ZCiIrw] Can't read metadata from store, will not reuse local files during restore
org.apache.lucene.index.IndexNotFoundException: no segments* file found in store(ByteSizeCachingDirectory(HybridDirectory@/var/jenkins_home/jobs/elasticsearch-pipeline/workspace/elasticsearch/client/rest-high-level/build/cluster/integTestCluster node0/elasticsearch-7.0.0-SNAPSHOT/data/nodes/0/indices/e9g2MZX_R52ynn7CqrkW0w/0/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@3bf9292)): files: [write.lock]
	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:675) ~[lucene-core-8.0.0-snapshot-a1c6e642aa.jar:8.0.0-snapshot-a1c6e642aa a1c6e642aad90d3615b4c71bf261a5aad7e32369 - nknize - 2019-01-02 14:49:37]
	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0-snapshot-a1c6e642aa.jar:8.0.0-snapshot-a1c6e642aa a1c6e642aad90d3615b4c71bf261a5aad7e32369 - nknize - 2019-01-02 14:49:37]
	at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0-snapshot-a1c6e642aa.jar:8.0.0-snapshot-a1c6e642aa a1c6e642aad90d3615b4c71bf261a5aad7e32369 - nknize - 2019-01-02 14:49:37]
	at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:200) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.store.Store.access$200(Store.java:128) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:839) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.store.Store$MetadataSnapshot.<init>(Store.java:772) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.store.Store.getMetadata(Store.java:286) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1173) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restore(FileRestoreContext.java:121) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:861) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:464) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:279) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:302) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:277) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1683) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$7(IndexShard.java:2207) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 211 LINES ...
[2019-01-17T15:41:19,723][ERROR][o.e.x.w.Watcher          ] [node-0] triggered watches could not be deleted [YWplWKWVsJ_15aad032-751e-4927-8aa0-9a37d0fbbcae-2019-01-17T15:41:19.722Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]]
... SKIPPED 61 LINES ...
[2019-01-17T15:41:20,791][ERROR][o.e.x.w.Watcher          ] [node-0] triggered watches could not be deleted [opJZLzKeuO_d35244fa-f80b-4946-ba37-016360092086-2019-01-17T15:41:20.712Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]]
... SKIPPED 252 LINES ...
[2019-01-17T15:41:27,524][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=0, maxOperationCount=5120, shardId=[logs-20200101][0], expectedHistoryUUID=MckyWE5cR6iDYFUhpeSaaA, pollTimeout=1m, maxBatchSize=32mb}]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
... SKIPPED 37 LINES ...
[2019-01-17T15:41:29,982][WARN ][o.e.x.c.a.ShardFollowNodeTask] [node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: No shard available for [Request{fromSeqNo=2, maxOperationCount=5120, shardId=[leader][0], expectedHistoryUUID=BIrVQ7NATVunPovqvTyZHQ, pollTimeout=1m, maxBatchSize=32mb}]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:233) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:219) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1100(TransportSingleShardAction.java:144) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:276) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:451) ~[?:?]
	at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:384) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$5(GlobalCheckpointListeners.java:220) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$notifyListeners$6(GlobalCheckpointListeners.java:214) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:38429][indices:data/read/xpack/ccr/shard_changes[s]]
Caused by: org.elasticsearch.index.shard.IndexShardClosedException: CurrentState[CLOSED] Closed
	at org.elasticsearch.index.shard.GlobalCheckpointListeners.close(GlobalCheckpointListeners.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:1234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.closeShard(IndexService.java:447) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.removeShard(IndexService.java:430) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:634) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:234) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
... SKIPPED 1937 LINES ...
[2019-01-17T15:42:04,191][ERROR][o.e.x.m.j.p.a.w.JsonDataToProcessWriter] [node-0] Cannot parse timestamp '' as epoch value
[2019-01-17T15:42:04,191][ERROR][o.e.x.m.j.p.a.w.JsonDataToProcessWriter] [node-0] Cannot parse timestamp '' as epoch value
[2019-01-17T15:42:04,196][ERROR][o.e.x.m.j.p.a.w.JsonDataToProcessWriter] [node-0] Cannot parse timestamp '' as epoch value
... SKIPPED 97 LINES ...
[2019-01-17T15:42:06,193][WARN ][o.e.x.m.d.DatafeedManager] [node-0] [start-datafeed-job] Datafeed lookback retrieved no data
... SKIPPED 437 LINES ...
[2019-01-17T15:42:15,512][ERROR][o.e.x.i.IndexLifecycleRunner] [node-0] policy [my_policy] for index [my_index] failed on step [{"phase":"warm","action":"shrink","name":"shrink"}]. Moving to ERROR step
java.lang.IllegalArgumentException: can't shrink an index with only one shard
	at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validateShrinkIndex(MetaDataCreateIndexService.java:664) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.prepareResizeIndexSettings(MetaDataCreateIndexService.java:744) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$IndexCreationTask.execute(MetaDataCreateIndexService.java:408) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:685) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:310) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:210) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2019-01-17T15:42:15,555][ERROR][o.e.x.i.IndexLifecycleRunner] [node-0] policy [my_policy] for index [my_index] failed on step [{"phase":"warm","action":"shrink","name":"shrink"}]. Moving to ERROR step
java.lang.IllegalArgumentException: can't shrink an index with only one shard
	at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validateShrinkIndex(MetaDataCreateIndexService.java:664) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.prepareResizeIndexSettings(MetaDataCreateIndexService.java:744) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$IndexCreationTask.execute(MetaDataCreateIndexService.java:408) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:685) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:310) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:210) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 32 LINES ...
[2019-01-17T15:42:16,313][WARN ][o.e.l.LicenseService     ] [node-0] LICENSE [EXPIRED] ON [THURSDAY, JANUARY 01, 1970].
# IF YOU HAVE A NEW LICENSE, PLEASE UPDATE IT. OTHERWISE, PLEASE REACH OUT TO
# YOUR SUPPORT CONTACT.
# 
# COMMERCIAL PLUGINS OPERATING WITH REDUCED FUNCTIONALITY
# - security
#  - Cluster health, cluster stats and indices stats operations are blocked
#  - All data operations (read and write) continue to work
# - watcher
#  - PUT / GET watch APIs are disabled, DELETE watch API continues to work
#  - Watches execute and write to the history
#  - The actions of the watches don't execute
# - monitoring
#  - The agent will stop collecting cluster and indices metrics
#  - The agent will stop automatically cleaning indices older than [xpack.monitoring.history.duration]
# - graph
#  - Graph explore APIs are disabled
# - ml
#  - Machine learning APIs are disabled
# - logstash
#  - Logstash will continue to poll centrally-managed pipelines
# - beats
#  - Beats will continue to poll centrally-managed configuration
# - deprecation
#  - Deprecation APIs are disabled
# - upgrade
#  - Upgrade API is disabled
# - sql
#  - SQL support is disabled
# - rollup
#  - Creating and Starting rollup jobs will no longer be allowed.
#  - Stopping/Deleting existing jobs, RollupCaps API and RollupSearch continue to function.
[2019-01-17T15:42:16,313][WARN ][o.e.l.LicenseService     ] [node-0] LICENSE [EXPIRED] ON [THURSDAY, JANUARY 01, 1970].
# IF YOU HAVE A NEW LICENSE, PLEASE UPDATE IT. OTHERWISE, PLEASE REACH OUT TO
# YOUR SUPPORT CONTACT.
# 
# COMMERCIAL PLUGINS OPERATING WITH REDUCED FUNCTIONALITY
# - security
#  - Cluster health, cluster stats and indices stats operations are blocked
#  - All data operations (read and write) continue to work
# - watcher
#  - PUT / GET watch APIs are disabled, DELETE watch API continues to work
#  - Watches execute and write to the history
#  - The actions of the watches don't execute
# - monitoring
#  - The agent will stop collecting cluster and indices metrics
#  - The agent will stop automatically cleaning indices older than [xpack.monitoring.history.duration]
# - graph
#  - Graph explore APIs are disabled
# - ml
#  - Machine learning APIs are disabled
# - logstash
#  - Logstash will continue to poll centrally-managed pipelines
# - beats
#  - Beats will continue to poll centrally-managed configuration
# - deprecation
#  - Deprecation APIs are disabled
# - upgrade
#  - Upgrade API is disabled
# - sql
#  - SQL support is disabled
# - rollup
#  - Creating and Starting rollup jobs will no longer be allowed.
#  - Stopping/Deleting existing jobs, RollupCaps API and RollupSearch continue to function.
[2019-01-17T15:42:16,323][WARN ][o.e.x.c.a.AutoFollowCoordinator] [node-0] skipping auto-follower coordination
org.elasticsearch.ElasticsearchSecurityException: current license is non-compliant for [ccr]
	at org.elasticsearch.license.LicenseUtils.newComplianceException(LicenseUtils.java:27) ~[x-pack-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator.updateAutoFollowers(AutoFollowCoordinator.java:178) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator.clusterChanged(AutoFollowCoordinator.java:291) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateListeners$7(ClusterApplierService.java:498) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) [?:?]
	at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) [?:?]
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) [?:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:495) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:478) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 6 LINES ...
[2019-01-17T15:42:16,454][WARN ][o.e.x.c.a.AutoFollowCoordinator] [node-0] skipping auto-follower coordination
org.elasticsearch.ElasticsearchSecurityException: current license is non-compliant for [ccr]
	at org.elasticsearch.license.LicenseUtils.newComplianceException(LicenseUtils.java:27) ~[x-pack-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator.updateAutoFollowers(AutoFollowCoordinator.java:178) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator.clusterChanged(AutoFollowCoordinator.java:291) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateListeners$7(ClusterApplierService.java:498) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) [?:?]
	at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) [?:?]
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) [?:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:495) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:478) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 51 LINES ...
[2019-01-17T15:42:17,447][WARN ][o.e.r.b.FileRestoreContext] [node-0] [[test_index_restored][0]] [snapshot_1/V9FwD0UnSiGM6p5AFEe3hA] Can't read metadata from store, will not reuse local files during restore
org.apache.lucene.index.IndexNotFoundException: no segments* file found in store(ByteSizeCachingDirectory(HybridDirectory@/var/jenkins_home/jobs/elasticsearch-pipeline/workspace/elasticsearch/client/rest-high-level/build/cluster/integTestCluster node0/elasticsearch-7.0.0-SNAPSHOT/data/nodes/0/indices/z7f-71byQm6UyXJLR67gtg/0/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@3bf9292)): files: [write.lock]
	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:675) ~[lucene-core-8.0.0-snapshot-a1c6e642aa.jar:8.0.0-snapshot-a1c6e642aa a1c6e642aad90d3615b4c71bf261a5aad7e32369 - nknize - 2019-01-02 14:49:37]
	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0-snapshot-a1c6e642aa.jar:8.0.0-snapshot-a1c6e642aa a1c6e642aad90d3615b4c71bf261a5aad7e32369 - nknize - 2019-01-02 14:49:37]
	at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0-snapshot-a1c6e642aa.jar:8.0.0-snapshot-a1c6e642aa a1c6e642aad90d3615b4c71bf261a5aad7e32369 - nknize - 2019-01-02 14:49:37]
	at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:200) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.store.Store.access$200(Store.java:128) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:839) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.store.Store$MetadataSnapshot.<init>(Store.java:772) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.store.Store.getMetadata(Store.java:286) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1173) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restore(FileRestoreContext.java:121) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:861) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:464) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:279) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:302) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:277) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1683) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$7(IndexShard.java:2207) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
... SKIPPED 15 LINES ...
[2019-01-17T15:42:17,649][WARN ][r.suppressed             ] [node-0] path: /_snapshot/test_repository/test_snapshot, params: {master_timeout=30s, repository=test_repository, snapshot=test_snapshot}
org.elasticsearch.repositories.RepositoryException: [test_repository] concurrent modification of the index-N file, expected current generation [13], actual current generation [14] - possibly due to simultaneous snapshot deletion requests
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeIndexGen(BlobStoreRepository.java:712) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.deleteSnapshot(BlobStoreRepository.java:420) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.snapshots.SnapshotsService.lambda$deleteSnapshotFromRepository$5(SnapshotsService.java:1337) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
=========================================

> Task :client:rest-high-level:integTestRunner FAILED
> Task :client:rest-high-level:integTestCluster#stop
@jaymode jaymode added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI labels Jan 17, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@original-brownbear original-brownbear self-assigned this Jan 17, 2019
@original-brownbear
Copy link
Member

original-brownbear commented Jan 18, 2019 •
edited
Loading

@jaymode this looks like some actual bug in snapshotting, there shouldn't be any concurrent actions on this file. Just to be sure, you're not using any kind of weird file system/disk are you? :)

nevermind figured out where this is coming from, no it :)

original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Jan 18, 2019
@original-brownbear
Fix Race in Concurrent Snapshot Delete and Create
cdf9597 
* The repo id was determined wrong when the delete picked up on an in progress snapshot
  * NOTE: This solution is still a best-effort fix and there's a slight chance of running into concurrency issues here
when multiple create and delete requests for the same snapshot name are happening concurrently, but these require a sequence
of multiple cluster state updates between the changed method reading the genId and submitting its cluster state update task
* Added test reproduced the issue reliably in about 50% of runs
* Closes elastic#37581
@original-brownbear original-brownbear mentioned this issue Jan 18, 2019
Merged
@original-brownbear
Copy link
Member

Fix incoming in #37612

original-brownbear added a commit that referenced this issue Jan 21, 2019
@original-brownbear
Fix Race in Concurrent Snapshot Delete and Create (#37612)
3a3f5b3 
* The repo id was determined wrong when the delete picked up on an in progress snapshot
  * NOTE: This solution is still a best-effort fix and there's a slight chance of running into concurrency issues here
when multiple create and delete requests for the same snapshot name are happening concurrently, but these require a sequence
of multiple cluster state updates between the changed method reading the genId and submitting its cluster state update task
* Added test reproduced the issue reliably in about 50% of runs
* Closes #37581
@dakrone
Copy link
Member

dakrone commented Nov 15, 2019

This failed again today in a PR build: https://gradle-enterprise.elastic.co/s/67kanac5qy4ok but this time with a slightly different error:

00:32:14 org.elasticsearch.client.SnapshotIT > testCreateSnapshot FAILED
00:32:14     org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=repository_exception, reason=[test_repository] Could not determine repository generation from root blobs]
00:32:14         at __randomizedtesting.SeedInfo.seed([F37D9ECE4D19A47B:D1B0B7D7BAA5662]:0)
00:32:14         at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
00:32:14         at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1793)
00:32:14         at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1770)
00:32:14         at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1527)
00:32:14         at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1484)
00:32:14         at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
00:32:14         at org.elasticsearch.client.SnapshotClient.delete(SnapshotClient.java:344)
00:32:14         at org.elasticsearch.client.ESRestHighLevelClientTestCase.execute(ESRestHighLevelClientTestCase.java:90)
00:32:14         at org.elasticsearch.client.ESRestHighLevelClientTestCase.execute(ESRestHighLevelClientTestCase.java:81)
00:32:14         at org.elasticsearch.client.SnapshotIT.testCreateSnapshot(SnapshotIT.java:167)
00:32:14 
00:32:14         Caused by:
00:32:14         org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=no_such_file_exception, reason=/dev/shm/elastic+elasticsearch+pull-request-1/client/rest-high-level/build/testclusters/integTest-0/repo/index-12]
00:32:14             at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496)
00:32:14             at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407)
00:32:14             at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:437)
00:32:14             at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:603)
00:32:14             at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:169)
00:32:14             ... 9 more

I was unable to reproduce this with

./gradlew ':client:rest-high-level:integTestRunner' --tests "org.elasticsearch.client.SnapshotIT.testCreateSnapshot" -Dtests.seed=F37D9ECE4D19A47B -Dtests.security.manager=true -Dtests.locale=ar-AE -Dtests.timezone=America/Mazatlan -Dcompiler.java=12 -Druntime.java=11

@original-brownbear I think this is something you've fixed before, does it just need retries?

@dakrone dakrone reopened this Nov 15, 2019
@dakrone dakrone mentioned this issue Nov 15, 2019
Merged
@original-brownbear
Copy link
Member

@dakrone huh, this funny enough is a completely unrelated bug to the one initially fixed here :)

The problem is that we're looping over all the files in a directory and then reading their sizes one by one when doing listBlobsByPrefix => if we file gets concurrently deleted => that failure.

-> I'll open a PR to fix that in a bit.

original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Nov 15, 2019
@original-brownbear
Make FsBlobContainer Listing Resilient to Concurrent Modifications
aed677a 
If we list out files in a folder via the lazily computed directory
stream, we have to deal with concurrent deletes when reading the file
attributes since we don't have a lock on the directory in any way.

Closes elastic#37581
@original-brownbear original-brownbear mentioned this issue Nov 15, 2019
Merged
@original-brownbear
Copy link
Member

#49142 Should do the trick here

original-brownbear added a commit that referenced this issue Nov 15, 2019
@original-brownbear
Make FsBlobContainer Listing Resilient to Concurrent Modifications (#…
91aed88 
…49142)

* Make FsBlobContainer Listing Resilient to Concurrent Modifications

If we list out files in a folder via the lazily computed directory
stream, we have to deal with concurrent deletes when reading the file
attributes since we don't have a lock on the directory in any way.

Closes #37581
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Nov 15, 2019
@original-brownbear
Make FsBlobContainer Listing Resilient to Concurrent Modifications (e…
6f7e71c 
…lastic#49142)

* Make FsBlobContainer Listing Resilient to Concurrent Modifications

If we list out files in a folder via the lazily computed directory
stream, we have to deal with concurrent deletes when reading the file
attributes since we don't have a lock on the directory in any way.

Closes elastic#37581
@original-brownbear original-brownbear mentioned this issue Nov 15, 2019
Merged
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Nov 15, 2019
@original-brownbear
Make FsBlobContainer Listing Resilient to Concurrent Modifications (e…
2a3da2a 
…lastic#49142)

* Make FsBlobContainer Listing Resilient to Concurrent Modifications

If we list out files in a folder via the lazily computed directory
stream, we have to deal with concurrent deletes when reading the file
attributes since we don't have a lock on the directory in any way.

Closes elastic#37581
@original-brownbear original-brownbear mentioned this issue Nov 15, 2019
Merged
original-brownbear added a commit that referenced this issue Nov 15, 2019
@original-brownbear
Make FsBlobContainer Listing Resilient to Concurrent Modifications (#…
2886d4c 
…49142) (#49176)

* Make FsBlobContainer Listing Resilient to Concurrent Modifications

If we list out files in a folder via the lazily computed directory
stream, we have to deal with concurrent deletes when reading the file
attributes since we don't have a lock on the directory in any way.

Closes #37581
original-brownbear added a commit that referenced this issue Nov 15, 2019
@original-brownbear
Make FsBlobContainer Listing Resilient to Concurrent Modifications (#…
7e2e903 
…49142) (#49177)

* Make FsBlobContainer Listing Resilient to Concurrent Modifications

If we list out files in a folder via the lazily computed directory
stream, we have to deal with concurrent deletes when reading the file
attributes since we don't have a lock on the directory in any way.

Closes #37581
This was referenced Feb 3, 2020
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@original-brownbear original-brownbear

Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
4 participants
@dakrone @jaymode @original-brownbear @elasticmachine