[CI] TransformIT testStopWaitForCheckpoint failing

[mark-vieira]

Might be something up here. When running this locally, the test just hung. Tagging security here as well since this might be FIPS related?

Build scan:
https://gradle-enterprise.elastic.co/s/hksi6yfmojuq4/tests/:x-pack:plugin:transform:qa:multi-node-tests:javaRestTest/org.elasticsearch.xpack.transform.integration.TransformIT/testStopWaitForCheckpoint

Reproduction line:
./gradlew ':x-pack:plugin:transform:qa:multi-node-tests:javaRestTest' --tests "org.elasticsearch.xpack.transform.integration.TransformIT.testStopWaitForCheckpoint" -Dtests.seed=4E51A8494E673B01 -Dtests.security.manager=true -Dtests.locale=es-NI -Dtests.timezone=Etc/GMT+8 -Druntime.java=8 -Dtests.fips.enabled=true

Applicable branches:
7.12

Reproduces locally?:
Yes

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.transform.integration.TransformIT&tests.test=testStopWaitForCheckpoint

Failure excerpt:

java.lang.AssertionError: (No message provided)

  at org.junit.Assert.fail(Assert.java:86)
  at org.junit.Assert.assertTrue(Assert.java:41)
  at org.junit.Assert.assertTrue(Assert.java:52)
  at org.elasticsearch.xpack.transform.integration.TransformIT.testStopWaitForCheckpoint(TransformIT.java:302)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  at java.lang.Thread.run(Thread.java:748)

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[jkakavas]

It passes locally for me with 8.0.282-zulu ( which is what this ran with in CI ). @mark-vieira what 8 version did you run this with locally ? ( We only run with zulu8 and corretto8 in FIPS matrix currently, not with adoptopenjdk or oracle 8 because of #64379 )

[benwtrent]

From what I can tell, there were race conditions in the checkpoint storage.

All the transform test run logs:
testclusters.zip

The logs have the following, indicating some sort of conflict with storing the stats?:

2021-04-01T22:46:14,642][DEBUG][o.e.x.t.t.TransformIndexer] [javaRestTest-0] [transform-wait-for-checkpoint] Querying for changes: {"size":0,"query":{"bool":{"filter":[{"match_all":{"boost":1.0}},{"range":{"timestamp":{"from":1617317167124,"to":1617317173548,"include_lower":true,"include_upper":false,"format":"epoch_millis","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"_transform_change_collector":{"composite":{"size":500,"sources":[{"by-business":{"terms":{"field":"business_id","missing_bucket":false,"order":"asc"}}},{"by-user":{"terms":{"field":"user_id","missing_bucket":false,"order":"asc"}}}]}}}}
[2021-04-01T22:46:14,740][DEBUG][o.e.x.t.t.ClientTransformIndexer] [javaRestTest-0] [transform-wait-for-checkpoint] updating persistent state of transform to [{"task_state":"started","indexer_state":"indexing","checkpoint":2,"progress":{"docs_indexed":0,"docs_processed":0},"should_stop_at_checkpoint":true}].
[2021-04-01T22:46:14,777][ERROR][o.e.x.t.t.ClientTransformIndexer] [javaRestTest-0] [transform-wait-for-checkpoint] updating stats of transform failed.
java.lang.RuntimeException: Failed to persist transform statistics for transform [transform-wait-for-checkpoint]
	at org.elasticsearch.xpack.transform.persistence.IndexBasedTransformConfigManager.lambda$putOrUpdateTransformStoredDoc$14(IndexBasedTransformConfigManager.java:528) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:125) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:92) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:172) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.lambda$wrapBulkResponse$0(TransportSingleItemBulkWriteAction.java:53) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:165) ~[?:?]
	at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:167) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$5.onResponse(ActionListener.java:286) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:546) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:527) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:516) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:83) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:77) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:165) ~[?:?]
	at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:167) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishOnSuccess(TransportReplicationAction.java:877) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:796) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:787) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$6.handleResponse(TransportService.java:739) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1280) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1358) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1338) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:41) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:32) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:16) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$5.onResponse(ActionListener.java:286) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$runWithPrimaryShardReference$2(TransportReplicationAction.java:413) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:76) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.ReplicationOperation.finish(ReplicationOperation.java:336) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.ReplicationOperation.decPendingAndFinishIfNeeded(ReplicationOperation.java:317) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.ReplicationOperation.access$300(ReplicationOperation.java:46) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.ReplicationOperation$2.onResponse(ReplicationOperation.java:201) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.ReplicationOperation$2.onResponse(ReplicationOperation.java:194) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.RetryableAction$RetryingListener.onResponse(RetryableAction.java:133) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:43) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$6.handleResponse(TransportService.java:739) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1280) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:291) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:275) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:128) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:84) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:690) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:131) [elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:106) [elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:71) [elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:63) [transport-netty4-client-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: org.elasticsearch.index.engine.VersionConflictEngineException: [data_frame_transform_state_and_stats-transform-wait-for-checkpoint]: version conflict, required seqNo [7], primary term [1]. current document has seqNo [9] and primary term [1]
[2021-04-01T22:46:14,824][DEBUG][o.e.x.t.t.TransformIndexer] [javaRestTest-0] [transform-wait-for-checkpoint] Querying for data: {"size":0,"query":{"bool":{"filter":[{"match_all":{"boost":1.0}},{"range":{"timestamp":{"from":null,"to":1617317173548,"include_lower":true,"include_upper":false,"format":"epoch_millis","boost":1.0}}},{"bool":{"filter":[{"terms":{"business_id":["business_19","business_2","business_1","business_18","business_0","business_17","business_16","business_15","business_6","business_14","business_5","business_13","business_4","business_12","business_3","business_9","business_8","business_7","business_22","business_21","business_20","business_24","business_23","business_11","business_10"],"boost":1.0}},{"range":{"timestamp":{"from":1617235200000,"to":null,"include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}},{"terms":{"user_id":["user_43","user_44"],"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"_transform":{"composite":{"size":500,"sources":[{"by-business":{"terms":{"field":"business_id","missing_bucket":false,"order":"asc"}}},{"by-day":{"date_histogram":{"field":"timestamp","missing_bucket":false,"order":"asc","calendar_interval":"1d"}}},{"by-user":{"terms":{"field":"user_id","missing_bucket":false,"order":"asc"}}}]},"aggregations":{"review_score":{"avg":{"field":"stars"}},"timestamp":{"max":{"field":"timestamp"}}}}}}
[2021-04-01T22:46:15,055][DEBUG][o.e.x.t.t.TransformTask  ] [javaRestTest-0] [transform-wait-for-checkpoint] attempted to set task to stop at checkpoint [false] with state [{"task_state":"started","indexer_state":"indexing","checkpoint":2,"progress":{"docs_indexed":50,"docs_processed":50},"should_stop_at_checkpoint":true}]
[2021-04-01T22:46:15,085][DEBUG][o.e.x.t.t.ClientTransformIndexer] [javaRestTest-0] [transform-wait-for-checkpoint] updating persistent state of transform to [{"task_state":"started","indexer_state":"indexing","position":{"indexer_position":{"by-business":"business_9","by-day":1617235200000,"by-user":"user_44"}},"checkpoint":2,"progress":{"docs_indexed":50,"docs_processed":50},"should_stop_at_checkpoint":false}].
[2021-04-01T22:46:15,094][ERROR][o.e.x.t.t.ClientTransformIndexer] [javaRestTest-0] [transform-wait-for-checkpoint] updating stats of transform failed.
java.lang.RuntimeException: Failed to persist transform statistics for transform [transform-wait-for-checkpoint]
	at org.elasticsearch.xpack.transform.persistence.IndexBasedTransformConfigManager.lambda$putOrUpdateTransformStoredDoc$14(IndexBasedTransformConfigManager.java:528) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:125) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:92) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:172) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.lambda$wrapBulkResponse$0(TransportSingleItemBulkWriteAction.java:53) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:165) ~[?:?]
	at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:167) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$5.onResponse(ActionListener.java:286) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:546) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:527) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:516) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:83) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:77) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:165) ~[?:?]
	at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:167) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishOnSuccess(TransportReplicationAction.java:877) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:796) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:787) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$6.handleResponse(TransportService.java:739) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1280) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1358) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1338) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:41) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:32) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:16) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$5.onResponse(ActionListener.java:286) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$runWithPrimaryShardReference$2(TransportReplicationAction.java:413) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:76) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.ReplicationOperation.finish(ReplicationOperation.java:336) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.ReplicationOperation.decPendingAndFinishIfNeeded(ReplicationOperation.java:317) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.ReplicationOperation.access$300(ReplicationOperation.java:46) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.ReplicationOperation$2.onResponse(ReplicationOperation.java:201) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.ReplicationOperation$2.onResponse(ReplicationOperation.java:194) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.RetryableAction$RetryingListener.onResponse(RetryableAction.java:133) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:43) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$6.handleResponse(TransportService.java:739) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1280) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:291) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:275) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:128) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:84) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:690) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:131) [elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:106) [elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:71) [elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:63) [transport-netty4-client-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: org.elasticsearch.index.engine.VersionConflictEngineException: [data_frame_transform_state_and_stats-transform-wait-for-checkpoint]: version conflict, required seqNo [7], primary term [1]. current document has seqNo [9] and primary term [1]
[2021-04-01T22:46:15,111][DEBUG][o.e.x.t.t.TransformIndexer] [javaRestTest-0] [transform-wait-for-checkpoint] Querying for data: {"size":0,"query":{"bool":{"filter":[{"match_all":{"boost":1.0}},{"range":{"timestamp":{"from":null,"to":1617317173548,"include_lower":true,"include_upper":false,"format":"epoch_millis","boost":1.0}}},{"bool":{"filter":[{"terms":{"business_id":["business_19","business_2","business_1","business_18","business_0","business_17","business_16","business_15","business_6","business_14","business_5","business_13","business_4","business_12","business_3","business_9","business_8","business_7","business_22","business_21","business_20","business_24","business_23","business_11","business_10"],"boost":1.0}},{"range":{"timestamp":{"from":1617235200000,"to":null,"include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}},{"terms":{"user_id":["user_43","user_44"],"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"_transform":{"composite":{"size":500,"sources":[{"by-business":{"terms":{"field":"business_id","missing_bucket":false,"order":"asc"}}},{"by-day":{"date_histogram":{"field":"timestamp","missing_bucket":false,"order":"asc","calendar_interval":"1d"}}},{"by-user":{"terms":{"field":"user_id","missing_bucket":false,"order":"asc"}}}],"after":{"by-business":"business_9","by-day":1617235200000,"by-user":"user_44"}},"aggregations":{"review_score":{"avg":{"field":"stars"}},"timestamp":{"max":{"field":"timestamp"}}}}}}
[2021-04-01T22:46:15,210][INFO ][o.e.c.m.MetadataDeleteIndexService] [javaRestTest-0] [reviews-by-user-business-day/jr8mQI6wQLm9P_MILmo3Xg] deleting index
[2021-04-01T22:46:15,211][INFO ][o.e.c.m.MetadataDeleteIndexService] [javaRestTest-0] [.transform-notifications-000002/NoKWs7J3QaqcBn4FNeP7zg] deleting index
[2021-04-01T22:46:15,211][INFO ][o.e.c.m.MetadataDeleteIndexService] [javaRestTest-0] [wait-for-checkpoint-reviews/-sNhRY7HQ0GZoQtbJpCtug] deleting index
[2021-04-01T22:46:15,211][INFO ][o.e.c.m.MetadataDeleteIndexService] [javaRestTest-0] [.transform-internal-006/Q67WqvagSJKaQ7W_I2BNrw] deleting index
[2021-04-01T22:46:15,249][DEBUG][o.e.x.t.t.TransformIndexer] [javaRestTest-0] [transform-wait-for-checkpoint] Querying for changes: {"size":0,"query":{"bool":{"filter":[{"match_all":{"boost":1.0}},{"range":{"timestamp":{"from":1617317167124,"to":1617317173548,"include_lower":true,"include_upper":false,"format":"epoch_millis","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"_transform_change_collector":{"composite":{"size":500,"sources":[{"by-business":{"terms":{"field":"business_id","missing_bucket":false,"order":"asc"}}},{"by-user":{"terms":{"field":"user_id","missing_bucket":false,"order":"asc"}}}],"after":{"by-business":"business_9","by-user":"user_44"}}}}}
[2021-04-01T22:46:15,323][DEBUG][o.e.x.c.i.AsyncTwoPhaseIndexer] [javaRestTest-0] Finished indexing for job [transform-wait-for-checkpoint], saving state and shutting down.
[2021-04-01T22:46:15,395][DEBUG][o.e.x.t.TransformClusterStateListener] [javaRestTest-0] Skipped creating alias for deprecated data frame notifications index
[2021-04-01T22:46:15,397][DEBUG][o.e.x.t.t.TransformIndexer] [javaRestTest-0] [transform-wait-for-checkpoint] finished indexing for transform checkpoint [2].
[2021-04-01T22:46:15,398][DEBUG][o.e.x.t.t.ClientTransformIndexer] [javaRestTest-0] [transform-wait-for-checkpoint] updating persistent state of transform to [{"task_state":"started","indexer_state":"started","checkpoint":3,"progress":{"docs_indexed":50,"docs_processed":50},"should_stop_at_checkpoint":false}].
[2021-04-01T22:46:15,417][INFO ][o.e.c.m.MetadataCreateIndexService] [javaRestTest-0] [.transform-notifications-000002] creating index, cause [auto(bulk api)], templates [.transform-notifications-000002], shards [1]/[1]
[2021-04-01T22:46:15,463][DEBUG][o.e.x.t.TransformClusterStateListener] [javaRestTest-0] Skipped creating alias for deprecated data frame notifications index
[2021-04-01T22:46:15,623][INFO ][o.e.c.m.MetadataCreateIndexService] [javaRestTest-0] [.transform-internal-006] creating index, cause [auto(bulk api)], templates [], shards [1]/[1]
[2021-04-01T22:46:15,904][ERROR][o.e.x.t.t.ClientTransformIndexer] [javaRestTest-0] [transform-wait-for-checkpoint] updating stats of transform failed.
java.lang.RuntimeException: Failed to persist transform statistics for transform [transform-wait-for-checkpoint]
	at org.elasticsearch.xpack.transform.persistence.IndexBasedTransformConfigManager.lambda$putOrUpdateTransformStoredDoc$14(IndexBasedTransformConfigManager.java:528) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:125) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:92) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:172) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.lambda$wrapBulkResponse$0(TransportSingleItemBulkWriteAction.java:53) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:165) ~[?:?]
	at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:167) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$5.onResponse(ActionListener.java:286) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:546) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:527) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:516) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:83) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:77) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:165) ~[?:?]
	at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:167) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishOnSuccess(TransportReplicationAction.java:877) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:796) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:787) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$6.handleResponse(TransportService.java:739) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1280) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:291) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:275) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:128) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:84) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:690) ~[elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:131) [elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:106) [elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:71) [elasticsearch-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:63) [transport-netty4-client-7.12.1-SNAPSHOT.jar:7.12.1-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: org.elasticsearch.index.engine.VersionConflictEngineException: [data_frame_transform_state_and_stats-transform-wait-for-checkpoint]: version conflict, required seqNo [7], primary term [1]. but no document was found

[mark-vieira]

@mark-vieira what 8 version did you run this with locally ?

Ah, good call. I believe I used Oracle JDK locally. Seems this probably isn't FIPS related anyhow though.

[hendrikmuhs]

We had a similar one in #70416

It's fixed, however not backported to 7.12. This and the failures I see on Gradle Enterprise are 7.12.

The fix is a larger change, I think it should merge cleanly. Still wonder if we should accept such a large patch.

[martijnvg]

Another instances:

• https://gradle-enterprise.elastic.co/s/caxz56qf72tae
• https://gradle-enterprise.elastic.co/s/a6vkbhiuj45no

The fix is a larger change, I think it should merge cleanly. Still wonder if we should accept such a large patch.

Maybe this should be backported, since the refactoring fixes a bug too?

[hendrikmuhs]

I created a backport PR.