[Transfrom] prevent concurrent state persistence when indexer gets triggered during shutdown (#69551)

the latest state, during the 2 stages a new run might get triggered and
run into a race condition where a new state persists runs while the old
has not finished yet. This change prevents the trigger if the indexer is
in the described intermediate state

fixes #67121

Author: Hendrik Muhs

x-pack/plugin/transform/src/main/java/org/elasticsearch/xpack/transform/transforms/TransformIndexer.java

@@ -492,6 +492,22 @@ public synchronized boolean maybeTriggerAsyncJob(long now) {
             return false;
         }
 
+        /*
+         * ignore if indexer thread is shutting down (after finishing a checkpoint)
+         * shutting down means:
+         *  - indexer has finished a checkpoint and called onFinish
+         *  - indexer state has changed from indexing to started
+         *  - state persistence has been called but has _not_ returned yet
+         *
+         *  If we trigger the indexer in this situation the 2nd indexer thread might
+         *  try to save state at the same time, causing a version conflict
+         *  see gh#67121
+         */
+        if (indexerThreadShuttingDown) {
+            logger.debug("[{}] indexer thread is shutting down. Ignoring trigger.", getJobId());
+            return false;
+        }
+
         return super.maybeTriggerAsyncJob(now);
     }