Fix task manager polling flow controls

[mikecote]

Fixes #151938

In this PR, I'm re-writing the Task Manager poller so it doesn't run concurrently when timeouts occur while also fixing the issue where polling requests would pile up when polling takes time. To support this, I've also made the following changes:

• Removed the observable monitor and the xpack.task_manager.max_poll_inactivity_cycles setting
• Make the task store search and updateByQuery functions have no retries. This prevents the request from retrying 5x whenever a timeout occurs, causing each call taking up to 2 1/2 minutes before Kibana sees the error (now down to 30s each). We have polling to manage retries in these situations.
• Switch the task poller tests to use sinon for faking timers
• Removing the assertStillInSetup checks on plugin setup. Felt like a maintenance burden that wasn't necessary to fix with my code changes.

The main code changes are within these files (to review thoroughly so the polling cycle doesn't suddenly stop):

• x-pack/plugins/task_manager/server/polling/task_poller.ts
• x-pack/plugins/task_manager/server/polling_lifecycle.ts (easier to review if you disregard whitespace ?w=1)

To verify

1. Tasks run normally (create a rule or something that goes through task manager regularly).
2. When the update by query takes a while, the request is cancelled after 30s or the time manually configured.
3. When the search for claimed tasks query takes a while, the request is cancelled after 30s or the time manually configured.

Tips:

how to slowdown search for claimed task queries

diff --git a/x-pack/plugins/task_manager/server/queries/task_claiming.ts b/x-pack/plugins/task_manager/server/queries/task_claiming.ts
index 07042650a37..2caefd63672 100644
--- a/x-pack/plugins/task_manager/server/queries/task_claiming.ts
+++ b/x-pack/plugins/task_manager/server/queries/task_claiming.ts
@@ -247,7 +247,7 @@ export class TaskClaiming {
         taskTypes,
       });

-    const docs = tasksUpdated > 0 ? await this.sweepForClaimedTasks(taskTypes, size) : [];
+    const docs = await this.sweepForClaimedTasks(taskTypes, size);

     this.emitEvents(docs.map((doc) => asTaskClaimEvent(doc.id, asOk(doc))));

@@ -346,6 +346,13 @@ export class TaskClaiming {
       size,
       sort: SortByRunAtAndRetryAt,
       seq_no_primary_term: true,
+      aggs: {
+        delay: {
+          shard_delay: {
+            value: '40s',
+          },
+        },
+      },
     });

     return docs;

how to slow down update by queries

Not the cleanest way but you'll see occasional request timeouts from the updateByQuery calls. I had more luck creating rules running every 1s.

diff --git a/x-pack/plugins/task_manager/server/task_store.ts b/x-pack/plugins/task_manager/server/task_store.ts
index a06ee7b918a..07aa81e5388 100644
--- a/x-pack/plugins/task_manager/server/task_store.ts
+++ b/x-pack/plugins/task_manager/server/task_store.ts
@@ -126,6 +126,7 @@ export class TaskStore {
       // Timeouts are retried and make requests timeout after (requestTimeout * (1 + maxRetries))
       // The poller doesn't need retry logic because it will try again at the next polling cycle
       maxRetries: 0,
+      requestTimeout: 900,
     });
   }

@@ -458,6 +459,7 @@ export class TaskStore {
           ignore_unavailable: true,
           refresh: true,
           conflicts: 'proceed',
+          requests_per_second: 1,
           body: {
             ...opts,
             max_docs,

[mikecote]

@elasticmachine merge upstream

[mikecote]

@elasticmachine merge upstream

[mikecote]

@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[jbudz]

Does this need to go through the deprecation / breaking change process?

[mikecote]

The setting isn't documented so I figured we could simply remove it. I could change it to a no-op so it doesn't become breaking for anyone. Thoughts?

[mikecote]

@jbudz ^^

[jbudz]

I'll leave it up to the team - not familiar with the setting and whether its use has been advised in any production scenarios.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 76d838b

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
securitySolution	398	401	+3
total			+5

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
securitySolution	478	481	+3
total			+5

History

• 💚 Build #124770 succeeded 4e3a312
• 💔 Build #124540 failed 7d18e7b
• 💔 Build #124373 failed fcd6a48
• 💔 Build #124287 failed 2c07e95
• 💔 Build #124276 failed 2f09391
• 💔 Build #124264 failed 7a79043

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @mikecote

[ymao1]

LGTM! Created a bunch of rules and saw them running as expected. Verified the ES timeouts as per PR instructions as well.