Task manager enhancements for error handling in alerting and actions (#…

…39829) (#42004)

* Allow mtask definitions to overwrite default setting maxAttemps

* Leverage scheduledAt from task manager

* Treat maxAttempts like attempts and not retries

* Add support for second intervals

* Min 1 attempt

* Reverse relying on scheduledAt

* Add new startedAt attribute in task manager that keeps track when task started running

* Don't extend runAt when claiming a task

* Remove startedAt from state

* Attempt trying to define custom getBackpressureDelay function

* Pass error object to getBackpressureDelay

* Cleanup processResultForRecurringTask code

* Add backpressure to timed out tasks

* Change default timeout backpressure calculation

* getBackpressureDelay to return seconds instead of milliseconds

* Add comment for task store query

* Compress query

* Revert alert / actions specific code

* Add more interval tests

* Fix failing jest tests

* Fix test

* Add more unit tests

* Fix integration tests

* Fix sorting of tasks to process

* WIP

* Always provide error when getBackpressureDelay is called

* Rename getBackpressureDelay to getRetryDelay

* retryAt to be calculated from timeout time by default

* Remove invalid test

* Add unit tests

* Consider timeout before scheduling a retryAt

* Remove backpressure terminology

* Remove support for 0 based intervals and timeouts

* Apply PR feedback

* Fix last place using Math.abs

* Modify migrations to allow running a script when converting an index to an alias

* Convert task manager to use saved objects

* Fix broken test

* Fix broken tests pt1

* Remove index from task manager config schema

* Accept platform changes

* PR feedback

* Apply PR feedback

* Apply PR feedback pt2

* Apply PR feedback pt3

* Apply PR feedback pt4

* Fix feedback pt3

* Rename RawSavedObjectDoc to SavedObjectsRawDoc

docs/development/core/server/kibana-plugin-server.md

@@ -19,6 +19,8 @@ The plugin integrates with the core system via lifecycle events: `setup`<!-- -->
 |  [KibanaRequest](./kibana-plugin-server.kibanarequest.md) | Kibana specific abstraction for an incoming request. |
 |  [Router](./kibana-plugin-server.router.md) |  |
 |  [SavedObjectsErrorHelpers](./kibana-plugin-server.savedobjectserrorhelpers.md) |  |
+|  [SavedObjectsSchema](./kibana-plugin-server.savedobjectsschema.md) |  |
+|  [SavedObjectsSerializer](./kibana-plugin-server.savedobjectsserializer.md) |  |
 |  [ScopedClusterClient](./kibana-plugin-server.scopedclusterclient.md) | Serves the same purpose as "normal" <code>ClusterClient</code> but exposes additional <code>callAsCurrentUser</code> method that doesn't use credentials of the Kibana internal user (as <code>callAsInternalUser</code> does) to request Elasticsearch API, but rather passes HTTP headers extracted from the current user request to the API |
 
 ## Interfaces
@@ -60,6 +62,7 @@ The plugin integrates with the core system via lifecycle events: `setup`<!-- -->
 |  [SavedObjectsFindOptions](./kibana-plugin-server.savedobjectsfindoptions.md) |  |
 |  [SavedObjectsFindResponse](./kibana-plugin-server.savedobjectsfindresponse.md) |  |
 |  [SavedObjectsMigrationVersion](./kibana-plugin-server.savedobjectsmigrationversion.md) | A dictionary of saved object type -<!-- -->&gt; version used to determine what migrations need to be applied to a saved object. |
+|  [SavedObjectsRawDoc](./kibana-plugin-server.savedobjectsrawdoc.md) | A raw document as represented directly in the saved object index. |
 |  [SavedObjectsService](./kibana-plugin-server.savedobjectsservice.md) |  |
 |  [SavedObjectsUpdateOptions](./kibana-plugin-server.savedobjectsupdateoptions.md) |  |
 |  [SavedObjectsUpdateResponse](./kibana-plugin-server.savedobjectsupdateresponse.md) |  |

docs/development/core/server/kibana-plugin-server.savedobjectsrawdoc._id.md

@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsRawDoc](./kibana-plugin-server.savedobjectsrawdoc.md) &gt; [\_id](./kibana-plugin-server.savedobjectsrawdoc._id.md)
+
+## SavedObjectsRawDoc.\_id property
+
+<b>Signature:</b>
+
+```typescript
+_id: string;
+```

docs/development/core/server/kibana-plugin-server.savedobjectsrawdoc._primary_term.md

@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsRawDoc](./kibana-plugin-server.savedobjectsrawdoc.md) &gt; [\_primary\_term](./kibana-plugin-server.savedobjectsrawdoc._primary_term.md)
+
+## SavedObjectsRawDoc.\_primary\_term property
+
+<b>Signature:</b>
+
+```typescript
+_primary_term?: number;
+```

docs/development/core/server/kibana-plugin-server.savedobjectsrawdoc._seq_no.md

@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsRawDoc](./kibana-plugin-server.savedobjectsrawdoc.md) &gt; [\_seq\_no](./kibana-plugin-server.savedobjectsrawdoc._seq_no.md)
+
+## SavedObjectsRawDoc.\_seq\_no property
+
+<b>Signature:</b>
+
+```typescript
+_seq_no?: number;
+```

docs/development/core/server/kibana-plugin-server.savedobjectsrawdoc._source.md

@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsRawDoc](./kibana-plugin-server.savedobjectsrawdoc.md) &gt; [\_source](./kibana-plugin-server.savedobjectsrawdoc._source.md)
+
+## SavedObjectsRawDoc.\_source property
+
+<b>Signature:</b>
+
+```typescript
+_source: any;
+```

docs/development/core/server/kibana-plugin-server.savedobjectsrawdoc._type.md

@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsRawDoc](./kibana-plugin-server.savedobjectsrawdoc.md) &gt; [\_type](./kibana-plugin-server.savedobjectsrawdoc._type.md)
+
+## SavedObjectsRawDoc.\_type property
+
+<b>Signature:</b>
+
+```typescript
+_type?: string;
+```

docs/development/core/server/kibana-plugin-server.savedobjectsrawdoc.md

@@ -0,0 +1,24 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsRawDoc](./kibana-plugin-server.savedobjectsrawdoc.md)
+
+## SavedObjectsRawDoc interface
+
+A raw document as represented directly in the saved object index.
+
+<b>Signature:</b>
+
+```typescript
+export interface RawDoc 
+```
+
+## Properties
+
+|  Property | Type | Description |
+|  --- | --- | --- |
+|  [\_id](./kibana-plugin-server.savedobjectsrawdoc._id.md) | <code>string</code> |  |
+|  [\_primary\_term](./kibana-plugin-server.savedobjectsrawdoc._primary_term.md) | <code>number</code> |  |
+|  [\_seq\_no](./kibana-plugin-server.savedobjectsrawdoc._seq_no.md) | <code>number</code> |  |
+|  [\_source](./kibana-plugin-server.savedobjectsrawdoc._source.md) | <code>any</code> |  |
+|  [\_type](./kibana-plugin-server.savedobjectsrawdoc._type.md) | <code>string</code> |  |
+

docs/development/core/server/kibana-plugin-server.savedobjectsschema.(constructor).md

@@ -0,0 +1,20 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsSchema](./kibana-plugin-server.savedobjectsschema.md) &gt; [(constructor)](./kibana-plugin-server.savedobjectsschema.(constructor).md)
+
+## SavedObjectsSchema.(constructor)
+
+Constructs a new instance of the `SavedObjectsSchema` class
+
+<b>Signature:</b>
+
+```typescript
+constructor(schemaDefinition?: SavedObjectsSchemaDefinition);
+```
+
+## Parameters
+
+|  Parameter | Type | Description |
+|  --- | --- | --- |
+|  schemaDefinition | <code>SavedObjectsSchemaDefinition</code> |  |
+

docs/development/core/server/kibana-plugin-server.savedobjectsschema.getindexfortype.md

@@ -0,0 +1,22 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsSchema](./kibana-plugin-server.savedobjectsschema.md) &gt; [getIndexForType](./kibana-plugin-server.savedobjectsschema.getindexfortype.md)
+
+## SavedObjectsSchema.getIndexForType() method
+
+<b>Signature:</b>
+
+```typescript
+getIndexForType(type: string): string | undefined;
+```
+
+## Parameters
+
+|  Parameter | Type | Description |
+|  --- | --- | --- |
+|  type | <code>string</code> |  |
+
+<b>Returns:</b>
+
+`string | undefined`
+

docs/development/core/server/kibana-plugin-server.savedobjectsschema.ishiddentype.md

@@ -0,0 +1,22 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsSchema](./kibana-plugin-server.savedobjectsschema.md) &gt; [isHiddenType](./kibana-plugin-server.savedobjectsschema.ishiddentype.md)
+
+## SavedObjectsSchema.isHiddenType() method
+
+<b>Signature:</b>
+
+```typescript
+isHiddenType(type: string): boolean;
+```
+
+## Parameters
+
+|  Parameter | Type | Description |
+|  --- | --- | --- |
+|  type | <code>string</code> |  |
+
+<b>Returns:</b>
+
+`boolean`
+

docs/development/core/server/kibana-plugin-server.savedobjectsschema.isnamespaceagnostic.md

@@ -0,0 +1,22 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsSchema](./kibana-plugin-server.savedobjectsschema.md) &gt; [isNamespaceAgnostic](./kibana-plugin-server.savedobjectsschema.isnamespaceagnostic.md)
+
+## SavedObjectsSchema.isNamespaceAgnostic() method
+
+<b>Signature:</b>
+
+```typescript
+isNamespaceAgnostic(type: string): boolean;
+```
+
+## Parameters
+
+|  Parameter | Type | Description |
+|  --- | --- | --- |
+|  type | <code>string</code> |  |
+
+<b>Returns:</b>
+
+`boolean`
+

docs/development/core/server/kibana-plugin-server.savedobjectsschema.md

@@ -0,0 +1,26 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsSchema](./kibana-plugin-server.savedobjectsschema.md)
+
+## SavedObjectsSchema class
+
+<b>Signature:</b>
+
+```typescript
+export declare class SavedObjectsSchema 
+```
+
+## Constructors
+
+|  Constructor | Modifiers | Description |
+|  --- | --- | --- |
+|  [(constructor)(schemaDefinition)](./kibana-plugin-server.savedobjectsschema.(constructor).md) |  | Constructs a new instance of the <code>SavedObjectsSchema</code> class |
+
+## Methods
+
+|  Method | Modifiers | Description |
+|  --- | --- | --- |
+|  [getIndexForType(type)](./kibana-plugin-server.savedobjectsschema.getindexfortype.md) |  |  |
+|  [isHiddenType(type)](./kibana-plugin-server.savedobjectsschema.ishiddentype.md) |  |  |
+|  [isNamespaceAgnostic(type)](./kibana-plugin-server.savedobjectsschema.isnamespaceagnostic.md) |  |  |
+

docs/development/core/server/kibana-plugin-server.savedobjectsserializer.(constructor).md

@@ -0,0 +1,20 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsSerializer](./kibana-plugin-server.savedobjectsserializer.md) &gt; [(constructor)](./kibana-plugin-server.savedobjectsserializer.(constructor).md)
+
+## SavedObjectsSerializer.(constructor)
+
+Constructs a new instance of the `SavedObjectsSerializer` class
+
+<b>Signature:</b>
+
+```typescript
+constructor(schema: SavedObjectsSchema);
+```
+
+## Parameters
+
+|  Parameter | Type | Description |
+|  --- | --- | --- |
+|  schema | <code>SavedObjectsSchema</code> |  |
+

docs/development/core/server/kibana-plugin-server.savedobjectsserializer.generaterawid.md

@@ -0,0 +1,26 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsSerializer](./kibana-plugin-server.savedobjectsserializer.md) &gt; [generateRawId](./kibana-plugin-server.savedobjectsserializer.generaterawid.md)
+
+## SavedObjectsSerializer.generateRawId() method
+
+Given a saved object type and id, generates the compound id that is stored in the raw document.
+
+<b>Signature:</b>
+
+```typescript
+generateRawId(namespace: string | undefined, type: string, id?: string): string;
+```
+
+## Parameters
+
+|  Parameter | Type | Description |
+|  --- | --- | --- |
+|  namespace | <code>string &#124; undefined</code> |  |
+|  type | <code>string</code> |  |
+|  id | <code>string</code> |  |
+
+<b>Returns:</b>
+
+`string`
+

docs/development/core/server/kibana-plugin-server.savedobjectsserializer.israwsavedobject.md

@@ -0,0 +1,24 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsSerializer](./kibana-plugin-server.savedobjectsserializer.md) &gt; [isRawSavedObject](./kibana-plugin-server.savedobjectsserializer.israwsavedobject.md)
+
+## SavedObjectsSerializer.isRawSavedObject() method
+
+Determines whether or not the raw document can be converted to a saved object.
+
+<b>Signature:</b>
+
+```typescript
+isRawSavedObject(rawDoc: RawDoc): any;
+```
+
+## Parameters
+
+|  Parameter | Type | Description |
+|  --- | --- | --- |
+|  rawDoc | <code>RawDoc</code> |  |
+
+<b>Returns:</b>
+
+`any`
+

docs/development/core/server/kibana-plugin-server.savedobjectsserializer.md

@@ -0,0 +1,27 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsSerializer](./kibana-plugin-server.savedobjectsserializer.md)
+
+## SavedObjectsSerializer class
+
+<b>Signature:</b>
+
+```typescript
+export declare class SavedObjectsSerializer 
+```
+
+## Constructors
+
+|  Constructor | Modifiers | Description |
+|  --- | --- | --- |
+|  [(constructor)(schema)](./kibana-plugin-server.savedobjectsserializer.(constructor).md) |  | Constructs a new instance of the <code>SavedObjectsSerializer</code> class |
+
+## Methods
+
+|  Method | Modifiers | Description |
+|  --- | --- | --- |
+|  [generateRawId(namespace, type, id)](./kibana-plugin-server.savedobjectsserializer.generaterawid.md) |  | Given a saved object type and id, generates the compound id that is stored in the raw document. |
+|  [isRawSavedObject(rawDoc)](./kibana-plugin-server.savedobjectsserializer.israwsavedobject.md) |  | Determines whether or not the raw document can be converted to a saved object. |
+|  [rawToSavedObject(doc)](./kibana-plugin-server.savedobjectsserializer.rawtosavedobject.md) |  | Converts a document from the format that is stored in elasticsearch to the saved object client format. |
+|  [savedObjectToRaw(savedObj)](./kibana-plugin-server.savedobjectsserializer.savedobjecttoraw.md) |  | Converts a document from the saved object client format to the format that is stored in elasticsearch. |
+

docs/development/core/server/kibana-plugin-server.savedobjectsserializer.rawtosavedobject.md

@@ -0,0 +1,24 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsSerializer](./kibana-plugin-server.savedobjectsserializer.md) &gt; [rawToSavedObject](./kibana-plugin-server.savedobjectsserializer.rawtosavedobject.md)
+
+## SavedObjectsSerializer.rawToSavedObject() method
+
+Converts a document from the format that is stored in elasticsearch to the saved object client format.
+
+<b>Signature:</b>
+
+```typescript
+rawToSavedObject(doc: RawDoc): SanitizedSavedObjectDoc;
+```
+
+## Parameters
+
+|  Parameter | Type | Description |
+|  --- | --- | --- |
+|  doc | <code>RawDoc</code> |  |
+
+<b>Returns:</b>
+
+`SanitizedSavedObjectDoc`
+

docs/development/core/server/kibana-plugin-server.savedobjectsserializer.savedobjecttoraw.md

@@ -0,0 +1,24 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-server](./kibana-plugin-server.md) &gt; [SavedObjectsSerializer](./kibana-plugin-server.savedobjectsserializer.md) &gt; [savedObjectToRaw](./kibana-plugin-server.savedobjectsserializer.savedobjecttoraw.md)
+
+## SavedObjectsSerializer.savedObjectToRaw() method
+
+Converts a document from the saved object client format to the format that is stored in elasticsearch.
+
+<b>Signature:</b>
+
+```typescript
+savedObjectToRaw(savedObj: SanitizedSavedObjectDoc): RawDoc;
+```
+
+## Parameters
+
+|  Parameter | Type | Description |
+|  --- | --- | --- |
+|  savedObj | <code>SanitizedSavedObjectDoc</code> |  |
+
+<b>Returns:</b>
+
+`RawDoc`
+

src/core/server/index.ts

@@ -103,6 +103,9 @@ export {
   SavedObjectsFindOptions,
   SavedObjectsFindResponse,
   SavedObjectsMigrationVersion,
+  SavedObjectsRawDoc,
+  SavedObjectsSchema,
+  SavedObjectsSerializer,
   SavedObjectsService,
   SavedObjectsUpdateOptions,
   SavedObjectsUpdateResponse,

src/core/server/saved_objects/index.ts

@@ -22,3 +22,5 @@ export * from './service';
 export { SavedObjectsSchema } from './schema';
 
 export { SavedObjectsManagement } from './management';
+
+export { SavedObjectsSerializer, RawDoc as SavedObjectsRawDoc } from './serialization';