Merge branch 'feature/ingest-node-pipelines' of github.com:elastic/ki…

…bana into pipeline-editor-part-mvp-2

* 'feature/ingest-node-pipelines' of github.com:elastic/kibana: (152 commits)
  [Ingest pipelines] Simulate pipeline (elastic#64223)
  Ability to get scoped call cluster from alerting and action executors (elastic#64432)
  Add editApp and editPath to embeddable (elastic#64297)
  TSVB validation: Allow numeric values for axes (elastic#63553)
  [ML] Fixing optional plugin dependency types (elastic#64450)
  [Logs UI] Alerting (elastic#62806)
  [Endpoint] Show Policy Status on Host Details using Policy Response API (elastic#64116)
  [Maps] update LayerWizard previewLayer to take layerDescriptor instead of ISource (elastic#64461)
  Remove SO root property index signature (elastic#64434)
  [ML] Functional tests - stabilize job row details validations (elastic#64503)
  [Ingest] Add Global settings flyout (elastic#64276)
  Bump cypress dev-dependency from 4.2.0 to 4.4.1 (elastic#64408)
  Migrate saved object of type url to kibana platform (elastic#64043)
  [NP] Migrate ui capabilities (elastic#64185)
  Bump karma-mocha dev-dependency from 1.3.0 to 2.0.0 (elastic#64407)
  Migrate kql_telemetry saved object registration to Kibana platform (elastic#64149)
  Remove SO autocreateindex error and error page (elastic#64037)
  Fix issue with yarn.lock (elastic#64496)
  Bump @hapi/boom dependency from 7.4.2 to 7.4.11 (elastic#64433)
  Bump gonzales-pe dev-dependency from 4.2.4 to 4.3.0 (elastic#64401)
  ...

# Conflicts:
#	x-pack/plugins/ingest_pipelines/public/application/components/pipeline_form/pipeline_form.tsx
#	x-pack/plugins/ingest_pipelines/public/shared_imports.ts

.eslintrc.js

@@ -96,7 +96,7 @@ module.exports = {
       },
     },
     {
-      files: ['x-pack/legacy/plugins/cross_cluster_replication/**/*.{js,ts,tsx}'],
+      files: ['x-pack/plugins/cross_cluster_replication/**/*.{js,ts,tsx}'],
       rules: {
         'jsx-a11y/click-events-have-key-events': 'off',
       },
@@ -185,31 +185,40 @@ module.exports = {
             zones: [
               {
                 target: [
-                  'src/legacy/**/*',
-                  'x-pack/**/*',
-                  '!x-pack/**/*.test.*',
-                  '!x-pack/test/**/*',
+                  '(src|x-pack)/legacy/**/*',
                   '(src|x-pack)/plugins/**/(public|server)/**/*',
-                  'src/core/(public|server)/**/*',
                   'examples/**/*',
                 ],
                 from: [
                   'src/core/public/**/*',
-                  '!src/core/public/index.ts',
-                  '!src/core/public/mocks.ts',
-                  '!src/core/public/*.test.mocks.ts',
+                  '!src/core/public/index.ts', // relative import
+                  '!src/core/public/mocks{,.ts}',
+                  '!src/core/server/types{,.ts}',
                   '!src/core/public/utils/**/*',
+                  '!src/core/public/*.test.mocks{,.ts}',
 
                   'src/core/server/**/*',
-                  '!src/core/server/index.ts',
-                  '!src/core/server/mocks.ts',
-                  '!src/core/server/types.ts',
-                  '!src/core/server/test_utils.ts',
+                  '!src/core/server/index.ts', // relative import
+                  '!src/core/server/mocks{,.ts}',
+                  '!src/core/server/types{,.ts}',
+                  '!src/core/server/test_utils',
                   // for absolute imports until fixed in
                   // https://github.com/elastic/kibana/issues/36096
-                  '!src/core/server/types',
-                  '!src/core/server/*.test.mocks.ts',
-
+                  '!src/core/server/*.test.mocks{,.ts}',
+                ],
+                allowSameFolder: true,
+                errorMessage:
+                  'Plugins may only import from top-level public and server modules in core.',
+              },
+              {
+                target: [
+                  '(src|x-pack)/legacy/**/*',
+                  '(src|x-pack)/plugins/**/(public|server)/**/*',
+                  'examples/**/*',
+                  '!(src|x-pack)/**/*.test.*',
+                  '!(x-pack/)?test/**/*',
+                ],
+                from: [
                   '(src|x-pack)/plugins/**/(public|server)/**/*',
                   '!(src|x-pack)/plugins/**/(public|server)/(index|mocks).{js,ts,tsx}',
                 ],
@@ -730,6 +739,19 @@ module.exports = {
       },
     },
 
+    /**
+     * Alerting Services overrides
+     */
+    {
+      // typescript only for front and back end
+      files: [
+        'x-pack/{,legacy/}plugins/{alerting,alerting_builtins,actions,task_manager,event_log}/**/*.{ts,tsx}',
+      ],
+      rules: {
+        '@typescript-eslint/no-explicit-any': 'error',
+      },
+    },
+
     /**
      * Lens overrides
      */

.github/CODEOWNERS

@@ -18,6 +18,7 @@
 /src/plugins/vis_type_timelion/ @elastic/kibana-app
 /src/plugins/dashboard/ @elastic/kibana-app
 /src/plugins/discover/ @elastic/kibana-app
+/src/plugins/input_control_vis/ @elastic/kibana-app
 /src/plugins/visualize/ @elastic/kibana-app
 /src/plugins/vis_type_timeseries/ @elastic/kibana-app
 /src/plugins/vis_type_metric/ @elastic/kibana-app

.github/paths-labeller.yml

@@ -8,6 +8,9 @@
   - "Feature:ExpressionLanguage":
     - "src/plugins/expressions/**/*.*"
     - "src/plugins/bfetch/**/*.*"
+  - "Team:apm"
+    - "x-pack/plugins/apm/**/*.*"
+    - "x-pack/legacy/plugins/apm/**/*.*"
   - "Team:uptime":
     - "x-pack/plugins/uptime/**/*.*"
     - "x-pack/legacy/plugins/uptime/**/*.*"

.i18nrc.json

@@ -14,7 +14,7 @@
     "esUi": "src/plugins/es_ui_shared",
     "devTools": "src/plugins/dev_tools",
     "expressions": "src/plugins/expressions",
-    "inputControl": "src/legacy/core_plugins/input_control_vis",
+    "inputControl": "src/plugins/input_control_vis",
     "inspector": "src/plugins/inspector",
     "inspectorViews": "src/legacy/core_plugins/inspector_views",
     "interpreter": "src/legacy/core_plugins/interpreter",
@@ -49,9 +49,9 @@
     "visTypeMarkdown": "src/plugins/vis_type_markdown",
     "visTypeMetric": "src/plugins/vis_type_metric",
     "visTypeTable": "src/plugins/vis_type_table",
-    "visTypeTagCloud": "src/legacy/core_plugins/vis_type_tagcloud",
+    "visTypeTagCloud": "src/plugins/vis_type_tagcloud",
     "visTypeTimeseries": ["src/legacy/core_plugins/vis_type_timeseries", "src/plugins/vis_type_timeseries"],
-    "visTypeVega": "src/legacy/core_plugins/vis_type_vega",
+    "visTypeVega": "src/plugins/vis_type_vega",
     "visTypeVislib": "src/legacy/core_plugins/vis_type_vislib",
     "visTypeXy": "src/legacy/core_plugins/vis_type_xy",
     "visualizations": "src/plugins/visualizations",

.sass-lint.yml

@@ -4,11 +4,11 @@ files:
     - 'src/legacy/core_plugins/timelion/**/*.s+(a|c)ss'
     - 'src/legacy/core_plugins/vis_type_vislib/**/*.s+(a|c)ss'
     - 'src/legacy/core_plugins/vis_type_xy/**/*.s+(a|c)ss'
-    - 'x-pack/legacy/plugins/rollup/**/*.s+(a|c)ss'
     - 'x-pack/legacy/plugins/security/**/*.s+(a|c)ss'
     - 'x-pack/legacy/plugins/canvas/**/*.s+(a|c)ss'
     - 'x-pack/plugins/triggers_actions_ui/**/*.s+(a|c)ss'
     - 'x-pack/plugins/lens/**/*.s+(a|c)ss'
+    - 'x-pack/plugins/cross_cluster_replication/**/*.s+(a|c)ss'
     - 'x-pack/legacy/plugins/maps/**/*.s+(a|c)ss'
     - 'x-pack/plugins/maps/**/*.s+(a|c)ss'
   ignore:

Jenkinsfile

@@ -11,7 +11,7 @@ kibanaPipeline(timeoutMinutes: 135, checkPrChanges: true) {
         'kibana-intake-agent': workers.intake('kibana-intake', './test/scripts/jenkins_unit.sh'),
         'x-pack-intake-agent': workers.intake('x-pack-intake', './test/scripts/jenkins_xpack.sh'),
         'kibana-oss-agent': workers.functional('kibana-oss-tests', { kibanaPipeline.buildOss() }, [
-          // 'oss-firefoxSmoke': kibanaPipeline.functionalTestProcess('kibana-firefoxSmoke', './test/scripts/jenkins_firefox_smoke.sh'),
+          'oss-firefoxSmoke': kibanaPipeline.functionalTestProcess('kibana-firefoxSmoke', './test/scripts/jenkins_firefox_smoke.sh'),
           'oss-ciGroup1': kibanaPipeline.ossCiGroupProcess(1),
           'oss-ciGroup2': kibanaPipeline.ossCiGroupProcess(2),
           'oss-ciGroup3': kibanaPipeline.ossCiGroupProcess(3),
@@ -28,7 +28,7 @@ kibanaPipeline(timeoutMinutes: 135, checkPrChanges: true) {
           // 'oss-visualRegression': kibanaPipeline.functionalTestProcess('visualRegression', './test/scripts/jenkins_visual_regression.sh'),
         ]),
         'kibana-xpack-agent': workers.functional('kibana-xpack-tests', { kibanaPipeline.buildXpack() }, [
-          // 'xpack-firefoxSmoke': kibanaPipeline.functionalTestProcess('xpack-firefoxSmoke', './test/scripts/jenkins_xpack_firefox_smoke.sh'),
+          'xpack-firefoxSmoke': kibanaPipeline.functionalTestProcess('xpack-firefoxSmoke', './test/scripts/jenkins_xpack_firefox_smoke.sh'),
           'xpack-ciGroup1': kibanaPipeline.xpackCiGroupProcess(1),
           'xpack-ciGroup2': kibanaPipeline.xpackCiGroupProcess(2),
           'xpack-ciGroup3': kibanaPipeline.xpackCiGroupProcess(3),

docs/developer/core/development-functional-tests.asciidoc

@@ -154,16 +154,16 @@ A test suite is a collection of tests defined by calling `describe()`, and then
 Use tags in `describe()` function to group functional tests. Tags include:
 * `ciGroup{id}` - Assigns test suite to a specific CI worker
 * `skipCloud` and `skipFirefox` - Excludes test suite from running on Cloud or Firefox
-* `smoke` - Groups tests that run on Chrome and Firefox
+* `includeFirefox` - Groups tests that run on Chrome and Firefox
 
 **Cross-browser testing**:::
-On CI, all the functional tests are executed in Chrome by default. To also run a suite against Firefox, assign the `smoke` tag:
+On CI, all the functional tests are executed in Chrome by default. To also run a suite against Firefox, assign the `includeFirefox` tag:
 
 ["source","js"]
 -----------
 // on CI test suite will be run twice: in Chrome and Firefox
 describe('My Cross-browser Test Suite', function () {
-  this.tags('smoke');
+  this.tags('includeFirefox');
 
   it('My First Test');
 }

docs/development/core/server/kibana-plugin-core-server.md

@@ -263,13 +263,14 @@ The plugin integrates with the core system via lifecycle events: `setup`<!-- -->
 |  [SavedObjectAttribute](./kibana-plugin-core-server.savedobjectattribute.md) | Type definition for a Saved Object attribute value |
 |  [SavedObjectAttributeSingle](./kibana-plugin-core-server.savedobjectattributesingle.md) | Don't use this type, it's simply a helper type for [SavedObjectAttribute](./kibana-plugin-core-server.savedobjectattribute.md) |
 |  [SavedObjectMigrationFn](./kibana-plugin-core-server.savedobjectmigrationfn.md) | A migration function for a [saved object type](./kibana-plugin-core-server.savedobjectstype.md) used to migrate it to a given version |
-|  [SavedObjectSanitizedDoc](./kibana-plugin-core-server.savedobjectsanitizeddoc.md) |  |
-|  [SavedObjectsClientContract](./kibana-plugin-core-server.savedobjectsclientcontract.md) | Saved Objects is Kibana's data persisentence mechanism allowing plugins to use Elasticsearch for storing plugin state.<!-- -->\#\# SavedObjectsClient errors<!-- -->Since the SavedObjectsClient has its hands in everything we are a little paranoid about the way we present errors back to to application code. Ideally, all errors will be either:<!-- -->1. Caused by bad implementation (ie. undefined is not a function) and as such unpredictable 2. An error that has been classified and decorated appropriately by the decorators in [SavedObjectsErrorHelpers](./kibana-plugin-core-server.savedobjectserrorhelpers.md)<!-- -->Type 1 errors are inevitable, but since all expected/handle-able errors should be Type 2 the <code>isXYZError()</code> helpers exposed at <code>SavedObjectsErrorHelpers</code> should be used to understand and manage error responses from the <code>SavedObjectsClient</code>.<!-- -->Type 2 errors are decorated versions of the source error, so if the elasticsearch client threw an error it will be decorated based on its type. That means that rather than looking for <code>error.body.error.type</code> or doing substring checks on <code>error.body.error.reason</code>, just use the helpers to understand the meaning of the error:<!-- -->\`\`\`<!-- -->js if (SavedObjectsErrorHelpers.isNotFoundError(error)) { // handle 404 }<!-- -->if (SavedObjectsErrorHelpers.isNotAuthorizedError(error)) { // 401 handling should be automatic, but in case you wanted to know }<!-- -->// always rethrow the error unless you handle it throw error; \`\`\`<!-- -->\#\#\# 404s from missing index<!-- -->From the perspective of application code and APIs the SavedObjectsClient is a black box that persists objects. One of the internal details that users have no control over is that we use an elasticsearch index for persistance and that index might be missing.<!-- -->At the time of writing we are in the process of transitioning away from the operating assumption that the SavedObjects index is always available. Part of this transition is handling errors resulting from an index missing. These used to trigger a 500 error in most cases, and in others cause 404s with different error messages.<!-- -->From my (Spencer) perspective, a 404 from the SavedObjectsApi is a 404; The object the request/call was targeting could not be found. This is why \#14141 takes special care to ensure that 404 errors are generic and don't distinguish between index missing or document missing.<!-- -->\#\#\# 503s from missing index<!-- -->Unlike all other methods, create requests are supposed to succeed even when the Kibana index does not exist because it will be automatically created by elasticsearch. When that is not the case it is because Elasticsearch's <code>action.auto_create_index</code> setting prevents it from being created automatically so we throw a special 503 with the intention of informing the user that their Elasticsearch settings need to be updated.<!-- -->See [SavedObjectsClient](./kibana-plugin-core-server.savedobjectsclient.md) See [SavedObjectsErrorHelpers](./kibana-plugin-core-server.savedobjectserrorhelpers.md) |
+|  [SavedObjectSanitizedDoc](./kibana-plugin-core-server.savedobjectsanitizeddoc.md) | Describes Saved Object documents that have passed through the migration framework and are guaranteed to have a <code>references</code> root property. |
+|  [SavedObjectsClientContract](./kibana-plugin-core-server.savedobjectsclientcontract.md) | Saved Objects is Kibana's data persisentence mechanism allowing plugins to use Elasticsearch for storing plugin state.<!-- -->\#\# SavedObjectsClient errors<!-- -->Since the SavedObjectsClient has its hands in everything we are a little paranoid about the way we present errors back to to application code. Ideally, all errors will be either:<!-- -->1. Caused by bad implementation (ie. undefined is not a function) and as such unpredictable 2. An error that has been classified and decorated appropriately by the decorators in [SavedObjectsErrorHelpers](./kibana-plugin-core-server.savedobjectserrorhelpers.md)<!-- -->Type 1 errors are inevitable, but since all expected/handle-able errors should be Type 2 the <code>isXYZError()</code> helpers exposed at <code>SavedObjectsErrorHelpers</code> should be used to understand and manage error responses from the <code>SavedObjectsClient</code>.<!-- -->Type 2 errors are decorated versions of the source error, so if the elasticsearch client threw an error it will be decorated based on its type. That means that rather than looking for <code>error.body.error.type</code> or doing substring checks on <code>error.body.error.reason</code>, just use the helpers to understand the meaning of the error:<!-- -->\`\`\`<!-- -->js if (SavedObjectsErrorHelpers.isNotFoundError(error)) { // handle 404 }<!-- -->if (SavedObjectsErrorHelpers.isNotAuthorizedError(error)) { // 401 handling should be automatic, but in case you wanted to know }<!-- -->// always rethrow the error unless you handle it throw error; \`\`\`<!-- -->\#\#\# 404s from missing index<!-- -->From the perspective of application code and APIs the SavedObjectsClient is a black box that persists objects. One of the internal details that users have no control over is that we use an elasticsearch index for persistance and that index might be missing.<!-- -->At the time of writing we are in the process of transitioning away from the operating assumption that the SavedObjects index is always available. Part of this transition is handling errors resulting from an index missing. These used to trigger a 500 error in most cases, and in others cause 404s with different error messages.<!-- -->From my (Spencer) perspective, a 404 from the SavedObjectsApi is a 404; The object the request/call was targeting could not be found. This is why \#14141 takes special care to ensure that 404 errors are generic and don't distinguish between index missing or document missing.<!-- -->See [SavedObjectsClient](./kibana-plugin-core-server.savedobjectsclient.md) See [SavedObjectsErrorHelpers](./kibana-plugin-core-server.savedobjectserrorhelpers.md) |
 |  [SavedObjectsClientFactory](./kibana-plugin-core-server.savedobjectsclientfactory.md) | Describes the factory used to create instances of the Saved Objects Client. |
 |  [SavedObjectsClientFactoryProvider](./kibana-plugin-core-server.savedobjectsclientfactoryprovider.md) | Provider to invoke to retrieve a [SavedObjectsClientFactory](./kibana-plugin-core-server.savedobjectsclientfactory.md)<!-- -->. |
 |  [SavedObjectsClientWrapperFactory](./kibana-plugin-core-server.savedobjectsclientwrapperfactory.md) | Describes the factory used to create instances of Saved Objects Client Wrappers. |
 |  [SavedObjectsFieldMapping](./kibana-plugin-core-server.savedobjectsfieldmapping.md) | Describe a [saved object type mapping](./kibana-plugin-core-server.savedobjectstypemappingdefinition.md) field.<!-- -->Please refer to [elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html) For the mapping documentation |
 |  [SavedObjectsNamespaceType](./kibana-plugin-core-server.savedobjectsnamespacetype.md) | The namespace type dictates how a saved object can be interacted in relation to namespaces. Each type is mutually exclusive: \* single (default): this type of saved object is namespace-isolated, e.g., it exists in only one namespace. \* multiple: this type of saved object is shareable, e.g., it can exist in one or more namespaces. \* agnostic: this type of saved object is global.<!-- -->Note: do not write logic that uses this value directly; instead, use the appropriate accessors in the [type registry](./kibana-plugin-core-server.savedobjecttyperegistry.md)<!-- -->. |
+|  [SavedObjectUnsanitizedDoc](./kibana-plugin-core-server.savedobjectunsanitizeddoc.md) | Describes Saved Object documents from Kibana &lt; 7.0.0 which don't have a <code>references</code> root property defined. This type should only be used in migrations. |
 |  [ScopeableRequest](./kibana-plugin-core-server.scopeablerequest.md) | A user credentials container. It accommodates the necessary auth credentials to impersonate the current user.<!-- -->See [KibanaRequest](./kibana-plugin-core-server.kibanarequest.md)<!-- -->. |
 |  [ServiceStatusLevel](./kibana-plugin-core-server.servicestatuslevel.md) | A convenience type that represents the union of each value in [ServiceStatusLevels](./kibana-plugin-core-server.servicestatuslevels.md)<!-- -->. |
 |  [SharedGlobalConfig](./kibana-plugin-core-server.sharedglobalconfig.md) |  |

docs/development/core/server/kibana-plugin-core-server.savedobjectsanitizeddoc.md

@@ -4,6 +4,7 @@
 
 ## SavedObjectSanitizedDoc type
 
+Describes Saved Object documents that have passed through the migration framework and are guaranteed to have a `references` root property.
 
 <b>Signature:</b>
 

docs/development/core/server/kibana-plugin-core-server.savedobjectsclientcontract.md

@@ -30,10 +30,6 @@ At the time of writing we are in the process of transitioning away from the oper
 
 From my (Spencer) perspective, a 404 from the SavedObjectsApi is a 404; The object the request/call was targeting could not be found. This is why \#14141 takes special care to ensure that 404 errors are generic and don't distinguish between index missing or document missing.
 
-\#\#\# 503s from missing index
-
-Unlike all other methods, create requests are supposed to succeed even when the Kibana index does not exist because it will be automatically created by elasticsearch. When that is not the case it is because Elasticsearch's `action.auto_create_index` setting prevents it from being created automatically so we throw a special 503 with the intention of informing the user that their Elasticsearch settings need to be updated.
-
 See [SavedObjectsClient](./kibana-plugin-core-server.savedobjectsclient.md) See [SavedObjectsErrorHelpers](./kibana-plugin-core-server.savedobjectserrorhelpers.md)
 
 <b>Signature:</b>