[Fleet] Add consistent `_meta` property to all Fleet ES assets #119380

kpollich · 2021-11-22T20:43:34Z

Summary

Adds/alters the _meta property on Elasticsearch assets generated by Fleet to consistently indicate that the assets come from Fleet, and when possible tie them back to an integration.

To Test

Run Fleet setup
Install any integrations
Verify the assets generated included a _meta field of the format

{
  managed: true,
  managed_by: "fleet",
  package: {
    name: "foo"
  }
}

In cases like the .fleet-final-pipeline ingest pipeline and others, the package value will be omitted.

To verify assets, use these Kibana dev tools queries:

GET /_index_template
GET /_component_template
GET /_ingest/pipeline
GET /_ilm/policy
GET /_transform/_all

Examples

Asset Type	Screenshot
Index template
Component template
Ingest pipeline
ILM policy
Transform

elasticmachine · 2021-11-22T20:43:39Z

Pinging @elastic/fleet (Team:Fleet)

kpollich · 2021-11-22T20:44:11Z

x-pack/plugins/fleet/server/constants/fleet_es_assets.ts

@@ -36,10 +40,14 @@ export const FLEET_GLOBAL_COMPONENT_TEMPLATE_CONTENT = {
  },
 };

-export const FLEET_FINAL_PIPELINE_VERSION = 1;
+export const FLEET_FINAL_PIPELINE_VERSION = 2;


@nchaulet - I wasn't 100% sure whether we should increment this pipeline version since we're adding _meta below.

I think incrementing the version will work it will trigger to update the pipeline to the latest version on fleet setup.

kpollich · 2021-11-22T20:45:18Z

x-pack/plugins/fleet/server/constants/fleet_es_assets.ts

 // If the content is updated you probably need to update the FLEET_FINAL_PIPELINE_VERSION too to allow upgrade of the pipeline
 export const FLEET_FINAL_PIPELINE_CONTENT = `---
 version: ${FLEET_FINAL_PIPELINE_VERSION}
+_meta:


Interpolating the full _meta object generated by getESAssetMetadata() proved difficult, so I elected to just interpolate the individual fields here. We don't need package for this global pipeline, but this may need to be revisited if we ever add more data to our _meta object.

kpollich · 2021-11-22T20:46:57Z

x-pack/plugins/fleet/server/services/epm/elasticsearch/meta.ts

+  if (pipeline.extension === 'yml') {
+    // Convert the YML content to JSON, append the `_meta` value, then convert it back to
+    // YML and return the resulting YML
+    const parsedPipelineContent = safeLoad(pipeline.contentForInstallation);
+    parsedPipelineContent._meta = meta;
+
+    return {
+      ...pipeline,
+      contentForInstallation: `---\n${safeDump(parsedPipelineContent)}`,
+    };
+  }


This proved to be a bit of a tricky use case here. I felt that our choices to insert the _meta field into a pipeline's yml data were either to do some kind of string replacement, or parse the yml -> add the value -> re-convert back into yml. I opted for the second choice here. Open to suggestions here certainly.

kpollich · 2021-11-22T20:47:27Z

x-pack/plugins/fleet/server/services/epm/elasticsearch/meta.ts

+
+    return {
+      ...pipeline,
+      contentForInstallation: `---\n${safeDump(parsedPipelineContent)}`,


Needed to re-add the yml document start marker --- in order for the ES client to be happy. safeDump seems to remove it.

juliaElastic · 2021-11-23T09:47:33Z

x-pack/plugins/fleet/server/services/epm/elasticsearch/transform/install.ts

      return {
        installationName: getTransformNameForInstallation(
          installablePackage,
          path,
          installNameSuffix
        ),
-        content: getAsset(path).toString('utf-8'),
+        content,


doesn't content have to be converted back to string?

The eventual putTransform call that uses this body actually expects an object, but I believe the ES client calls parse strings to JSON generally so that's why this was working before. Elsewhere is the same story - ES client calls accept either a string that will be parsed to JSON or a Record<string, any>.

juliaElastic

LGTM

kibana-ci · 2021-11-23T17:39:24Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💔 Build #8728 failed e48f0e7
💔 Build #8556 failed f8cbae0

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @kpollich

kibanamachine · 2021-11-23T18:17:54Z

💚 Backport successful

Status	Branch	Result
✅	8.0

This backport PR will be merged automatically after passing CI.

…ic#119380) * Add consistent _meta property to all Fleet ES assets * Fix meta logic for ILM + transforms * Fix tests + snapshots * Fix failing tests

…) (#119524) * Add consistent _meta property to all Fleet ES assets * Fix meta logic for ILM + transforms * Fix tests + snapshots * Fix failing tests Co-authored-by: Kyle Pollich <kyle.pollich@elastic.co>

* Add consistent _meta property to all Fleet ES assets * Fix meta logic for ILM + transforms * Fix tests + snapshots * Fix failing tests

…ic#119380) * Add consistent _meta property to all Fleet ES assets * Fix meta logic for ILM + transforms * Fix tests + snapshots * Fix failing tests

Add consistent _meta property to all Fleet ES assets

f8cbae0

kpollich added release_note:enhancement v8.0.0 Team:Fleet Team label for Observability Data Collection Fleet team auto-backport Deprecated - use backport:version if exact versions are needed v8.1.0 labels Nov 22, 2021

kpollich self-assigned this Nov 22, 2021

kpollich requested a review from a team as a code owner November 22, 2021 20:43

kpollich commented Nov 22, 2021

View reviewed changes

Fix meta logic for ILM + transforms

686c234

juliaElastic reviewed Nov 23, 2021

View reviewed changes

juliaElastic approved these changes Nov 23, 2021

View reviewed changes

kpollich added 2 commits November 23, 2021 08:41

Fix tests + snapshots

e48f0e7

Fix failing tests

47501a9

kpollich merged commit cee8dff into elastic:main Nov 23, 2021

kibanamachine mentioned this pull request Nov 23, 2021

[8.0] [Fleet] Add consistent _meta property to all Fleet ES assets (#119380) #119524

Merged

dmlemeshko pushed a commit that referenced this pull request Nov 29, 2021

[Fleet] Add consistent _meta property to all Fleet ES assets (#119380)

3c1db5f

* Add consistent _meta property to all Fleet ES assets * Fix meta logic for ILM + transforms * Fix tests + snapshots * Fix failing tests

joshdover mentioned this pull request Mar 2, 2022

[Fleet] Move mappings from index template to component template #124013

Merged

jsoriano mentioned this pull request Apr 6, 2022

Fix dump subcommand in 8.x elastic/elastic-package#785

Merged

joshdover mentioned this pull request Jun 2, 2022

Ingest Pipelines UI should indicate managed pipelines and allow filtering #133382

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Add consistent `_meta` property to all Fleet ES assets #119380

[Fleet] Add consistent `_meta` property to all Fleet ES assets #119380

kpollich commented Nov 22, 2021

elasticmachine commented Nov 22, 2021

kpollich Nov 22, 2021

nchaulet Nov 23, 2021

kpollich Nov 22, 2021

kpollich Nov 22, 2021

kpollich Nov 22, 2021

juliaElastic Nov 23, 2021

kpollich Nov 23, 2021

juliaElastic left a comment

kibana-ci commented Nov 23, 2021

kibanamachine commented Nov 23, 2021

[Fleet] Add consistent _meta property to all Fleet ES assets #119380

[Fleet] Add consistent _meta property to all Fleet ES assets #119380

Conversation

kpollich commented Nov 22, 2021

Summary

To Test

Examples

elasticmachine commented Nov 22, 2021

kpollich Nov 22, 2021

Choose a reason for hiding this comment

nchaulet Nov 23, 2021

Choose a reason for hiding this comment

kpollich Nov 22, 2021

Choose a reason for hiding this comment

kpollich Nov 22, 2021

Choose a reason for hiding this comment

kpollich Nov 22, 2021

Choose a reason for hiding this comment

juliaElastic Nov 23, 2021

Choose a reason for hiding this comment

kpollich Nov 23, 2021

Choose a reason for hiding this comment

juliaElastic left a comment

Choose a reason for hiding this comment

kibana-ci commented Nov 23, 2021

💚 Build Succeeded

Metrics [docs]

History

kibanamachine commented Nov 23, 2021

💚 Backport successful

[Fleet] Add consistent `_meta` property to all Fleet ES assets #119380

[Fleet] Add consistent `_meta` property to all Fleet ES assets #119380