[Fleet] Add toggle for experimental synthetic `_source` support in Fleet data streams #140132

kpollich · 2022-09-06T20:32:10Z

Summary

Adds a "synthetic source" toggle to the policy editor for all data streams.

In cases where a data stream contains mappings that are incompatible with synthetic source, the errors will be bubbled directly to the user and the policy will not be save-able. e.g.

TODO

Figure out docs link

kpollich · 2022-09-07T12:25:46Z

x-pack/plugins/cloud_security_posture/server/fleet_integration/fleet_integration.ts

@@ -85,7 +86,7 @@ export const onPackagePolicyPostCreateCallback = async (
 * Callback to handle deletion of PackagePolicies in Fleet
 */
 export const removeCspRulesInstancesCallback = async (
-  deletedPackagePolicy: DeletePackagePoliciesResponse[number],
+  deletedPackagePolicy: DeepReadonly<DeletePackagePoliciesResponse[number]>,


This fixes a type issue that was revealed when we added an array field to the PackagePolicyPackage type that gets used in the DeletePackagePoliciesResponse type.

I think updating this type to expect a DeepReadOnly is actually just correct based on the usage of this callback.

kpollich · 2022-09-07T12:26:35Z

...olicy/create_package_policy_page/components/steps/components/package_policy_input_stream.tsx

-              const varConfigEntry = packagePolicyInputStream.vars?.[varName];
-              const value = varConfigEntry?.value;
-              const frozen = varConfigEntry?.frozen ?? false;
+      <>


A lot of this diff is whitespace due to introducing a fragment at the top level. GitHub's ignore whitespace setting for this PR is probably your friend here 🙂

kpollich · 2022-09-07T12:27:33Z

x-pack/plugins/fleet/server/saved_objects/index.ts

+            features: {
+              type: 'nested',
+              properties: {
+                synthetic_source: { type: 'boolean' },


I felt it was better to be explicit about what features exist here, rather than just accepting any set of key/values. We'll need to expand this with additional settings moving forward.

kpollich · 2022-09-07T12:29:36Z

x-pack/plugins/fleet/server/services/package_policies/experimental_datastream_features.ts

+  );
+
+  // Delete the experimental features map from the package policy so it doesn't get persisted
+  delete packagePolicy.package.experimental_data_stream_features;


We pass the experimental data stream features array on the actual package policy object that comes back through the package policy create/update APIs. This is great for getting the data into our service here, but we don't actually want to persist that experimental features array on the package policy object.

This function "hoists" the experimental features array up out of the package policy to store it at a higher level, so we need to clean it up at the end.

elasticmachine · 2022-09-07T15:45:37Z

Pinging @elastic/fleet (Team:Fleet)

x-pack/plugins/fleet/server/services/package_policies/experimental_datastream_features.ts

kpollich · 2022-09-07T16:14:43Z

x-pack/plugins/fleet/server/services/package_policies/experimental_datastream_features.ts

+        mappings: {
+          ...componentTemplate.template.mappings,
+          _source: {
+            mode: featureMapEntry.features.synthetic_source ? 'synthetic' : 'stored',


@nchaulet here we determine the value for the synthetic source property.

Although looking at this now I think we can optimize to not perform a PUT if the value is unchanged.

nchaulet · 2022-09-07T16:23:47Z

@kpollich Are missing some code also to handle the upgrade of a package, I think the experiment will be loss on package update here no? maybe the same for force reinstall too?

nchaulet · 2022-09-07T16:25:34Z

x-pack/plugins/fleet/server/services/package_policy.ts

@@ -873,6 +907,10 @@ class PackagePolicyService implements PackagePolicyServiceInterface {
          namespace: newPolicy.namespace ?? 'default',
          description: newPolicy.description ?? '',
          enabled: newPolicy.enabled ?? true,
+          package: {
+            ...newPP.package!,
+            experimental_data_stream_features: newPolicy.package?.experimental_data_stream_features,


Should we persist the experimental features on the package policy, or populate that from the package saved object? it seems to me that that feature is shared by all the package policy using this package

We only have the value on the package policy here so we can pass it easily to and from the policy editor. It gets persisted on the epm-package saved object.

Putting it on packagePolicy.package here just made it easiest to access when rendering the policy editor. Open to other approaches.

Putting it on packagePolicy.package here just made it easiest to access when rendering the policy editor. Open to other approaches.

let's consider you have two package policy using nginx package (package policy 1, package policy 2)

you edit package policy 1 to enable the experiment

you visit package policy 2 the experiment is shown as disabled but in reality it's enabled no?

I think we should fetch the installation in the package policy editor.

it is clearer?

I added logic to packagePolicyService.get in this PR to provide the experimental info on packagePolicy.package in the API response for package policies. We do fetch from the installation saved object and hydrate the resulting API response body with the experimental feature data.

It might make a little more sense to return it separately somehow, or to have a separate API request for the policy editor to fetch the installation on its own, though. I could see an argument for that.

If we fetch it in the API response there is probably no need to persist on the package policy saved object no?

Yeah we don't want to persist it on the package policy saved object. It shouldn't be saved there at all. We are deleteing the property before it gets saved. If it's being saved then it's a bug in this PR.

kpollich · 2022-09-07T16:32:12Z

@kpollich Are missing some code also to handle the upgrade of a package, I think the experiment will be loss on package update here no? maybe the same for force reinstall too?

Yeah I think we need some logic in our upgrade API to pull the values of these experimental opt-ins. I will take a look at this.

kpollich · 2022-09-07T16:32:59Z

Force install is an interesting case. I would expect a force reinstall to wipe out any existing package level settings like this, so I think this is working as intended. Curious to hear if there are objections to that approach.

…kibana into 140095-synthetic-source-toggle

kpollich · 2022-09-07T17:20:01Z

x-pack/plugins/fleet/server/services/package_policy.ts

+    let experimentalFeatures: ExperimentalDataStreamFeature[] | undefined;
+
+    if (packagePolicySO.attributes.package?.name) {
+      const installation = await soClient.get<Installation>(
+        PACKAGES_SAVED_OBJECT_TYPE,
+        packagePolicySO.attributes.package?.name
+      );
+
+      if (installation && !installation.error) {
+        experimentalFeatures = installation.attributes?.experimental_data_stream_features;
+      }
+    }
+
+    const response = {
      id: packagePolicySO.id,
      version: packagePolicySO.version,
      ...packagePolicySO.attributes,
    };
+
+    // If possible, return the experimental features map for the package policy's `package` field
+    if (experimentalFeatures && response.package) {
+      response.package.experimental_data_stream_features = experimentalFeatures;
+    }
+
+    return response;
  }


@nchaulet here's where we fetch the installation and include it in the package policy object returned from packagePolicyService.get()

nchaulet · 2022-09-07T22:34:57Z

I think I found another path where we need to handle that, when you bump FLEET_INSTALL_FORMAT_VERSION looks like the settings is not preserved on the next setup

kpollich · 2022-09-08T13:33:26Z

Upgrade path has been updated to handle these new settings in 50f2edd..

I think I found another path where we need to handle that, when you bump FLEET_INSTALL_FORMAT_VERSION looks like the settings is not preserved on the next setup

Taking a look at this next.

kpollich · 2022-09-08T13:33:37Z

@elasticmachine merge upstream

kpollich · 2022-09-09T12:30:46Z

@nchaulet Can you clarify the code path where FLEET_FORMAT_VERSION is being updated and blowing away the experimental features settings? I've been digging through the code but it's not something I've come across before and I'm having trouble tracking down the specific path we're talking about here.

kpollich · 2022-09-09T12:33:49Z

I removed the "Review our docs" link from this UI for now since we don't have docs to link to for these experimental features. When we take on the other two features it'd probably make sense to reach out to the docs team to get something together then, but for now with just the one-off synthetic source toggle that's mainly going to be used by integration maintainers to test the feature I think it's fine to remove the link.

kpollich · 2022-09-09T12:33:56Z

@elasticmachine merge upstream

kpollich · 2022-09-09T16:57:01Z

I ran through this process and everything worked as expected

Install a package and set synthetic source = true on a data stream
Bump the hard coded FLEET_INSTALL_FORMAT_VERSION constant in Fleet’s config
Re-run Fleet setup
Verify that the synthetic source setting remains enabled on the data stream

All of this worked as expected after a few fixes I made in d8d751a

I've also added a few tests for the experimental data stream settings utilities.

kpollich · 2022-09-09T16:57:06Z

@elasticmachine merge upstream

kpollich · 2022-09-12T12:34:26Z

@elasticmachine merge upstream

kibana-ci · 2022-09-12T13:33:44Z

💚 Build Succeeded

Buildkite Build
Commit: c4a08be

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	873	875	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	865.8KB	867.6KB	+1.7KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`fleet`	10	11	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	105.7KB	105.7KB	+40.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/saved-objects-service.html#_mappings

id	before	after	diff
`epm-packages`	22	26	+4

Unknown metric groups

API count

id	before	after	diff
`fleet`	970	972	+2

History

💚 Build #70953 succeeded 50ad37d
💚 Build #70814 succeeded ee4339a
💚 Build #70441 succeeded 3b3ed46
💚 Build #70180 succeeded f79124d
💛 Build #70107 was flaky 39934df
💔 Build #70057 failed 30a89e8

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @kpollich

First pass at synthetic source support in Fleet

ed268ca

kpollich added release_note:enhancement Team:Fleet Team label for Observability Data Collection Fleet team labels Sep 6, 2022

kpollich self-assigned this Sep 6, 2022

Fix some type failures + clean up naming

30a89e8

kpollich commented Sep 7, 2022

View reviewed changes

Fix more types

39934df

kpollich marked this pull request as ready for review September 7, 2022 15:45

kpollich requested review from a team as code owners September 7, 2022 15:45

Merge branch 'main' into 140095-synthetic-source-toggle

67e00f7

nchaulet reviewed Sep 7, 2022

View reviewed changes

x-pack/plugins/fleet/server/services/package_policies/experimental_datastream_features.ts Show resolved Hide resolved

kpollich commented Sep 7, 2022

View reviewed changes

nchaulet reviewed Sep 7, 2022

View reviewed changes

kpollich added 2 commits September 7, 2022 12:37

Avoid needless PUTs to component templates when opt-in is unchanged

671acb5

Merge branch '140095-synthetic-source-toggle' of github.com:kpollich/…

f79124d

…kibana into 140095-synthetic-source-toggle

kpollich commented Sep 7, 2022

View reviewed changes

Honor experimental data stream settings during dry-run/upgrade workflow

50f2edd

Merge branch 'main' into 140095-synthetic-source-toggle

3b3ed46

orouz approved these changes Sep 9, 2022

View reviewed changes

Remove docs link for now

49e045c

kibanamachine and others added 2 commits September 9, 2022 06:33

Merge branch 'main' into 140095-synthetic-source-toggle

ee4339a

Fix various bugs in logic/lookups + add tests

d8d751a

Merge branch 'main' into 140095-synthetic-source-toggle

50ad37d

kpollich requested a review from nchaulet September 12, 2022 12:34

Merge branch 'main' into 140095-synthetic-source-toggle

c4a08be

kpollich mentioned this pull request Sep 12, 2022

[Fleet] Create UI for experimental indexing features #139862

Closed

14 tasks

nchaulet approved these changes Sep 12, 2022

View reviewed changes

kpollich merged commit cc5ff75 into elastic:main Sep 12, 2022

kibanamachine added v8.5.0 backport:skip This commit does not require backporting labels Sep 12, 2022

jen-huang added the QA:Ready for Testing Code is merged and ready for QA to validate label Sep 19, 2022

kpollich deleted the 140095-synthetic-source-toggle branch September 23, 2022 12:45

kpollich mentioned this pull request Oct 11, 2022

[Fleet] Fix bugs w/ synthetic source toggle #143097

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Add toggle for experimental synthetic `_source` support in Fleet data streams #140132

[Fleet] Add toggle for experimental synthetic `_source` support in Fleet data streams #140132

kpollich commented Sep 6, 2022 •

edited

Loading

kpollich Sep 7, 2022

kpollich Sep 7, 2022

kpollich Sep 7, 2022

kpollich Sep 7, 2022

elasticmachine commented Sep 7, 2022

kpollich Sep 7, 2022

nchaulet commented Sep 7, 2022

nchaulet Sep 7, 2022

kpollich Sep 7, 2022

nchaulet Sep 7, 2022

kpollich Sep 7, 2022

nchaulet Sep 7, 2022

kpollich Sep 7, 2022

kpollich commented Sep 7, 2022

kpollich commented Sep 7, 2022

kpollich Sep 7, 2022

nchaulet commented Sep 7, 2022

kpollich commented Sep 8, 2022

kpollich commented Sep 8, 2022

kpollich commented Sep 9, 2022

kpollich commented Sep 9, 2022

kpollich commented Sep 9, 2022

kpollich commented Sep 9, 2022

kpollich commented Sep 9, 2022

kpollich commented Sep 12, 2022

kibana-ci commented Sep 12, 2022

API count

[Fleet] Add toggle for experimental synthetic _source support in Fleet data streams #140132

[Fleet] Add toggle for experimental synthetic _source support in Fleet data streams #140132

Conversation

kpollich commented Sep 6, 2022 • edited Loading

Summary

TODO

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Sep 7, 2022

Choose a reason for hiding this comment

nchaulet commented Sep 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kpollich commented Sep 7, 2022

kpollich commented Sep 7, 2022

Choose a reason for hiding this comment

nchaulet commented Sep 7, 2022

kpollich commented Sep 8, 2022

kpollich commented Sep 8, 2022

kpollich commented Sep 9, 2022

kpollich commented Sep 9, 2022

kpollich commented Sep 9, 2022

kpollich commented Sep 9, 2022

kpollich commented Sep 9, 2022

kpollich commented Sep 12, 2022

kibana-ci commented Sep 12, 2022

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Async chunks

Public APIs missing exports

Page load bundle

Saved Objects .kibana field count

API count

History

[Fleet] Add toggle for experimental synthetic `_source` support in Fleet data streams #140132

[Fleet] Add toggle for experimental synthetic `_source` support in Fleet data streams #140132

kpollich commented Sep 6, 2022 •

edited

Loading