[Fleet] Simplified package policy create API, enriching default values #119739

juliaElastic · 2021-11-25T14:56:32Z

Summary

Made most properties optional in POST /api/fleet/package_policies, loading default values from base package.
This is not a breaking change.

Remaining tasks:

add/update unit tests - DONE
update openapi spec - DONE

To verify:

send minimal request to API to check package policy is created with correct defaults
if one input is set to enabled: false, it is propagated down to all inputs.streams
if package.version does not match the installed base package version, there is an error coming
e.g.

POST http://elastic:changeme@localhost:5601/julia/api/fleet/package_policies
kbn-xsrf: kibana

{"name":"apache-5","inputs":[{"type":"logfile","enabled":true},{"type":"apache/metrics","enabled":false}],"package":{"name":"apache","version":"0.3.3"}}

create package policy from UI and customize values (agent_policy, input stream values, etc.) and verify custom values are persisted

Update on Nov 30, simplified update API. To verify:

create a package policy, note down the policy id
send a PUT request with minimal body (no required fields here) e.g. update namespace or description
verify that request was successful and only modified field has changed
the API allows resetting values as well e.g. description: "", enabled: false
example:

PUT http://elastic:changeme@localhost:5620/api/fleet/package_policies/64f13192-5dca-42f3-aa9c-62bc94c2f12c
kbn-xsrf: kibana

{ "description":"desc"}

Verify updated open api spec here:

Checklist

Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2021-11-25T14:56:38Z

Pinging @elastic/fleet (Team:Fleet)

juliaElastic · 2021-11-29T09:31:06Z

@elasticmachine merge upstream

x-pack/plugins/fleet/server/routes/package_policy/handlers.ts

juliaElastic · 2021-11-30T15:00:08Z

@elasticmachine merge upstream

juliaElastic · 2021-12-02T15:22:56Z

@elasticmachine merge upstream

nchaulet · 2021-12-02T17:58:09Z

x-pack/plugins/fleet/server/services/package_policy.ts

+      );
+      if (newPP) {
+        const inputs = newPolicy.inputs.map((input) => {
+          const defaultInput = newPP.inputs.find((i) => i.type === input.type);


Should we map on policy_template too here for granular integration like AWS?

@nchaulet In request only input.type and enabled is mandatory, so this code is mapping the rest of properties from the base package (streams, vars, etc.).
Could you give an example what you mean by the AWS use case?

Sure for example when you try to add the AWS billing integration this we are sending the following payload

as there is multiple input with the same type, so I think in the case where there is a policy_template on the payload input we should probably use it to find the default input.

It is clearer? I can rephrase otherwise :)

yes, it is clear now, so the enrich should consider policy_template too if set
I'll update it.

@nchaulet done, please check again when you get a chance

criamico · 2021-12-03T09:24:57Z

I tested locally and I'm having an issue when trying to update an integration policy for the fleet server package (I did PUT /package_policies/{packagePolicyId} with body { description: "desc"}. I'm getting a 500 with message [inputs.0.compiled_input]: definition for this key is missing.
I tried with other integrations (apache, aws, etc) and they are working fine. I'm not sure why this package would be different.

juliaElastic · 2021-12-03T11:39:35Z

@criamico good catch! I've fixed in latest commit, thanks

juliaElastic · 2021-12-06T08:04:34Z

@elasticmachine merge upstream

criamico

LGTM

…kibana into package-policy-api

kibana-ci · 2021-12-06T17:50:35Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💚 Build #11197 succeeded 4d8afcb
💛 Build #10919 was flaky b939ef3
💚 Build #10680 succeeded bdb5b79
💚 Build #10014 succeeded 464b119
💔 Build #9974 failed 72f8fb2
💚 Build #9896 succeeded 86aa9c7

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @juliaElastic

nchaulet

🚀

elastic#119739) * simplified api, enriching default values * removed comment * reverted changes in common models * fixed handlers test * separated create schema * fixed test * removed throwing error * getting package info from registry only * fix type * fix test and new tests * simplified update api * updated open api spec * fixed compiled_input * added policy_template enrich Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-12-07T07:53:20Z

💚 Backport successful

Status	Branch	Result
✅	8.0

This backport PR will be merged automatically after passing CI.

#119739) (#120576) * simplified api, enriching default values * removed comment * reverted changes in common models * fixed handlers test * separated create schema * fixed test * removed throwing error * getting package info from registry only * fix type * fix test and new tests * simplified update api * updated open api spec * fixed compiled_input * added policy_template enrich Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>

…-chromium-before-compiling-pdf * 'main' of github.com:elastic/kibana: (121 commits) FTR should use the new kibana_system user (elastic#120436) [Lens] Temporarily exclude Mosaic/Waffle from the suggestions list (elastic#120488) [Reporting] Fix slow CSV with large max size bytes (elastic#120365) [CTI] Threat Intel Card on Overview page needs to accommodate Fleet TI integrations - (elastic#120459) [Dashboard] Added KibanaThemeProvider. (elastic#120122) add more rule_registry unit tests (elastic#120323) [Lens] update xpack.lens.pie.smallValuesWarningMessage text (elastic#120478) [Fleet] Simplified package policy create API, enriching default values (elastic#119739) mock `elastic-apm-node` in `@kbn/test` jest preset (elastic#120324) [RAC] Rename occurrences of alert_type to rule_type in Infra (elastic#120455) [Security Solutions] Removes tech debt of exporting all from linter rule for timeline plugin (elastic#120437) [Workplace Search] Add API Keys view to replace Access tokens (elastic#120147) [Security Solutions] Removes tech debt of exporting all from linter rule for cases plugin in the server section (elastic#120411) Add support for threat.feed.name (elastic#120250) [Rule Registry] Rewrite APM registry rules for Observability (elastic#117740) [docs] Fix artifact layout formatting (elastic#119386) [Workplace Search] Add a technical preview of connecting GitHub via GitHub apps (elastic#119764) add osquery notes for 7.16 (elastic#120407) chore(NA): splits types from code on @kbn/docs-utils (elastic#120533) [Reporting] Decouple screenshotting plugin from the reporting (elastic#120110) ... # Conflicts: # x-pack/plugins/reporting/server/browsers/chromium/driver_factory/index.test.ts # x-pack/plugins/reporting/server/browsers/chromium/driver_factory/index.ts # x-pack/plugins/reporting/server/lib/screenshots/observable.test.ts # x-pack/plugins/reporting/server/lib/screenshots/observable.ts # x-pack/plugins/reporting/server/test_helpers/create_mock_browserdriverfactory.ts

elastic#119739) * simplified api, enriching default values * removed comment * reverted changes in common models * fixed handlers test * separated create schema * fixed test * removed throwing error * getting package info from registry only * fix type * fix test and new tests * simplified update api * updated open api spec * fixed compiled_input * added policy_template enrich Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

simplified api, enriching default values

7a7121f

juliaElastic added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes auto-backport Deprecated - use backport:version if exact versions are needed v8.1.0 labels Nov 25, 2021

juliaElastic requested a review from a team as a code owner November 25, 2021 14:56

juliaElastic self-assigned this Nov 25, 2021

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Nov 25, 2021

juliaElastic added 4 commits November 25, 2021 16:04

removed comment

bf82f16

reverted changes in common models

c3cda6e

fixed handlers test

f7fea5a

separated create schema

3bc070b

kibanamachine and others added 4 commits November 29, 2021 04:31

Merge branch 'main' into package-policy-api

3d6ba45

fixed test

a97fdb6

removed throwing error

395a33c

getting package info from registry only

6b58701

juliaElastic commented Nov 29, 2021

View reviewed changes

x-pack/plugins/fleet/server/routes/package_policy/handlers.ts Show resolved Hide resolved

juliaElastic added 4 commits November 29, 2021 17:30

fix type

5f64b22

fix test and new tests

86aa9c7

simplified update api

975968a

updated open api spec

72f8fb2

Merge branch 'main' into package-policy-api

464b119

Merge branch 'main' into package-policy-api

bdb5b79

nchaulet reviewed Dec 2, 2021

View reviewed changes

fixed compiled_input

b939ef3

Merge branch 'main' into package-policy-api

4d8afcb

criamico approved these changes Dec 6, 2021

View reviewed changes

juliaElastic added 2 commits December 6, 2021 17:33

added policy_template enrich

b71480e

Merge branch 'package-policy-api' of https://github.com/juliaElastic/…

550a21a

…kibana into package-policy-api

nchaulet approved these changes Dec 6, 2021

View reviewed changes

juliaElastic merged commit ccffda0 into elastic:main Dec 7, 2021

juliaElastic deleted the package-policy-api branch December 7, 2021 07:50

kibanamachine mentioned this pull request Dec 7, 2021

[8.0] [Fleet] Simplified package policy create API, enriching default values (#119739) #120576

Merged

criamico mentioned this pull request Dec 9, 2021

[Fleet] Improve public API for Agent Policy, Package Policy #113750

Closed

joshdover mentioned this pull request Jan 28, 2022

[Fleet] Remove default integrations #108456

Closed

3 tasks

joshdover mentioned this pull request Feb 15, 2022

/package_policies: package vars should be on input level #125625

Closed

joshdover mentioned this pull request May 16, 2022

[Fleet] Add a human-readable API and documentation for configuring package policies #132263

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Simplified package policy create API, enriching default values #119739

[Fleet] Simplified package policy create API, enriching default values #119739

juliaElastic commented Nov 25, 2021 •

edited by joshdover

Loading

elasticmachine commented Nov 25, 2021

juliaElastic commented Nov 29, 2021

juliaElastic commented Nov 30, 2021

juliaElastic commented Dec 2, 2021

nchaulet Dec 2, 2021

juliaElastic Dec 3, 2021 •

edited

Loading

nchaulet Dec 6, 2021

juliaElastic Dec 6, 2021

juliaElastic Dec 6, 2021

criamico commented Dec 3, 2021

juliaElastic commented Dec 3, 2021

juliaElastic commented Dec 6, 2021

criamico left a comment

kibana-ci commented Dec 6, 2021

nchaulet left a comment

kibanamachine commented Dec 7, 2021

[Fleet] Simplified package policy create API, enriching default values #119739

[Fleet] Simplified package policy create API, enriching default values #119739

Conversation

juliaElastic commented Nov 25, 2021 • edited by joshdover Loading

Summary

Checklist

For maintainers

elasticmachine commented Nov 25, 2021

juliaElastic commented Nov 29, 2021

juliaElastic commented Nov 30, 2021

juliaElastic commented Dec 2, 2021

nchaulet Dec 2, 2021

Choose a reason for hiding this comment

juliaElastic Dec 3, 2021 • edited Loading

Choose a reason for hiding this comment

nchaulet Dec 6, 2021

Choose a reason for hiding this comment

juliaElastic Dec 6, 2021

Choose a reason for hiding this comment

juliaElastic Dec 6, 2021

Choose a reason for hiding this comment

criamico commented Dec 3, 2021

juliaElastic commented Dec 3, 2021

juliaElastic commented Dec 6, 2021

criamico left a comment

Choose a reason for hiding this comment

kibana-ci commented Dec 6, 2021

💚 Build Succeeded

Metrics [docs]

History

nchaulet left a comment

Choose a reason for hiding this comment

kibanamachine commented Dec 7, 2021

💚 Backport successful

juliaElastic commented Nov 25, 2021 •

edited by joshdover

Loading

juliaElastic Dec 3, 2021 •

edited

Loading