[Fleet] Add "Keep Policies up to Date" functionality for integrations

[kpollich]

Summary

Final implementation piece of #106048

Adds a "Keep Policies Up to Date" setting to Elastic Agent integrations that allows Fleet to automatically upgrade integration policies when a new version of a package is installed. This setting is only exposed for "managed" packages that are specified in AUTO_UPGRADE_PACKAGES or DEFAULT_PACKAGES.

During calls to /api/fleet/setup, we now upgrade policies for any packages flagged with keep_policies_up_to_date: true, or those that appear in our hardcoded AUTO_UPGRADE_PACKAGES list in our preconfiguration constants.

08811a1 fixes #113731

Screen Recording

To-Do

• Adjust Integrations > Integration details > Settings to match design
  • Keep policies up-to-date checkbox and description
  • This checkbox should read/write to a new setting in package saved objects
• In the backend, whenever an auto-upgrade package is upgraded, check to see if this new setting is truthy, and use the package policy upgrade API to upgrade all of its policies
  • Make sure any upgraded policies save with the new package version on the package metadata
  • Any upgraded package policies should also trigger an update on their parent agent policies
  • Add tests
  • Add a saved object migration to have the new setting be truthy for those packages
  • For policies which cannot be upgraded, leave those alone for now, in the future it would be nice to have a notification system to let the user know, but for now we will rely on the user to manually upgrade them using the Upgrade package policy UI described above
• Investigate how much we can log to Kibana logs when upgrades are incurred

[kpollich]

@juliaElastic thanks for the review! I've addressed your comments in f332f7e

Since CI is finally green for this I'm marking it as ready for review 🎉

[kpollich]

@elasticmachine merge upstream

[kpollich]

FYI I tested the changes to the overridePackageInputs method using @jen-huang's test case in #113204 and everything looks good. Integration + unit tests have been improved around this method as well.

[nchaulet]

Did some local testing and everything seems to work as expected 🚀

[kibanamachine]

💛 Build succeeded, but was flaky

• continuous-integration/kibana-ci/pull-request
• Commit: 9f1b9cb
• Storybooks Preview
• Documentation Changes
• Flaky suites:
  • xpack-securitySolutionCypressChrome

---

Test Failures

Kibana Pipeline / general / Allows the rule to be duplicated from the edit screen.indicator match Detection rules, Indicator Match Duplicates the indicator rule Allows the rule to be duplicated from the edit screen

Link to Jenkins

Stack Trace

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

AssertionError: Timed out retrying after 60000ms: expected '<button.euiContextMenuItem>' to be 'visible'

This element `<button.euiContextMenuItem>` is not visible because its parent `<div>` has CSS property: `visibility: hidden`
    at Context.eval (http://localhost:6121/__cypress/tests?p=cypress/integration/detection_rules/indicator_match_rule.spec.ts:31238:58)

---

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
fleet	576	577	+1

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
fleet	1096	1105	+9

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
fleet	598.2KB	600.3KB	+2.1KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	125.9KB	126.3KB	+415.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
epm-packages	18	19	+1

Unknown metric groups

API count

id	before	after	diff
fleet	1197	1206	+9

History

• 💚 Build #157741 succeeded 5cc5712
• 💔 Build #157658 failed 4c53fa5
• 💔 Build #157621 failed 08811a1
• 💔 Build #157622 failed 7ef307a
• 💔 Build #157114 failed 3519e65

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @kpollich

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

[dikshachauhan-qasource]

Hi @kpollich

We have below queries on this merges regarding auto upgrade of default or managed integrations.

As far as we have noticed earlier also, these integrations were capable of auto upgrading by default.

• However, could you please elaborate us more how this setting is gonna help users and use case of this setting.
• Also can you please confirm us if we keep it disable then would it not low package to upgrade by itself even if new versions are available on same build.
• Further, when we manually tried to degrade package version below message is shown up " "message": "endpoint-0.13.0 is out-of-date and cannot be installed or updated""

So here, question arises, in which situation does this setting be helpful, when during build creation it automatically downloads latest version for default packages.

Screenshot:
Attempted to degrade package version For system integration:

For endpoint security:

Thanks
QAS

[kpollich]

@dikshachauhan-qasource - thanks very much for your questions. I'll try to answer them below.

However, could you please elaborate us more how this setting is gonna help users and use case of this setting.

The "Keep Policies up to date" setting is intended to allow users to opt into keeping "managed" or "internal" integration policies for integrations like system endpoint etc up-to-date automatically. So when a new version of endpoint is installed, any deployed policies will also be upgraded at the same time.

If the user does not opt into this functionality using the checkbox, endpoint will still be updated to its latest version, the user will just need to manually upgrade all of their policies after the fact. The intent of the checkbox is to make this entire process automatic and avoid the need for user intervention. It's essentially an opt-in to allow Fleet to manage everything top-to-bottom for its associated with its various managed integrations.

Also can you please confirm us if we keep it disable then would it not low package to upgrade by itself even if new versions are available on same build.

The checkbox does not have any bearing on the actual package updating process. endpoint for example is specified as an "auto-update" package in the Fleet code here:

kibana/x-pack/plugins/fleet/common/constants/epm.ts

Lines 46 to 50 in de59cfe

	export const autoUpdatePackages = [
	FLEET_ENDPOINT_PACKAGE,
	FLEET_APM_PACKAGE,
	FLEET_SYNTHETICS_PACKAGE,
	];

The "Keep Policies up to date" checkbox only controls whether Fleet will automatically upgrade and deploy new policy versions when a new version of the package is installed.

Further, when we manually tried to degrade package version below message is shown up " "message": "endpoint-0.13.0 is out-of-date and cannot be installed or updated""

I'm not sure this is related to any new functionality. @jen-huang @nchaulet can you advise? That error message above seems to pre-date any of the policy upgrade implementations. I'm not sure we've ever supported downgrading managed packages in this way.

[nchaulet]

I'm not sure this is related to any new functionality. @jen-huang @nchaulet can you advise? That error message above seems to pre-date any of the policy upgrade implementations. I'm not sure we've ever supported downgrading managed packages in this way.

No it's not a new functionality you cannot install an old version of a package unless you pass the force: true flag in the request body

[dikshachauhan-qasource]

Hi @kpollich

We have validated auto upgrade of package with policies as well on 7.16 BC1 build and found below reported issues.

• [Security Solution] User is unable to upgrade the ES version from 1.1.1 to 1.2.0  #116396
• https://github.com/elastic/integrations/issues/2038

Validated on 7.16 BC1. Build details are as follows:
BUILD 45504
COMMIT 9231d80

Further, Once fixes are available, we will start working upon testcases too under this feature for appropriate coverage.

Thanks
QAS