[Fleet] Mark Fleet-managed Saved Object tags with managed: true

[kpollich]

When Fleet creates tags, we need to mark them as managed: true in order to prevent users from editing them directly in the Stack Management UI.

Relevant code

kibana/x-pack/plugins/fleet/server/services/epm/kibana/assets/tag_assets.ts

Lines 162 to 247 in c01ac61

	async function ensureManagedTag(
	opts: Pick<TagAssetsParams, 'spaceId' | 'savedObjectTagClient'>
	): Promise<string> {
	const { spaceId, savedObjectTagClient } = opts;
	const managedTagId = getManagedTagId(spaceId);
	const managedTag = await savedObjectTagClient.get(managedTagId).catch(() => {});
	if (managedTag) return managedTagId;
	const legacyManagedTag = await savedObjectTagClient.get(LEGACY_MANAGED_TAG_ID).catch(() => {});
	if (legacyManagedTag) return LEGACY_MANAGED_TAG_ID;
	await savedObjectTagClient.create(
	{
	name: MANAGED_TAG_NAME,
	description: '',
	color: MANAGED_TAG_COLOR,
	},
	{ id: managedTagId, overwrite: true, refresh: false }
	);
	return managedTagId;
	}
	async function ensurePackageTag(
	opts: Pick<TagAssetsParams, 'spaceId' | 'savedObjectTagClient' | 'pkgName' | 'pkgTitle'>
	): Promise<string> {
	const { spaceId, savedObjectTagClient, pkgName, pkgTitle } = opts;
	const packageTagId = getPackageTagId(spaceId, pkgName);
	const packageTag = await savedObjectTagClient.get(packageTagId).catch(() => {});
	if (packageTag) return packageTagId;
	const legacyPackageTagId = getLegacyPackageTagId(pkgName);
	const legacyPackageTag = await savedObjectTagClient.get(legacyPackageTagId).catch(() => {});
	if (legacyPackageTag) return legacyPackageTagId;
	await savedObjectTagClient.create(
	{
	name: pkgTitle,
	description: '',
	color: PACKAGE_TAG_COLOR,
	},
	{ id: packageTagId, overwrite: true, refresh: false }
	);
	return packageTagId;
	}
	// Ensure that asset tags coming from the kibana/tags.yml file are correctly parsed and created
	async function getPackageSpecTags(
	taggableAssets: ArchiveAsset[],
	opts: Pick<TagAssetsParams, 'spaceId' | 'savedObjectTagClient' | 'pkgName' | 'assetTags'>
	): Promise<PackageSpecTagsAssets[]> {
	const { spaceId, savedObjectTagClient, pkgName, assetTags } = opts;
	if (!assetTags || assetTags?.length === 0) return [];
	const assetsWithTags = await Promise.all(
	assetTags.map(async (tag) => {
	const uniqueTagId = getPackageSpecTagId(spaceId, pkgName, tag.text);
	const existingPackageSpecTag = await savedObjectTagClient.get(uniqueTagId).catch(() => {});
	if (!existingPackageSpecTag) {
	await savedObjectTagClient.create(
	{
	name: tag.text,
	description: 'Tag defined in package-spec',
	color: getRandomColor(),
	},
	{ id: uniqueTagId, overwrite: true, refresh: false }
	);
	}
	const assetTypes = getAssetTypesObjectReferences(tag?.asset_types, taggableAssets);
	const assetIds = getAssetIdsObjectReferences(tag?.asset_ids, taggableAssets);
	const totAssetsToAssign = assetTypes.concat(assetIds);
	const assetsToAssign = totAssetsToAssign.length > 0 ? uniqBy(totAssetsToAssign, 'id') : [];
	return { tagId: uniqueTagId, assets: assetsToAssign };
	})
	);
	return assetsWithTags;
	}

These tag objects should have managed: true provided when created.

We should also make sure we can update existing tags in users' deployments to have the managed: true property set. This can likely be done with an additional check when we discover an existing tag during the ensure calls above.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[drewdaemon]

Any progress on this? The Kibana managed content changes can't take effect until we have the managed flag for tags...

[kpollich]

Hey @drewdaemon - no progress on this yet. It's scheduled for delivery in 8.15.0. Is that an acceptable timeline here?

[drewdaemon]

@kpollich no pressure from me 👍 . Just making sure it's clear that the changes Fleet requested from platform WRT managed content won't be effectual until this is complete.

[jillguyonnet]

Hey @kpollich 👋 This draft PR contains the change to mark the tags with managed: true on package install.

On current main, after installing system from blank slate:

Vs. with the change, after installing system from blank slate:

I'm investigating routes for updating existing tags. The tagKibanaAssets function linked in the description is only called on package install, so the update needs to take place in another flow. The easiest solution (to me at least 🙂 ) would be updating the tags on package reinstall and/or update. This would allow eventual consistency, but I'm not sure if that would be acceptable timeline-wise. Otherwise, we might need to perform a backfill, which I'm unclear on at the moment. Do you have any recommendation on that?

[kpollich]

The easiest solution (to me at least 🙂 ) would be updating the tags on package reinstall and/or update. This would allow eventual consistency, but I'm not sure if that would be acceptable timeline-wise. Otherwise, we might need to perform a backfill, which I'm unclear on at the moment. Do you have any recommendation on that?

Definitely agree that the eventual consistency route would be the easiest lift in terms of development effort, but I'm not sure it's the right thing to do here. There's no guarantee that users will update all their integrations, and we'll have integrations that don't publish a new versions for many months that will prevent users from seeing these changes.

So, I think pursuing a backfill is probably the right solution here. We have some prior art here with a backfill at the Fleet setup level, e.g.

kibana/x-pack/plugins/fleet/server/services/setup.ts

Lines 204 to 205 in e35800d

	logger.debug('Backfilling output performance presets');
	await outputService.backfillAllOutputPresets(soClient, esClient);

kibana/x-pack/plugins/fleet/server/services/output.ts

Lines 1061 to 1085 in e35800d

	public async backfillAllOutputPresets(
	soClient: SavedObjectsClientContract,
	esClient: ElasticsearchClient
	) {
	const outputs = await this.list(soClient);
	await pMap(
	outputs.items.filter((output) => outputTypeSupportPresets(output.type) && !output.preset),
	async (output) => {
	const preset = getDefaultPresetForEsOutput(output.config_yaml ?? '', safeLoad);
	await outputService.update(
	soClient,
	esClient,
	output.id,
	{ preset },
	{ fromPreconfiguration: true }
	);
	await agentPolicyService.bumpAllAgentPoliciesForOutput(esClient, output.id);
	},
	{
	concurrency: 5,
	}
	);
	}

There might also be an avenue to migrate the tags saved objects themselves through a saved object migration, but I'm not sure where the actual tag saved object type is defined, and if we could add a migration for it. Maybe @elastic/kibana-core or @drewdaemon could help with identifying whether that's a possibility?

[drewdaemon]

Looks like the tag saved object is defined in x-pack/plugins/saved_objects_tagging/server/saved_objects/tag.ts.

I don't think saved object migrations would be a fit here because you are making a conditional change, not something for all tag saved objects. But I will let the core team advise.

cc @elastic/appex-sharedux , owners of the tag SO

[jillguyonnet]

I've been trying to see if I can update Fleet-created tags from Fleet setup, similar to the backfill linked above. The PR's second commit shows a WIP effort, which not actually working yet... I suspect there is much more to it 🙂

I came across #154515 while investigating how this might be implemented, although I'm not clear whether updates are supported yet. Perhaps someone from @elastic/kibana-core could chime in.

[pgayvallet]

I don't think saved object migrations would be a fit here because you are making a conditional change, not something for all tag saved objects

Yeah that's correct. Ideally "shared" types such as SOs shouldn't define migrations for "functional" migrations driven by a type consumer's need.

although I'm not clear whether updates are supported yet. Perhaps someone from @elastic/kibana-core could chime in.

Just to make sure, we're talking about updating the managed flag on existing objects, right?

In which case I think this is not something we're supporting right now, yeah, but ccing @TinaHeiligers as she's probably more up-to-date on that feature than I am.

[jillguyonnet]

Just to make sure, we're talking about updating the managed flag on existing objects, right?

Yes. The goal here is that Fleet-managed tag SO have managed: true. I found an easy way to make that happen for new tags (see #176924 (comment)), but we also need to update existing tags (which all existing Fleet setups would have).

[jillguyonnet]

After discussion, I opened #185972 for review with the restricted scope of marking new Fleet-managed tags with managed: true. Note that this won't lead to eventual consistency as we are missing a way to update existing tags, be that through package update/reinstall or backfill.

I have provisionally opened #186191 as a followup issue for backfilling existing tags, which will be prioritised depending on whether updating the managed flag on existing SO is supported or not.