Skip to content

Commit 54f7174

Browse files
juliaElastickibanamachine
juliaElastic
and
kibanamachine
authored
[Fleet] fix content package task interval config (#230369)
## Summary Follow up for elastic/ingest-dev#5685 Fix fleet config to change auto install content packages task interval: ``` # enable feature flag xpack.fleet.enableExperimental: ['enableAutoInstallContentPackages'] # change interval xpack.fleet.autoInstallContentPackages.taskInterval: 15s ``` Also use prerelease flag from settings to be able to test auto install prerelease content package like system_otel: <img width="875" height="1006" alt="image" src="https://github.com/user-attachments/assets/7d22bbc8-5b32-4af5-97ec-fb4dca644f42" /> Added `discovery.datasets` field to `system_otel` package to test with, but for some reason the datasets values are not showing up in the EPR response ([PR](elastic/integrations#14770)) Tested with the package built locally and starting a local registry: [system_otel-0.2.0.zip](https://github.com/user-attachments/files/21574295/system_otel-0.2.0.zip) Update: this now works, had to pull the latest registry docker image ``` # start local registry docker run -v /Users/juliabardi/kibana/x-pack/platform/test/fleet_api_integration/apis/fixtures/package_registry_config.yml:/package-registry/config.yml -v /Users/juliabardi/kibana/x-pack/platform/test/fleet_api_integration/apis/fixtures/test_packages:/packages/test-packages -v /Users/juliabardi/kibana/x-pack/platform/test/fleet_api_integration/apis/fixtures/package_verification/packages/zips:/packages/signed-test-packages -p 12345:8080 docker.elastic.co/package-registry/distribution:lite # use local registry in kibana config xpack.fleet.registryUrl: http://localhost:12345 http://localhost:12345/search?prerelease=true&type=content&spec.min=2.3&spec.max=3.4&discovery=datasets:hostmetricsreceiver.otel [ { name: "system_otel", title: "System OpenTelemetry Assets", version: "0.2.0", release: "beta", description: "Dashboards for the OpenTelemetry data collected with the `hostmetrics` receiver.", type: "content", ... discovery: { datasets: [ { name: "hostmetricsreceiver.otel" } ] } } ] ``` Generated some otel data by following the instructions in OpenTelemetry integration in a multipass VM: ``` arch=$(if ([[ $(arch) == "arm" || $(arch) == "aarch64" ]]); then echo "arm64"; else echo $(arch); fi) curl --output elastic-distro-9.1.0-linux-$arch.tar.gz --url https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.1.0-linux-$arch.tar.gz --proto '=https' --tlsv1.2 -fL && mkdir -p elastic-distro-9.1.0-linux-$arch && tar -xvf elastic-distro-9.1.0-linux-$arch.tar.gz -C "elastic-distro-9.1.0-linux-$arch" --strip-components=1 && cd elastic-distro-9.1.0-linux-$arch rm ./otel.yml && cp ./otel_samples/platformlogs_hostmetrics.yml ./otel.yml && mkdir -p ./data/otelcol && sed -i 's#\${env:STORAGE_DIR}#'"$PWD"/data/otelcol'#g' ./otel.yml && sed -i 's#\${env:ELASTIC_ENDPOINT}#http://192.168.64.1:9200#g' ./otel.yml && sed -i 's/\${env:ELASTIC_API_KEY}/NEdfc2RKZ0JobV8xTXQxMjk0cUY6XzliVWIzSDRYZVBFSU9Ea05FTFdJdw==/g' ./otel.yml ``` And updating pipelines: ``` vim otel.yml service: extensions: [file_storage] pipelines: metrics/hostmetrics: receivers: [hostmetrics/system] processors: [resourcedetection, attributes/dataset, resource/process] exporters: [elasticsearch/otel] # start collector sudo ./otelcol --config otel.yml ``` For some reason the ES|QL query in the task doesn't return the `hostmetricsreceiver.otel` dataset, but is returned when running the same query in dev tools. Might be a permission issue. ``` FROM logs-*,metrics-*,traces-* | KEEP @timestamp, data_stream.dataset | WHERE @timestamp > NOW() - 15 minutes | STATS COUNT(*) BY data_stream.dataset { "took": 6, "is_partial": false, "documents_found": 5752, "values_loaded": 5752, "columns": [ { "name": "COUNT(*)", "type": "long" }, { "name": "data_stream.dataset", "type": "keyword" } ], "values": [ [ 115, "elastic_agent" ], [ 27, "elastic_agent.filebeat" ], [ 3, "elastic_agent.status_change" ], [ 1696, "elastic_agent.metricbeat" ], [ 3336, "elastic_agent.fleet_server" ], [ 271, "fleet_server.agent_versions" ], [ 271, "fleet_server.agent_status" ] ] } ``` <img width="2511" height="1101" alt="image" src="https://github.com/user-attachments/assets/03721494-6ad0-4d69-8c79-a23d82ac1c84" /> Update: Yes, it seems `kibana_system` didn't have privileges to read the integration data streams. Tested locally by adding read access, and now the query returns the expected results and installs the otel content package. I'll create a pr to the elasticsearch repo. Here it is: elastic/elasticsearch#132400 ``` # checkout ES pr, start es from source yarn es source --license trial -E xpack.security.authc.api_key.enabled=true -E xpack.security.authc.token.enabled=true --source-path=/Users/juliabardi/elasticsearch -E path.data=/tmp/es-data -E xpack.ml.enabled=false -E http.host=0.0.0.0 # kibana logs that run the task [2025-08-04T15:51:43.748+02:00][DEBUG][plugins.fleet.fleet:auto-install-content-packages-task:1.0.1] [AutoInstallContentPackagesTask] Found datasets with data: hostmetricsreceiver.otel, elastic_agent, fleet_server.agent_status, system.process.summary, system.socket_summary, elastic_agent.filebeat_input, elastic_agent.metricbeat, elastic_agent.filebeat, system.memory, elastic_agent.elastic_agent, generic.otel, elastic_agent.status_change, hostmetricsreceiver, elastic_agent.fleet_server, system.uptime, system.cpu, system.process, system.filesystem, system.fsstat, system.load, fleet_server.agent_versions, system.network, system.diskio [2025-08-04T15:51:43.748+02:00][DEBUG][plugins.fleet.fleet:auto-install-content-packages-task:1.0.1] [AutoInstallContentPackagesTask] Content packages to install: system_otel@0.2.0 [2025-08-04T15:51:43.748+02:00][DEBUG][plugins.fleet] Kicking off install of system_otel-0.2.0 from registry ``` ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
1 parent 12172bd commit 54f7174

File tree

9 files changed

+183
-14
lines changed

9 files changed

+183
-14
lines changed

‎oas_docs/bundle.json‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21581,6 +21581,21 @@
2158121581
"discovery": {
2158221582
"additionalProperties": true,
2158321583
"properties": {
21584+
"datasets": {
21585+
"items": {
21586+
"additionalProperties": true,
21587+
"properties": {
21588+
"name": {
21589+
"type": "string"
21590+
}
21591+
},
21592+
"required": [
21593+
"name"
21594+
],
21595+
"type": "object"
21596+
},
21597+
"type": "array"
21598+
},
2158421599
"fields": {
2158521600
"items": {
2158621601
"additionalProperties": true,
@@ -23714,6 +23729,21 @@
2371423729
"discovery": {
2371523730
"additionalProperties": true,
2371623731
"properties": {
23732+
"datasets": {
23733+
"items": {
23734+
"additionalProperties": true,
23735+
"properties": {
23736+
"name": {
23737+
"type": "string"
23738+
}
23739+
},
23740+
"required": [
23741+
"name"
23742+
],
23743+
"type": "object"
23744+
},
23745+
"type": "array"
23746+
},
2371723747
"fields": {
2371823748
"items": {
2371923749
"additionalProperties": true,
@@ -24647,6 +24677,21 @@
2464724677
"discovery": {
2464824678
"additionalProperties": true,
2464924679
"properties": {
24680+
"datasets": {
24681+
"items": {
24682+
"additionalProperties": true,
24683+
"properties": {
24684+
"name": {
24685+
"type": "string"
24686+
}
24687+
},
24688+
"required": [
24689+
"name"
24690+
],
24691+
"type": "object"
24692+
},
24693+
"type": "array"
24694+
},
2465024695
"fields": {
2465124696
"items": {
2465224697
"additionalProperties": true,

‎oas_docs/bundle.serverless.json‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21581,6 +21581,21 @@
2158121581
"discovery": {
2158221582
"additionalProperties": true,
2158321583
"properties": {
21584+
"datasets": {
21585+
"items": {
21586+
"additionalProperties": true,
21587+
"properties": {
21588+
"name": {
21589+
"type": "string"
21590+
}
21591+
},
21592+
"required": [
21593+
"name"
21594+
],
21595+
"type": "object"
21596+
},
21597+
"type": "array"
21598+
},
2158421599
"fields": {
2158521600
"items": {
2158621601
"additionalProperties": true,
@@ -23714,6 +23729,21 @@
2371423729
"discovery": {
2371523730
"additionalProperties": true,
2371623731
"properties": {
23732+
"datasets": {
23733+
"items": {
23734+
"additionalProperties": true,
23735+
"properties": {
23736+
"name": {
23737+
"type": "string"
23738+
}
23739+
},
23740+
"required": [
23741+
"name"
23742+
],
23743+
"type": "object"
23744+
},
23745+
"type": "array"
23746+
},
2371723747
"fields": {
2371823748
"items": {
2371923749
"additionalProperties": true,
@@ -24647,6 +24677,21 @@
2464724677
"discovery": {
2464824678
"additionalProperties": true,
2464924679
"properties": {
24680+
"datasets": {
24681+
"items": {
24682+
"additionalProperties": true,
24683+
"properties": {
24684+
"name": {
24685+
"type": "string"
24686+
}
24687+
},
24688+
"required": [
24689+
"name"
24690+
],
24691+
"type": "object"
24692+
},
24693+
"type": "array"
24694+
},
2465024695
"fields": {
2465124696
"items": {
2465224697
"additionalProperties": true,

‎oas_docs/output/kibana.serverless.yaml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24051,6 +24051,16 @@ paths:
2405124051
additionalProperties: true
2405224052
type: object
2405324053
properties:
24054+
datasets:
24055+
items:
24056+
additionalProperties: true
24057+
type: object
24058+
properties:
24059+
name:
24060+
type: string
24061+
required:
24062+
- name
24063+
type: array
2405424064
fields:
2405524065
items:
2405624066
additionalProperties: true
@@ -25220,6 +25230,16 @@ paths:
2522025230
additionalProperties: true
2522125231
type: object
2522225232
properties:
25233+
datasets:
25234+
items:
25235+
additionalProperties: true
25236+
type: object
25237+
properties:
25238+
name:
25239+
type: string
25240+
required:
25241+
- name
25242+
type: array
2522325243
fields:
2522425244
items:
2522525245
additionalProperties: true
@@ -25851,6 +25871,16 @@ paths:
2585125871
additionalProperties: true
2585225872
type: object
2585325873
properties:
25874+
datasets:
25875+
items:
25876+
additionalProperties: true
25877+
type: object
25878+
properties:
25879+
name:
25880+
type: string
25881+
required:
25882+
- name
25883+
type: array
2585425884
fields:
2585525885
items:
2585625886
additionalProperties: true

‎oas_docs/output/kibana.yaml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26293,6 +26293,16 @@ paths:
2629326293
additionalProperties: true
2629426294
type: object
2629526295
properties:
26296+
datasets:
26297+
items:
26298+
additionalProperties: true
26299+
type: object
26300+
properties:
26301+
name:
26302+
type: string
26303+
required:
26304+
- name
26305+
type: array
2629626306
fields:
2629726307
items:
2629826308
additionalProperties: true
@@ -27462,6 +27472,16 @@ paths:
2746227472
additionalProperties: true
2746327473
type: object
2746427474
properties:
27475+
datasets:
27476+
items:
27477+
additionalProperties: true
27478+
type: object
27479+
properties:
27480+
name:
27481+
type: string
27482+
required:
27483+
- name
27484+
type: array
2746527485
fields:
2746627486
items:
2746727487
additionalProperties: true
@@ -28093,6 +28113,16 @@ paths:
2809328113
additionalProperties: true
2809428114
type: object
2809528115
properties:
28116+
datasets:
28117+
items:
28118+
additionalProperties: true
28119+
type: object
28120+
properties:
28121+
name:
28122+
type: string
28123+
required:
28124+
- name
28125+
type: array
2809628126
fields:
2809728127
items:
2809828128
additionalProperties: true

‎x-pack/platform/plugins/shared/fleet/common/types/models/package_spec.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,9 @@ export interface PackageSpecManifest {
3737
};
3838
asset_tags?: PackageSpecTags[];
3939
discovery?: {
40+
fields?: Array<{
41+
name: string;
42+
}>;
4043
datasets?: DiscoveryDataset[];
4144
};
4245
}

‎x-pack/platform/plugins/shared/fleet/server/config.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -333,6 +333,11 @@ export const config: PluginConfigDescriptor = {
333333
taskInterval: schema.maybe(schema.string()),
334334
})
335335
),
336+
autoInstallContentPackages: schema.maybe(
337+
schema.object({
338+
taskInterval: schema.maybe(schema.string()),
339+
})
340+
),
336341
integrationsHomeOverride: schema.maybe(schema.string()),
337342
prereleaseEnabledByDefault: schema.boolean({ defaultValue: false }),
338343
hideDashboards: schema.boolean({ defaultValue: false }),

‎x-pack/platform/plugins/shared/fleet/server/tasks/auto_install_content_packages_task.test.ts‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,9 @@ jest.mock('../services/epm/registry');
3434
jest.mock('../services/epm/packages', () => ({
3535
getInstalledPackages: jest.fn(),
3636
}));
37+
jest.mock('../services/epm/packages/get_prerelease_setting', () => ({
38+
getPrereleaseFromSettings: jest.fn().mockReturnValue(false),
39+
}));
3740

3841
const MockRegistry = jest.mocked(Registry);
3942

@@ -170,8 +173,10 @@ describe('AutoInstallContentPackagesTask', () => {
170173
});
171174
expect(packageClientMock.installPackage).toHaveBeenCalledTimes(2);
172175
expect(esClient.esql.query).toHaveBeenCalledWith({
173-
query:
174-
'FROM logs-*,metrics-*,traces-* | KEEP @timestamp, data_stream.dataset | WHERE @timestamp > NOW() - 10 minutes | STATS COUNT(*) BY data_stream.dataset | LIMIT 100 ',
176+
query: `FROM logs-*,metrics-*,traces-*
177+
| KEEP @timestamp, data_stream.dataset
178+
| WHERE @timestamp > NOW() - 10 minutes
179+
| STATS COUNT(*) BY data_stream.dataset `,
175180
});
176181
});
177182

@@ -195,8 +200,10 @@ describe('AutoInstallContentPackagesTask', () => {
195200
});
196201
expect(packageClientMock.installPackage).toHaveBeenCalledTimes(1);
197202
expect(esClient.esql.query).toHaveBeenCalledWith({
198-
query:
199-
'FROM logs-*,metrics-*,traces-* | KEEP @timestamp, data_stream.dataset | WHERE @timestamp > NOW() - 10 minutes | STATS COUNT(*) BY data_stream.dataset | LIMIT 100 | WHERE data_stream.dataset NOT IN ("system.test")',
203+
query: `FROM logs-*,metrics-*,traces-*
204+
| KEEP @timestamp, data_stream.dataset
205+
| WHERE @timestamp > NOW() - 10 minutes
206+
| STATS COUNT(*) BY data_stream.dataset | WHERE data_stream.dataset NOT IN ("system.test")`,
200207
});
201208
});
202209

‎x-pack/platform/plugins/shared/fleet/server/tasks/auto_install_content_packages_task.ts‎

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -28,9 +28,10 @@ import * as Registry from '../services/epm/registry';
2828

2929
import { MAX_CONCURRENT_EPM_PACKAGES_INSTALLATIONS, SO_SEARCH_LIMIT } from '../constants';
3030
import { getInstalledPackages } from '../services/epm/packages';
31+
import { getPrereleaseFromSettings } from '../services/epm/packages/get_prerelease_setting';
3132

3233
export const TYPE = 'fleet:auto-install-content-packages-task';
33-
export const VERSION = '1.0.0';
34+
export const VERSION = '1.0.1';
3435
const TITLE = 'Fleet Auto Install Content Packages Task';
3536
const SCOPE = ['fleet'];
3637
const DEFAULT_INTERVAL = '10m';
@@ -163,7 +164,7 @@ export class AutoInstallContentPackagesTask {
163164
this.logger.debug(
164165
`[AutoInstallContentPackagesTask] Fetching content packages to get discovery fields`
165166
);
166-
this.discoveryMap = await this.getContentPackagesDiscoveryMap();
167+
this.discoveryMap = await this.getContentPackagesDiscoveryMap(soClient);
167168
}
168169

169170
const installedPackages = await getInstalledPackages({
@@ -295,11 +296,11 @@ export class AutoInstallContentPackagesTask {
295296
.map((dataset) => `"${dataset}"`)
296297
.join(',')})`
297298
: '';
298-
const response = await esClient.esql.query({
299-
query: `FROM logs-*,metrics-*,traces-* | KEEP @timestamp, data_stream.dataset | WHERE @timestamp > NOW() - ${this.intervalToEsql(
300-
this.taskInterval
301-
)} | STATS COUNT(*) BY data_stream.dataset | LIMIT 100 ${whereClause}`,
302-
});
299+
const query = `FROM logs-*,metrics-*,traces-*
300+
| KEEP @timestamp, data_stream.dataset
301+
| WHERE @timestamp > NOW() - ${this.intervalToEsql(this.taskInterval)}
302+
| STATS COUNT(*) BY data_stream.dataset ${whereClause}`;
303+
const response = await esClient.esql.query({ query });
303304
this.logger.debug(`[AutoInstallContentPackagesTask] ESQL query took: ${response.took}ms`);
304305

305306
const datasetsWithData: string[] = response.values.map((value: any[]) => value[1]);
@@ -311,13 +312,15 @@ export class AutoInstallContentPackagesTask {
311312

312313
private intervalToEsql(interval: string): string {
313314
const value = parseInt(interval, 10);
314-
const unit = interval.includes('h') ? 'hours' : 'minutes';
315+
const unit = interval.includes('h') ? 'hours' : interval.includes('m') ? 'minutes' : 'seconds';
315316
return `${value} ${unit}`;
316317
}
317318

318-
private async getContentPackagesDiscoveryMap(): Promise<DiscoveryMap> {
319+
private async getContentPackagesDiscoveryMap(
320+
soClient: SavedObjectsClient
321+
): Promise<DiscoveryMap> {
319322
const type = 'content';
320-
const prerelease = false;
323+
const prerelease = await getPrereleaseFromSettings(soClient);
321324
const discoveryMap: DiscoveryMap = {};
322325
const registryItems = await Registry.fetchList({ prerelease, type });
323326

‎x-pack/platform/plugins/shared/fleet/server/types/rest_spec/epm.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -200,6 +200,7 @@ export const PackageInfoSchema = schema
200200
discovery: schema.maybe(
201201
schema.object({
202202
fields: schema.maybe(schema.arrayOf(schema.object({ name: schema.string() }))),
203+
datasets: schema.maybe(schema.arrayOf(schema.object({ name: schema.string() }))),
203204
})
204205
),
205206
})

0 commit comments

Comments
 (0)