[Ingest Manager] Match package spec `dataset`->`data_stream` and `config_templates`->`policy_templates` renaming #78699

jen-huang · 2020-09-28T22:12:50Z

Summary

Part of #74842. This PR supports the following renaming work done for packages:

Dataset to data stream package-spec#24
- datasets->data_streams property
- datasets.name->data_streams.dataset property
- /dataset->/data_stream directory
Rename config to policy package-spec#50
- config_templates->policy_templates property

Method names, types, etc were also renamed as needed. API integration fixtures (test packages) were updated to match the package spec changes too.

Important: These test suites were skipped fc6a2ba and should be re-enabled ASAP in a follow up PR as soon as a public docker image with the new package changes becomes available.

There is a private docker image that I used to validate these tests locally with, but that image is not accessible to Kibana CI. All skipped tests passed with that image, with the only exception of 3db32e2, an expected failure as Endpoint package did not have the changes yet.

Testing

Build local registry with new changes using instructions in PR Rename config to policy and dataset to datastream integrations#280 and comment Rename config to policy and dataset to datastream integrations#280 (comment)
Point Kibana to use your local registry
Comment out endpoint from default packages - Endpoint package is not available using the above registry instructions
Start Kibana, everything should work normally

…aming

… property renaming

…naming

…property renaming

…naming

…paths in test package manifests

…naming

This reverts commit 3db32e2.

elasticmachine · 2020-09-30T20:23:32Z

Pinging @elastic/ingest-management (Team:Ingest Management)

jonathan-buttner

Security solution changes look good 👍

jonathan-buttner · 2020-09-30T20:28:43Z

x-pack/test/security_solution_endpoint/apps/endpoint/index.ts

@@ -13,7 +13,7 @@ import {
 export default function (providerContext: FtrProviderContext) {
  const { loadTestFile, getService } = providerContext;

-  describe('endpoint', function () {
+  describe.skip('endpoint', function () {


Heads up @kevinlog we need to do this because of the breaking changes for dataset -> data_stream. We'll reenable as soon as the endpoint package with the same changes gets released: elastic/endpoint-package#89

no problem, I have PR right now where I'm re-enabling our other tests due to the one faulty package. #78954

I'm happy to to wait on it since this PR seems more complicated. I could also re-enable everything after the package releases.

FYI @nnamdifrankie @jen-huang

After speaking with @jen-huang offline, it's OK to merge this PR #78954. Tests will be disabled again with this PR, but we'll re-enable again after the next package.

jonathan-buttner · 2020-09-30T20:28:52Z

x-pack/test/security_solution_endpoint_api_int/apis/index.ts

@@ -10,7 +10,7 @@ import { getRegistryUrl as getRegistryUrlFromIngest } from '../../../plugins/ing
 export default function endpointAPIIntegrationTests(providerContext: FtrProviderContext) {
  const { loadTestFile, getService } = providerContext;

-  describe('Endpoint plugin', function () {
+  describe.skip('Endpoint plugin', function () {


jen-huang · 2020-09-30T20:33:16Z

x-pack/plugins/ingest_manager/dev_docs/indexing_strategy.md

@@ -6,48 +6,48 @@ Overall documentation of Ingest Management is now maintained in the `elastic/sta
 Ingest Management enforces an indexing strategy to allow the system to automatically detect indices and run queries on it. In short the indexing strategy looks as following:

 ```
-{dataset.type}-{dataset.name}-{dataset.namespace}
+{data_stream.type}-{data_stream.dataset}-{data_stream.namespace}


@ruflin could you look over the changes to this doc to make sure it's still accurate?

jen-huang · 2020-09-30T23:54:27Z

@elasticmachine merge upstream

kibanamachine · 2020-10-01T01:22:37Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: ed225af

Metrics [docs]

async chunks size

id	value	diff	baseline
`ingestManager`	1.1MB	+41.0B	1.1MB

page load bundle size

id	value	diff	baseline
`ingestManager`	509.8KB	+19.0B	509.8KB

History

💔 Build #78494 failed fc6a2ba
💚 Build #78100 succeeded a3f6450
💔 Build #77693 failed d42b9f3

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

neptunian · 2020-10-01T11:59:18Z

x-pack/plugins/ingest_manager/server/services/epm/registry/index.ts

-  // if it's a dataset
-  if (service === 'dataset') {
+  // if it's a data stream
+  if (service === 'data_stream') {
    // save the dataset name
    dataset = type;


shouldn't we be calling this dataStream now?

this should still actually be dataset as it refers to data_stream.dataset i.e. auth (as in system.auth)

skh

Thank you for the testing instructions! I tested this locally and everything works as described. 👍

(It would be great to get this merged soon, as I need to rebase #77986 on these changes.)

…fig_templates`->`policy_templates` renaming (elastic#78699) * Match elastic/package-spec#24 `datasets`->`data_streams` property renaming * Match elastic/package-spec#24 `datasets.name`->`data_streams.dataset` property renaming * Match elastic/package-spec#24 `/dataset`->`/data_stream` directory renaming * Match elastic/package-spec#50 `config_templates`->`policy_templates` property renaming * Update API integration test fixtures (test packages) * Temporarily skip API integration tests * Temporarily skip more API integration tests * Pin to custom docker image, unskip test suites, clean up broken icon paths in test package manifests * Skip the only (yay!) failing test suite * Revert "Skip the only (yay!) failing test suite" This reverts commit 3db32e2. * Re-skip tests and revert docker image Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

…fig_templates`->`policy_templates` renaming (#78699) (#79183) * Match elastic/package-spec#24 `datasets`->`data_streams` property renaming * Match elastic/package-spec#24 `datasets.name`->`data_streams.dataset` property renaming * Match elastic/package-spec#24 `/dataset`->`/data_stream` directory renaming * Match elastic/package-spec#50 `config_templates`->`policy_templates` property renaming * Update API integration test fixtures (test packages) * Temporarily skip API integration tests * Temporarily skip more API integration tests * Pin to custom docker image, unskip test suites, clean up broken icon paths in test package manifests * Skip the only (yay!) failing test suite * Revert "Skip the only (yay!) failing test suite" This reverts commit 3db32e2. * Re-skip tests and revert docker image Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

…aly-detection-partition-field * 'master' of github.com:elastic/kibana: (76 commits) Fix z-index of KQL Suggestions dropdown (elastic#79184) [babel] remove unused/unneeded babel plugins (elastic#79173) [Search] Fix timeout upgrade link (elastic#79045) Always Show Embeddable Panel Header in Edit Mode (elastic#79152) [Ingest]: add more test for transform index (elastic#79154) [ML] DF Analytics: Collapsable sections on results pages (elastic#76641) [Fleet] Fix agent policy change action migration (elastic#79046) [Ingest Manager] Match package spec `dataset`->`data_stream` and `config_templates`->`policy_templates` renaming (elastic#78699) Revert "[Metrics UI] Add ability to override datafeeds and job config for partition field (elastic#78875)" [ML] Update transform cloning to include description and new fields (elastic#78364) chore(NA): remove non existing plugin paths from case api integration tests (elastic#79127) [Ingest Manager] Ensure we trigger agent policy updated event when we bump revision. (elastic#78836) [Metrics UI] Display No Data context.values as [NO DATA] (elastic#78038) [Monitoring] Missing data alert (elastic#78208) [Lens] Fix embeddable title and description for reporting and dashboard tooltip (elastic#78767) [Lens] Consistent Drag and Drop styles (elastic#78674) [ML] Model management UI fixes and enhancements (elastic#79072) [Metrics UI] Add ability to override datafeeds and job config for partition field (elastic#78875) [Security Solution]Fix basepath used by endpoint telemetry tests (elastic#79027) update rum agent version which contains longtasks (elastic#79105) ...

jen-huang added 7 commits September 28, 2020 13:24

Match elastic/package-spec#24 datasets->data_streams property ren…

269bb16

…aming

Match elastic/package-spec#24 datasets.name->data_streams.dataset…

aa81a02

… property renaming

Match elastic/package-spec#24 /dataset->/data_stream directory re…

4c5e586

…naming

Match elastic/package-spec#50 config_templates->policy_templates …

e0bb46d

…property renaming

Update API integration test fixtures (test packages)

d6adad7

Temporarily skip API integration tests

6548109

Merge remote-tracking branch 'upstream/master' into ingest/package-re…

d42b9f3

…naming

jen-huang self-assigned this Sep 28, 2020

This was referenced Sep 29, 2020

[Ingest Manager] Follow up on data_stream and policy renaming #74842

Closed

[Ingest Manager] Install uploaded package #77986

Merged

jen-huang added 2 commits September 29, 2020 14:06

Temporarily skip more API integration tests

c478a20

Merge remote-tracking branch 'upstream/master' into ingest/package-re…

a3f6450

…naming

jonathan-buttner mentioned this pull request Sep 30, 2020

Rename changes and validation fixes elastic/endpoint-package#89

Merged

jen-huang added 5 commits September 30, 2020 12:40

Pin to custom docker image, unskip test suites, clean up broken icon …

e325111

…paths in test package manifests

Skip the only (yay!) failing test suite

3db32e2

Merge remote-tracking branch 'upstream/master' into ingest/package-re…

4362ec2

…naming

Revert "Skip the only (yay!) failing test suite"

c498cfc

This reverts commit 3db32e2.

Re-skip tests and revert docker image

fc6a2ba

jen-huang added release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.10.0 v8.0.0 labels Sep 30, 2020

jen-huang marked this pull request as ready for review September 30, 2020 20:23

jen-huang requested a review from a team as a code owner September 30, 2020 20:23

jen-huang requested a review from a team September 30, 2020 20:23

jen-huang requested a review from a team as a code owner September 30, 2020 20:23

jonathan-buttner approved these changes Sep 30, 2020

View reviewed changes

jen-huang commented Sep 30, 2020

View reviewed changes

Merge branch 'master' into ingest/package-renaming

ed225af

neptunian reviewed Oct 1, 2020

View reviewed changes

skh approved these changes Oct 1, 2020

View reviewed changes

neptunian approved these changes Oct 1, 2020

View reviewed changes

jen-huang merged commit 6d8f74a into elastic:master Oct 1, 2020

jen-huang deleted the ingest/package-renaming branch October 1, 2020 17:57

jen-huang mentioned this pull request Oct 1, 2020

[7.x] [Ingest Manager] Match package spec dataset->data_stream and config_templates->policy_templates renaming (#78699) #79183

Merged

jen-huang mentioned this pull request Oct 2, 2020

[Ingest Manager] Re-enable skipped tests #79242

Closed

jonathan-buttner mentioned this pull request Oct 2, 2020

[Security Solution][EPM] Reenabling the ingest and endpoint tests #79290

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ingest Manager] Match package spec `dataset`->`data_stream` and `config_templates`->`policy_templates` renaming #78699

[Ingest Manager] Match package spec `dataset`->`data_stream` and `config_templates`->`policy_templates` renaming #78699

jen-huang commented Sep 28, 2020 •

edited

Loading

elasticmachine commented Sep 30, 2020

jonathan-buttner left a comment

jonathan-buttner Sep 30, 2020

kevinlog Sep 30, 2020 •

edited

Loading

kevinlog Sep 30, 2020

jonathan-buttner Sep 30, 2020

jen-huang Sep 30, 2020

jen-huang commented Sep 30, 2020

kibanamachine commented Oct 1, 2020

neptunian Oct 1, 2020

jen-huang Oct 1, 2020

skh left a comment

[Ingest Manager] Match package spec dataset->data_stream and config_templates->policy_templates renaming #78699

[Ingest Manager] Match package spec dataset->data_stream and config_templates->policy_templates renaming #78699

Conversation

jen-huang commented Sep 28, 2020 • edited Loading

Summary

Testing

elasticmachine commented Sep 30, 2020

jonathan-buttner left a comment

Choose a reason for hiding this comment

jonathan-buttner Sep 30, 2020

Choose a reason for hiding this comment

kevinlog Sep 30, 2020 • edited Loading

Choose a reason for hiding this comment

kevinlog Sep 30, 2020

Choose a reason for hiding this comment

jonathan-buttner Sep 30, 2020

Choose a reason for hiding this comment

jen-huang Sep 30, 2020

Choose a reason for hiding this comment

jen-huang commented Sep 30, 2020

kibanamachine commented Oct 1, 2020

💚 Build Succeeded

Metrics [docs]

async chunks size

page load bundle size

History

neptunian Oct 1, 2020

Choose a reason for hiding this comment

jen-huang Oct 1, 2020

Choose a reason for hiding this comment

skh left a comment

Choose a reason for hiding this comment

[Ingest Manager] Match package spec `dataset`->`data_stream` and `config_templates`->`policy_templates` renaming #78699

[Ingest Manager] Match package spec `dataset`->`data_stream` and `config_templates`->`policy_templates` renaming #78699

jen-huang commented Sep 28, 2020 •

edited

Loading

kevinlog Sep 30, 2020 •

edited

Loading