Dataset to data stream #24

ycombinator · 2020-08-13T17:10:37Z

This PR updates the spec to rename dataset references in packages to data_stream.

Concretely, the following elements of packages are changed:

In the package's root manifest.yml file, the datasets property must now be called data_streams instead.
In the package's root folder, the dataset folder must now be called data_stream instead.

Resolves #18.

elasticmachine · 2020-08-13T17:14:10Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #24 updated]
Start Time: 2020-09-28T22:41:29.517+0000
Duration: 2 min 9 sec

versions/1/spec.yml

ruflin

One additional conversation we can have here is about our recommendation on the names of these data stream template directories. So far what we used is the second part of the dataset name that combined with the package it results in the dataset part. This applies in most cases but not all of them, that is why we allow to overwrite it.

I personally still like the naming we have as it makes browsing easy but I thought worth bringing up. In the end I don't expect this to change the spec, only our recommendation on naming.

versions/1/spec.yml

versions/1/data_stream/manifest.spec.yml

ruflin · 2020-08-14T07:47:09Z

versions/1/data_stream/spec.yml

@@ -1,10 +1,10 @@
 spec:
- description: Folder containing a single dataset definition
+- description: Folder containing a single data stream definition


Stumbled over this as it is not fully current. This a template for all data streams created for a specific type + dataset combination. All logs-nginx.access-* will use this template, independent of the namespace.

In my head, these are data stream templates.

I'm afraid I'm lost here. @ruflin, could you please elaborate more about this? data streams, templates, namespaces

Yes, I'm lost too. I think it might be useful to add a definitions.md or just a README.md file in the versions/1 folder defining all these terms and concepts and how they relate to each other. The spec (so far) describes the syntactic relationships but I also have to admit that I don't fully understand all the semantic relationships between these concepts.

@ruflin can you please make a new PR to add such a file? I think you are the only(?) one who understands all these terms (old and new), the corresponding concepts, and the relationships between them.

ruflin

@mtojek @ycombinator Before we get this change in, I would like to answer all the questions around dataset / data_stream to make sure we don't introduce breaking changes to the spec before we are 100% confident these are the right changes. I'll set up a call.

ycombinator · 2020-09-15T13:50:21Z

@ruflin @mtojek I'm resurrecting this PR after a while. I've rebased it on master so it has all the other changes now. Could you please re-review when you get a chance? Thanks!

mtojek · 2020-09-15T13:58:23Z

Ok, so once you push this change to master, we'll block introducing next changes to this repository until the package-registry and all integrations are adjusted?

ycombinator · 2020-09-15T14:01:46Z

Ok, so once you push this change to master, we'll block introducing next changes to this repository until the package-registry and all integrations are adjusted?

Yes. But to make the transition quick, I would say we should try to get the PRs ready in package-registry and integrations repos along side this one so we can merge all three at around the same time. We can also merge this PR into master and update elastic-package, but then make a PR to integrations updating elastic-package and making the necessary adjustments to the packages.

mtojek · 2020-09-15T14:07:52Z

I think we can safely push this to master. Then:

Update dependency on the elastic-package.
Adjust package-registry (not sure if this is safe to push changes, @ruflin ?)
Adjust integrations.

Points 1. and 2. can go in parallel.

I'm not sure about the package-storage, but I bet this one should go as the last one?

mtojek · 2020-09-15T22:20:41Z

Actually the same question goes to Kibana. Are we safe to introduce such renaming or is it better to schedule it? @ruflin

ruflin · 2020-09-16T07:21:48Z

We should schedule this change together with Kibana. For my two recent breaking changes to the registry @jfsiii has worked some magic that both version work which is also a best case scenario here. If we can do that, it means Kibana merges + Backports this first, then we get it in and roll it out to all of the parts mentioned above. Can you sync up with @jfsiii on this?

If it becomes complicated on the Kibana side to have it backward compatible, we do a coordinated merge which might break some things for 24h which depend on snapshots.

ycombinator · 2020-09-17T09:31:08Z

I'm unclear as to why this PR needs to be scheduled with a Kibana PR. This PR just changes the spec. It does not change any packages. So I'm not understanding the impact of this PR on Kibana. What am I missing?

ruflin · 2020-09-18T06:41:34Z

@ycombinator If we get this PR in without the changes ready on the Kibana side, it means if we have other changes to the package-spec and want to roll it out, we can't as it will break things. So just merging it will not have a direct impact on Kibana but it will hold development back on other things in the spec.

jfsiii · 2020-09-18T10:03:45Z

@ruflin I'm not totally clear on the how the spec changes map to Registry & Kibana code/type changes but am happy to sync up with @mtojek. @jen-huang did a lot of dataset.* to data_stream.* conversion in elastic/kibana@042254f

We do reference datasets in our RegsitryPackage type which mirrors the Registry's Package.Datasets

ruflin · 2020-09-21T06:40:20Z

@jfsiii @ycombinator Could you sync up to push this forward? Please pull me in if needed. Please directly also discuss #19 to make sure everyone is aware of all the changes coming.

mtojek · 2020-09-28T08:47:39Z

@ycombinator I'm not sure why the CI didn't report problems with test directories:

https://github.com/ycombinator/package-spec/tree/dataset-to-data_stream/code/go/internal/validator/test/packages/bad_deploy_variants
https://github.com/ycombinator/package-spec/tree/dataset-to-data_stream/code/go/internal/validator/test/packages/good

There are still "dataset" folders and it failed for me in the local environment.

EDIT:

You can also rebase the PR.

versions/1/data_stream/manifest.spec.yml

…aming

… property renaming

…naming

ycombinator · 2020-09-28T22:39:18Z

@mtojek Looks like CI just failed as expected: https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fpackage-spec/detail/PR-24/6/pipeline. Yay!

Fixing...

…fig_templates`->`policy_templates` renaming (#78699) * Match elastic/package-spec#24 `datasets`->`data_streams` property renaming * Match elastic/package-spec#24 `datasets.name`->`data_streams.dataset` property renaming * Match elastic/package-spec#24 `/dataset`->`/data_stream` directory renaming * Match elastic/package-spec#50 `config_templates`->`policy_templates` property renaming * Update API integration test fixtures (test packages) * Temporarily skip API integration tests * Temporarily skip more API integration tests * Pin to custom docker image, unskip test suites, clean up broken icon paths in test package manifests * Skip the only (yay!) failing test suite * Revert "Skip the only (yay!) failing test suite" This reverts commit 3db32e2. * Re-skip tests and revert docker image Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

…fig_templates`->`policy_templates` renaming (elastic#78699) * Match elastic/package-spec#24 `datasets`->`data_streams` property renaming * Match elastic/package-spec#24 `datasets.name`->`data_streams.dataset` property renaming * Match elastic/package-spec#24 `/dataset`->`/data_stream` directory renaming * Match elastic/package-spec#50 `config_templates`->`policy_templates` property renaming * Update API integration test fixtures (test packages) * Temporarily skip API integration tests * Temporarily skip more API integration tests * Pin to custom docker image, unskip test suites, clean up broken icon paths in test package manifests * Skip the only (yay!) failing test suite * Revert "Skip the only (yay!) failing test suite" This reverts commit 3db32e2. * Re-skip tests and revert docker image Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

…fig_templates`->`policy_templates` renaming (#78699) (#79183) * Match elastic/package-spec#24 `datasets`->`data_streams` property renaming * Match elastic/package-spec#24 `datasets.name`->`data_streams.dataset` property renaming * Match elastic/package-spec#24 `/dataset`->`/data_stream` directory renaming * Match elastic/package-spec#50 `config_templates`->`policy_templates` property renaming * Update API integration test fixtures (test packages) * Temporarily skip API integration tests * Temporarily skip more API integration tests * Pin to custom docker image, unskip test suites, clean up broken icon paths in test package manifests * Skip the only (yay!) failing test suite * Revert "Skip the only (yay!) failing test suite" This reverts commit 3db32e2. * Re-skip tests and revert docker image Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

ycombinator requested review from ruflin and mtojek August 13, 2020 17:13

ycombinator mentioned this pull request Aug 13, 2020

[Discuss] Impact of renaming dataset to data_stream #18

Closed

ycombinator commented Aug 13, 2020

View reviewed changes

versions/1/spec.yml Outdated Show resolved Hide resolved

ruflin reviewed Aug 14, 2020

View reviewed changes

mtojek approved these changes Aug 14, 2020

View reviewed changes

ruflin suggested changes Aug 17, 2020

View reviewed changes

ycombinator force-pushed the dataset-to-data_stream branch from c48677d to 72503d2 Compare September 15, 2020 13:49

ycombinator requested a review from ruflin September 15, 2020 13:50

mtojek mentioned this pull request Sep 28, 2020

Rename config to policy #50

Merged

mtojek reviewed Sep 28, 2020

View reviewed changes

versions/1/data_stream/manifest.spec.yml Outdated Show resolved Hide resolved

ycombinator added 4 commits September 28, 2020 08:49

Rename dataset => data_stream / data stream

0fa641e

Renaming spec folder

ee07863

Changing name => dataset

0b9e78f

Rebase changes

f706b65

ycombinator force-pushed the dataset-to-data_stream branch from 72503d2 to f706b65 Compare September 28, 2020 12:49

jen-huang added a commit to jen-huang/kibana that referenced this pull request Sep 28, 2020

Match elastic/package-spec#24 datasets->data_streams property ren…

269bb16

…aming

jen-huang added a commit to jen-huang/kibana that referenced this pull request Sep 28, 2020

Match elastic/package-spec#24 datasets.name->data_streams.dataset…

aa81a02

… property renaming

jen-huang added a commit to jen-huang/kibana that referenced this pull request Sep 28, 2020

Match elastic/package-spec#24 /dataset->/data_stream directory re…

4c5e586

…naming

Fixing field name

37fd891

Renaming dataset folders to data_stream

948d511

ycombinator requested review from jen-huang and mtojek September 28, 2020 22:46

jen-huang mentioned this pull request Sep 29, 2020

[Ingest Manager] Match package spec dataset->data_stream and config_templates->policy_templates renaming elastic/kibana#78699

Merged

mtojek approved these changes Sep 30, 2020

View reviewed changes

ycombinator merged commit 734e3cc into elastic:master Sep 30, 2020

ycombinator deleted the dataset-to-data_stream branch September 30, 2020 11:05

jonathan-buttner mentioned this pull request Sep 30, 2020

Rename changes and validation fixes elastic/endpoint-package#89

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dataset to data stream #24

Dataset to data stream #24

ycombinator commented Aug 13, 2020 •

edited

Loading

elasticmachine commented Aug 13, 2020 •

edited

Loading

Build stats

ruflin left a comment

ruflin Aug 14, 2020

mtojek Aug 14, 2020

ycombinator Aug 14, 2020 •

edited

Loading

ruflin left a comment

ycombinator commented Sep 15, 2020

mtojek commented Sep 15, 2020

ycombinator commented Sep 15, 2020

mtojek commented Sep 15, 2020 •

edited

Loading

mtojek commented Sep 15, 2020

ruflin commented Sep 16, 2020 •

edited

Loading

ycombinator commented Sep 17, 2020

ruflin commented Sep 18, 2020

jfsiii commented Sep 18, 2020 •

edited

Loading

ruflin commented Sep 21, 2020

mtojek commented Sep 28, 2020 •

edited

Loading

ycombinator commented Sep 28, 2020

Dataset to data stream #24

Dataset to data stream #24

Conversation

ycombinator commented Aug 13, 2020 • edited Loading

elasticmachine commented Aug 13, 2020 • edited Loading

💚 Build Succeeded

Build stats

ruflin left a comment

Choose a reason for hiding this comment

ruflin Aug 14, 2020

Choose a reason for hiding this comment

mtojek Aug 14, 2020

Choose a reason for hiding this comment

ycombinator Aug 14, 2020 • edited Loading

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

ycombinator commented Sep 15, 2020

mtojek commented Sep 15, 2020

ycombinator commented Sep 15, 2020

mtojek commented Sep 15, 2020 • edited Loading

mtojek commented Sep 15, 2020

ruflin commented Sep 16, 2020 • edited Loading

ycombinator commented Sep 17, 2020

ruflin commented Sep 18, 2020

jfsiii commented Sep 18, 2020 • edited Loading

ruflin commented Sep 21, 2020

mtojek commented Sep 28, 2020 • edited Loading

ycombinator commented Sep 28, 2020

ycombinator commented Aug 13, 2020 •

edited

Loading

elasticmachine commented Aug 13, 2020 •

edited

Loading

ycombinator Aug 14, 2020 •

edited

Loading

mtojek commented Sep 15, 2020 •

edited

Loading

ruflin commented Sep 16, 2020 •

edited

Loading

jfsiii commented Sep 18, 2020 •

edited

Loading

mtojek commented Sep 28, 2020 •

edited

Loading