Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial definition of content pack[age]s #777

Merged
merged 11 commits into from
Aug 27, 2024

Conversation

jsoriano
Copy link
Member

@jsoriano jsoriano commented Jul 19, 2024

What does this PR do?

Add the initial definition for content packages.

On this initial version, they only support the distribution of kibana assets, like dashboards.

This package type introduces a discovery mechanism based on the data the package can work with. For example the following definition would indicate that the package provides resources that are useful when used with data that includes the field process.pid:

discovery:
  fields:
    - name: "process.pid"

The package registry could add support to look for packages that have support for certain fields. For example a request like the following one could be used to discover packages that include resources that can leverage the process.pid or the user.id fields.

GET /search?discovery=fields:process.pid,user.id

Why is it important?

There are some use cases where developers want to distribute resources that can be useful with some data, without coupling them to the mechanism used to collect this data.
These resources could be for example dashboards that can be used with data collected with other integrations or even with other agents as the OTEL collector.

Paves the foundations for the use cases described in #351.

Checklist

Related issues

@jsoriano jsoriano self-assigned this Jul 19, 2024
@jsoriano jsoriano force-pushed the content-pack-ages branch from 6114b36 to 6900576 Compare July 19, 2024 14:44
@jsoriano jsoriano force-pushed the content-pack-ages branch from 6900576 to c6687d6 Compare July 19, 2024 14:45
@kpollich kpollich self-requested a review July 23, 2024 12:05
type: object
additionalProperties: false
definitions:
interested_on:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a tricky thing to name. What do you think about discovery_fields instead, in the interest of being more explicit about the behavior associated with this property?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I am not super happy with the name, so thanks for the suggestion!

I wouldn't include fields at this level because maybe we can support other things, like other packages, but discovery could fit.

discovery:
  fields:
    - name: "process.pid"
discovery:
  packages:
    - name: "system"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like leaving room for driving recommendations for other packages vs fields - I think this approach is solid.

@@ -23,25 +23,33 @@ type Filter struct {
type FilterResult struct {
Processed error
Removed error

UnusedProcessors []Processor
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added detection of unused rules, used by now only on tests, we could warn in the future about this.

@jsoriano jsoriano marked this pull request as ready for review August 26, 2024 15:01
@jsoriano jsoriano requested a review from a team as a code owner August 26, 2024 15:01
@jsoriano
Copy link
Member Author

Opening for review so we can start testing it.

Copy link
Contributor

@mrodm mrodm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
Added some questions about the limits for this new type of package

spec/content/spec.yml Outdated Show resolved Hide resolved
Comment on lines 10 to 11
totalSizeLimit: 50MB
sizeLimit: 30MB
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these limits related to the whole package ? As these are going to be mainly assets, probably we could set the same limits as the integration packages. WDYT ?

  totalSizeLimit: 250MB
  sizeLimit: 150MB

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, in any case we will revisit this, as content packages are intended to support bigger sizes than other packages.

@elasticmachine
Copy link

💚 Build Succeeded

History

cc @jsoriano

Copy link
Contributor

@mrodm mrodm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@jsoriano jsoriano merged commit 66ecd77 into elastic:main Aug 27, 2024
3 checks passed
@jsoriano jsoriano deleted the content-pack-ages branch August 27, 2024 09:41
jen-huang added a commit to elastic/kibana that referenced this pull request Oct 9, 2024
## Summary

Related to #192484. This PR adding [new content package types and
schemas](elastic/package-spec#777) so that
content packages can be returned correctly from EPR to unblock
development of those packages.

The only current content package is `kubernetes_otel`. You will need to
bump up the max allowed spec version and search with beta (prerelease)
packages enabled to find it:
```
xpack.fleet.internal.registry.spec.max: '3.4'
```

Tests will come with the rest of work for #192484

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
jen-huang added a commit to jen-huang/kibana that referenced this pull request Oct 9, 2024
## Summary

Related to elastic#192484. This PR adding [new content package types and
schemas](elastic/package-spec#777) so that
content packages can be returned correctly from EPR to unblock
development of those packages.

The only current content package is `kubernetes_otel`. You will need to
bump up the max allowed spec version and search with beta (prerelease)
packages enabled to find it:
```
xpack.fleet.internal.registry.spec.max: '3.4'
```

Tests will come with the rest of work for elastic#192484

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit bd6533f)

# Conflicts:
#	oas_docs/bundle.json
#	oas_docs/bundle.serverless.json
#	oas_docs/output/kibana.serverless.staging.yaml
#	oas_docs/output/kibana.serverless.yaml
#	oas_docs/output/kibana.staging.yaml
#	oas_docs/output/kibana.yaml
#	x-pack/plugins/fleet/server/types/rest_spec/epm.ts
jen-huang added a commit to elastic/kibana that referenced this pull request Oct 10, 2024
…195690)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[UII] Add types to return content packages correctly
(#195505)](#195505)

Note that this includes manual changes to openapi spec files. This
change is not present on the original PR as the main branch no longer
uses openapi spec files.

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Jen
Huang","email":"its.jenetic@gmail.com"},"sourceCommit":{"committedDate":"2024-10-09T20:08:18Z","message":"[UII]
Add types to return content packages correctly (#195505)\n\n##
Summary\r\n\r\nRelated to #192484. This PR adding [new content package
types and\r\nschemas](elastic/package-spec#777)
so that\r\ncontent packages can be returned correctly from EPR to
unblock\r\ndevelopment of those packages.\r\n\r\nThe only current
content package is `kubernetes_otel`. You will need to\r\nbump up the
max allowed spec version and search with beta (prerelease)\r\npackages
enabled to find it:\r\n```\r\nxpack.fleet.internal.registry.spec.max:
'3.4'\r\n```\r\n\r\nTests will come with the rest of work for
#192484\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"bd6533f30b58fc831670d400f25a61321379902c","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Fleet","v9.0.0","backport:prev-minor"],"number":195505,"url":"https://github.com/elastic/kibana/pull/195505","mergeCommit":{"message":"[UII]
Add types to return content packages correctly (#195505)\n\n##
Summary\r\n\r\nRelated to #192484. This PR adding [new content package
types and\r\nschemas](elastic/package-spec#777)
so that\r\ncontent packages can be returned correctly from EPR to
unblock\r\ndevelopment of those packages.\r\n\r\nThe only current
content package is `kubernetes_otel`. You will need to\r\nbump up the
max allowed spec version and search with beta (prerelease)\r\npackages
enabled to find it:\r\n```\r\nxpack.fleet.internal.registry.spec.max:
'3.4'\r\n```\r\n\r\nTests will come with the rest of work for
#192484\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"bd6533f30b58fc831670d400f25a61321379902c"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/195505","number":195505,"mergeCommit":{"message":"[UII]
Add types to return content packages correctly (#195505)\n\n##
Summary\r\n\r\nRelated to #192484. This PR adding [new content package
types and\r\nschemas](elastic/package-spec#777)
so that\r\ncontent packages can be returned correctly from EPR to
unblock\r\ndevelopment of those packages.\r\n\r\nThe only current
content package is `kubernetes_otel`. You will need to\r\nbump up the
max allowed spec version and search with beta (prerelease)\r\npackages
enabled to find it:\r\n```\r\nxpack.fleet.internal.registry.spec.max:
'3.4'\r\n```\r\n\r\nTests will come with the rest of work for
#192484\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"bd6533f30b58fc831670d400f25a61321379902c"}}]}]
BACKPORT-->

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants