Package containing transforms and not relying on data streams

[ajosh0504]

The package-spec documentation mentions that Transforms are supported in a Fleet package. The documentation also mentions that Elasticsearch assets can exist at the root level in a package, instead of residing strictly under the data_stream folder. However, it is unclear if this is actually supported.

Data streams are best suited for applications involving continuously generated data streaming into a common set of indices, but it's unclear how they could be leveraged for an application that involves data being written to indices that don't have anything in common.

I have a use case that consists of two chained transforms and two dashboards with drilldowns. The transforms don’t occur on logs data but instead on a custom index pattern. One of my transforms also needs an index with specific mappings created, before it can run. Is this possible using the current state of Fleet integrations?

cc: @mtojek

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[mtojek]

Let me pass this question to @ruflin . I'm not sure if all required blocks are present or simple to implement on the Kibana side.

[rw-access]

Should this be transferred to package-spec?
Also, I'm glad to help or drive the implementation if @ruflin is okay with it. Since it's more important to Security folks than it is to Fleet, so I'm expecting that we would end up doing most of the work, but with Fleet approval on PRs and such.

[mtojek]

Also, I'm glad to help or drive the implementation if @ruflin is okay with it. Since it's more important to Security folks than it is to Fleet, so I'm expecting that we would end up doing most of the work, but with Fleet approval on PRs and such.

Yes, that's probably what will execution look like.

Do you think you can prepare a package draft which contains just transforms?

The package-spec documentation mentions that Transforms are supported in a Fleet package. The documentation also mentions that Elasticsearch assets can exist at the root level in a package, instead of residing strictly under the data_stream folder. However, it is unclear if this is actually supported.

@ajosh0504 Could you please quote the sentence in the mentioned README? I couldn't find this paragraph (if it's there it's wrong). If you want to check what's allowed in the package (which files and where), please review this directory: https://github.com/elastic/package-spec/tree/master/versions/1

[ruflin]

Before we get started on the implementation of this I would like to understand a bit better the use case. What is this custom index pattern? Why is the data in this custom index pattern? Even though I agree on the top level we can support just any assets I would like to make sure that we use assets related to the data stream naming scheme whereever we can as it will keep assets more under control and easier to manage.

[ajosh0504]

@mtojek : Under Asset organization -> Supported assets:

For a quick overview, these are the assets typically found in an Elastic Package. The package spec will always contain the fully up-to-date list.

Elasticsearch
Ingest Pipeline
Index Template
Transform
Index template settings

Under Asset organization:

In contrast, any asset added on the top level will be picked up as json document, pushed to the corresponding Elasticsearch / Kibana APIs and used as is.

[mtojek]

@ruflin It seems that you introduced this paragraph in elastic/package-spec#170 . Do you think we should correct this paragraph? I agree with Apoorva that it introduces confusion. Spec doesn't allow for any Elasticsearch/Kibana definition that aren't bound to data streams.

[ajosh0504]

@ruflin As I mentioned, our application has two transforms. One of them works on indices consisting of alerts coming from detection rules, which are directed to hidden indices which look like .siem-security-default-000001, .siem-security-default-000002 and so on. However, the index pattern corresponding to these indices i.e. .siem-security-default-* is not automatically created. We also have dashboards that are based on the destination indices of the two transforms that I mention, which will also need index patterns to be created beforehand.

[ruflin]

@mtojek Yes, we should update this paragraph :-(
@ajosh0504 Do these indices have an alias in front? Is this the RAC indexing strategy? Do you expect these dashboards to be part of the packages?

I think we are jumping currently to conclusions on how to do it too quickly and should take a step back and ask how it "should" work. Happy to jump on a zoom call to discuss in more detail.

[ajosh0504]

@ruflin Not sure what you mean by "alias in front" or what the RAC indexing strategy is, but yes the dashboards need to be a part of the package. And I'm down to jump on a quick Zoom to show you what I'm looking for as well.

[ruflin]

I'm referring to elastic/kibana#102586 Is there an issue with the overall effort you are working on? If private link, feel free to share it with me on Slack.

[botelastic]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[szeitlin]

@ajosh0504 do we still need this ticket? It seems maybe out of date/captured elsewhere/partly done?

[ajosh0504]

I think this particular functionality i.e. transforms in packages and defining mappings for transform destination indices has been covered in this PR. Closing this issue.