Test LogsDB early with integrations

[ruflin]

The Elasticsearch team is working on the logsdb mode for data streams which will provide an optimised way to store logs. To provide the Elasticsearch team with early feedback if this causes any issues with integrations, we should find a way to enable this on integrations at least for test runs. More details on logsdb here: elastic/elasticsearch#106489

Like TSDS, logsdb is an index mode setting on the data stream. How can we inject this setting for a test run on all integrations without having to change each integration? At first, enabling logsdb might reject documents so this goes well together with the testing with failure store enabled: #1832

An alternative way of approaching this is using Fleet / EPM to enable the flag for all logs integrations @kpollich . For example a feature flag in Kibana. Last, it could also be a feature flag in Elasticsearch that enables it. The best place to enable it would be logs@settings. This would not only enable it for all integrations but also for all data going into logs-- which seems ideal for broader testing.

Below a screenshot on all the templates that an integration imports.

@salvatore-campagna Having written down the above option, to me the easiest way to test LogsDB seems to be having a feature flag / setting in Elasticsearch that changes logs@settings and adds the logsdb index mode. It is already possible today to start up elastic-package with specific settings for Elasticsearch so no modifications to elastic-package would be needed. In addition, it means we have a simple way to also test it in our test clusters.

Links

• Setting template in code: https://github.com/elastic/elasticsearch/blob/088f051a893dd9d9742e6023d1f62a3e28293dad/x-pack/plugin/core/template-resources/src/main/resources/logs%40settings.json#L4

[ruflin]

elastic/elasticsearch#108762 was just filed which should bring a feature flag to Elasticsearch to change the setting. This means, to test integrations, the only thing needed would be to pass in a config option on startup to Elasticsearch. @jsoriano Am I missing something?

[jsoriano]

elastic/elasticsearch#108762 was just filed which should bring a feature flag to Elasticsearch to change the setting. This means, to test integrations, the only thing needed would be to pass in a config option on startup to Elasticsearch. @jsoriano Am I missing something?

Yes, we could add a user setting to enable this, and add it in CI.

We will also need to add logsdb to the list of possible index modes in the package-spec.

[jsoriano]

We will also need to add logsdb to the list of possible index modes in the package-spec.

To be clear, I mean we would need this in the mid/long term for proper support in packages, but we won't need it to test early with integrations if the feature flag is added to ES.

[ruflin]

We will also need to add logsdb to the list of possible index modes in the package-spec.

Good callout. This is especially for the scenario if an integration would want to disable it. Question is if we should allow it? 🤔 Users could always to it in the @custom template.

[flash1293]

Users could always to it in the @custom template

@salvatore-campagna just to make sure, will it be possible to change the index mode via component template? I vaguely remember that was mentioned in a meeting we had.

[jsoriano]

We will also need to add logsdb to the list of possible index modes in the package-spec.

Good callout. This is especially for the scenario if an integration would want to disable it. Question is if we should allow it? 🤔 Users could always to it in the @custom template.

Well this would be useful if we migrate packages one by one as we did with tsdb. But if the plan is to make all logs datastreams to use this index mode at a given release, then we wouldn't need the flag.

[flash1293]

As subobjects: false is coupled to the logs index mode which is a breaking change by my understanding, I think we either need to make it a one-by-one package thing or explicitly set subobjects: true if the data stream didn't opt into subobjects: false.

[kpollich]

@jsoriano - I think if I understand correclt, once elastic/elasticsearch#108762 lands we can add a flag to start up ES with index.mode: logs set by default for all logs-* data streams and run system tests as normal to test integrations. I think this would satisfy the need for early testing/sanity checking. Do you agree with that?

Adding index.mode: logs as an option in package-spec + supporting it in Fleet would be the more long-term enhancement here that we could capture, separately.

[jsoriano]

Agree, we can support both things.

we can add a flag to start up ES with index.mode: logs set by default for all logs-* data streams and run system tests as normal to test integrations.

This should still be something that developers can enable locally. We would need to keep testing in CI with the default index mode for packages that don't have index.mode and users that don't use the feature flag.

[flash1293]

@kpollich @jsoriano Seems like elastic/elasticsearch#108762 just landed in ES, so it should be possible to run this test now.

[ruflin]

For reference, this is the PR with the change and description around the flag to be used: elastic/elasticsearch#109025

[ruflin]

To start test logsdb on my local setup, I modified elastic-package in the following way: main...ruflin:elastic-package:logsdb-setting It adds cluster.logsdb.enabled: true to the Elasticsearch config. @jsoriano is there a way to pass this in as environment variable so no code change would be needed?

[jsoriano]

@jsoriano is there a way to pass this in as environment variable so no code change would be needed?

No, we need to add this to the config embedded in elastic-package, but I think it would be fine to add a profile setting for this.

[ruflin]

Can profile settings be set through an environment variable? The reason I'm asking is that I want to make it for users not familiar with elastic-package to have a single command I can share for them to get a stack running with logsdb.

[jsoriano]

Can profile settings be set through an environment variable? The reason I'm asking is that I want to make it for users not familiar with elastic-package to have a single command I can share for them to get a stack running with logsdb.

They cannot be set through an environment variable at the moment, but they can be set with flags, so a single command is still possible. When added, it would be something like:

elastic-package stack up -d -U stack.logsdb_enabled=true

[ruflin]

elastic-package stack up -d -U stack.logsdb_enabled=true This looks great!