Add support for runtime fields in packages #39

ruflin · 2020-08-26T13:18:28Z

Soon Elasticsearch is planned to support runtime fields: elastic/elasticsearch#59332 I assume these can not be specified currently in our fields.yml. We should add support for these fields in fields.yml and on the Kibana side be able to create the correct Elasticsearch templates and Kibana index patterns.

ruflin · 2020-08-26T13:19:33Z

@ycombinator for the spec
@skh @neptunian For awareness for the template / index pattern creation
@ph For awareness
@andresrc We should figure out if and where we want to use this in packages in the future.

* Fix: create missing build directory * Fix

ruflin · 2021-05-31T11:35:39Z

Adding to this, Elasticsearch supports to set the dynamic: runtime: https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html A package dev should be able to select this option as the default for dynamic fields. APM asked for this in the context of the apm package defaults (@simitt ).

simitt · 2021-07-06T08:15:50Z

@elastic/apm-server feel free to pick this up for 7.15

ruflin · 2022-12-23T08:01:34Z

We would really like to see this prioritised. The way I would propose this to be implemented is as following:

    - name: foo.bar
      type: long
      runtime: true

This is slightly different from the way it looks in Elasticsearch where a separate block for runtime fields has to be specified. Having it directly as a boolean in the fields definition allows a very simple way to enable / disable it for devs. Fleet still has to create the additional block.

amitkanfer · 2022-12-23T09:06:25Z

@jlind23 putting this on your radar.

joshdover · 2023-01-11T16:38:59Z

A couple things to consider here. Runtime fields is mostly optimized for extracting new fields out of other indexed fields that are available in the "doc values" and the runtime field's script has to be written differently depend on whether or not the field that we're extracting from is being indexed. So there's really 2 use cases here:

Runtime fields that have supply a custom script to extract data from another indexed field
Runtime fields that simply populate a value from _source that is not indexed
- This is the use case that is covered by @ruflin's comment in Add support for runtime fields in packages #39 (comment)

When Fleet creates the index templates for runtime fields, the script field for a runtime field will need to differ based on whether or not the field is indexed. If index: false is specified or the field is not mapped at all, the field will not be available in doc['foo'] but will be available in params._source['foo'].

We can probably support a design like this for both use cases:

- name: foo.bar
  type: long
  runtime: true
- name: foo.baz
  type: long
  runtime: |
    doc['message'].value().doSomething()

I think the implementation could work something like this:

If runtime is a boolean true, then create a runtime script with params._source['<field>']
If runtime is a string, just use that for the script directly

mrodm · 2023-04-18T16:15:09Z

The approach mentioned by @joshdover in #39 (comment) LGTM

For adding this new field, according to the docs it should be checked that runtime is just added/defined when some of the types mentioned there are used for that specific field:

NOTE: currently, package-spec does not support all the types mentioned there for field types.
Those checks look like could be done using JSON schema conditionals like in this PR (Add support for aggregate_metric_double field type #500)

I'll try to create a proposal PR in package-spec to include that field.

Fleet should also add support to create those runtime fields, @kpollich do you foresee any issue having that field being boolean or string for fleet?

In elastic-package there could be some improvements to be added:

Add into the generated READMEs some indication that the field is a Runtime field (example apache docs) ?
- README files should not be changed if there are no runtime fields.
To be checked once the fleet support is added to include runtime fields when the package is installed, whether or not system tests need to be updated to give special support for runtime fields (could it be checked previously @jsoriano ?)

jsoriano · 2023-04-19T09:10:02Z

Plan sounds good to me too.

If runtime is a boolean true, then create a runtime script with params._source['<field>']

This seems to be already the behaviour of Elasticsearch when no script is set (see here). In principle it seems enough to add the field as runtime field, without any script, and it does the equivalent to params._source['<field>'].

Fleet should also add support to create those runtime fields

@mrodm could you please check if there is already an open issue for that, and create one otherwise?

do you foresee any issue having that field being boolean or string for fleet?

It seems possible to have variables with multiple types in typescript, I think this will need more special handling in Go if we need to do something with this field 🙂

could it be checked previously @jsoriano ?

Maybe we can try to remove a field in a package, and add it as a runtime field in the index_template configuration of its manifest. Then check what system tests do, and what sample documents they generate.

kpollich · 2023-04-19T14:39:20Z

Fleet should also add support to create those runtime fields, @kpollich do you foresee any issue having that field being boolean or string for fleet?

I don't foresee this being an issue on Fleet's end. Just a small piece of logic for how the field is used when generating mappings.

mrodm · 2023-04-21T08:05:48Z

Maybe we can try to remove a field in a package, and add it as a runtime field in the index_template configuration of its manifest. Then check what system tests do, and what sample documents they generate.

I've tested this approach, and what I observed is that runtime fields are just present in the fields in the documents instead of _source. As it is indicated here:

Use the fields parameter on the _search API to retrieve the values of runtime fields. Runtime fields won’t display in _source, but the fields API works for all fields, even those that were not sent as part of the original _source.

In that case, elastic-package should also take into account the fields from the documents retrieved from Elasticsearch to validate the fields defined in the package.

Created issue to support runtime fields in elastic-package elastic/elastic-package#1229

Also tested that if nowadays a field with the runtime key is added into a package, this is not going to fail and it will just add a new mapping with the given type. Because of that, it is also needed to add a new validation rule in package-spec to ensure a minimum Kibana version (8.8 ? 8.9?)

jen-huang · 2023-04-21T17:34:06Z

Thanks @mrodm for the work here so far. I'm going to move this issue to Sprint 12 but of course it could get closed out by #505 if that merges soon. Follow up issues elastic/elastic-package#1229, elastic/kibana#155255) will be moved to Sprint 12 as well.

jlind23 · 2023-05-05T05:06:59Z

@mrodm @jsoriano what is missing here as we have merged #505 ?

mrodm · 2023-05-05T08:28:37Z

@mrodm @jsoriano what is missing here as we have merged #505 ?

The support for package-spec has been added through that PR, but runtime fields need to be supported also in Kibana/Fleet. There is another issue for that elastic/kibana#155255. According to the comments in the PR, it should be for 8.9.0 https://github.com/elastic/package-spec/pull/505/files#r1173839069

Should we close this one and just let the other issue in kibana open ? @jlind23

ruflin · 2023-05-05T09:22:41Z

We should not close this issue. This is about adding support fo runtime fields in packages. As long as Fleet does not support it, it is from my perspective not supported in packages even though users could add it.

jlind23 · 2023-05-05T11:28:07Z

@mrodm moving this issue out of our current sprint as they are is nothing to do here from a package-spec perspective. We will close this one later as ruflin stated as soon as elastic/kibana#155255 is fixed.

@timestamp

## Summary Closes #155255 Closes elastic/package-spec#39 Add support in Fleet for Runtime fields, based on these docs: - Defining runtime fields: - https://www.elastic.co/guide/en/elasticsearch/reference/8.8/runtime-mapping-fields.html - https://www.elastic.co/guide/en/elasticsearch/reference/8.8/runtime-retrieving-fields.html - Mapping runtime fields in dynamic templates: - https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html#dynamic-mapping-runtime-fields - Adding runtime fields under groups Given these field definitions in packages: ```yaml - name: bar type: boolean - name: uptime type: keyword - name: runtime_boolean type: boolean runtime: true - name: runtime.day type: keyword runtime: >- emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT)) - name: to_be_long type: long runtime: true - name: runtime.date type: date date_format: 'yyyy-MM-dd' runtime: >- emit(doc['@timestamp'].value.toEpochMilli()) - name: runtime.epoch_milli type: long runtime: >- emit(doc['@timestamp'].value.toEpochMilli()) - name: lowercase type: keyword runtime: >- emit(doc['uppercase'].value.toLowerCase()) - name: labels.* type: long object_type_mapping_type: double runtime: true - name: responses type: group fields: - name: runtime_group_boolean type: boolean runtime: true - name: foo type: boolean ``` and this definition in the manifest ```yaml elasticsearch: index_template: mappings: runtime: day_of_week_two: type: keyword script: source: "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))" ``` This PR adds the required fields into the `mappings` key when installing the package. For this example, the resulting mappings are (just showing the relevant data for these changes): ```json { ".ds-logs-runtime_fields.foo-default-2023.07.10-000001": { "mappings": { "dynamic_templates": [ { "labels.*": { "path_match": "labels.*", "match_mapping_type": "double", "runtime": { "type": "long" } } } ], "runtime": { "day_of_week_two": { "type": "keyword", "script": { "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))", "lang": "painless" } }, "labels.a": { "type": "long" }, "labels.b": { "type": "long" }, "lowercase": { "type": "keyword", "script": { "source": "emit(doc['uppercase'].value.toLowerCase())", "lang": "painless" } }, "responses.runtime_group_boolean": { "type": "boolean" }, "runtime.date": { "type": "date", "script": { "source": "emit(doc['@timestamp'].value.toEpochMilli())", "lang": "painless" }, "format": "yyyy-MM-dd" }, "runtime.day": { "type": "keyword", "script": { "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))", "lang": "painless" } }, "runtime.epoch_milli": { "type": "long", "script": { "source": "emit(doc['@timestamp'].value.toEpochMilli())", "lang": "painless" } }, "runtime_boolean": { "type": "boolean" }, "to_be_long": { "type": "long" } }, "properties": { "@timestamp": { "type": "date", "ignore_malformed": false }, "bar": { "type": "boolean" }, "data_stream": { "properties": { "dataset": { "type": "constant_keyword" }, "namespace": { "type": "constant_keyword" }, "type": { "type": "constant_keyword" } } }, "labels": { "type": "object" }, "message": { "type": "keyword", "ignore_above": 1024 }, "responses": { "properties": { "foo": { "type": "boolean" } } }, "uppercase": { "type": "keyword", "ignore_above": 1024 }, "user": { "properties": { "id": { "type": "keyword", "ignore_above": 1024 } } } } } } } ``` Tested manually installing a package containing runtime field definitions as the example above. Tested also indexing some documents and retrieving the runtime fields: - Indexing documents: ```json POST /logs-runtime_fields.foo-default/_doc/ { "@timestamp": "2023-07-07T13:32:09.000Z", "datastream": { "dataset": "logs-runtime_fields.foo", "namespace": "default", "type": "logs" }, "user": { "id": "8a4f500d" }, "message": "Login successful", "labels": { "a": 1.6, "b": 2.5 }, "uppercase": "SOMETHING", "to_be_long": 1.6, "runtime_boolean": true, "responses.runtime_group_boolean": false } ``` - Retrieving runtime fields (`_source` disabled): ```json GET logs-runtime_fields.foo-default/_search { "fields": [ "@timestamp", "runtime_boolean", "responses.runtime_group_boolean", "runtime.day", "runtime.date", "runtime.epoch_milli", "labels.*", "uppercase", "lowercase", "to_be_long" ], "_source": false } ``` - Output: ```json ... "hits": [ { "_index": ".ds-logs-runtime_fields.foo-default-2023.07.10-000001", "_id": "_7p1P4kBtEvrlGnsxiFN", "_score": 1, "fields": { "uppercase": [ "SOMETHING" ], "runtime.date": [ "2023-07-10" ], "@timestamp": [ "2023-07-10T09:33:09.000Z" ], "lowercase": [ "something" ], "to_be_long": [ 1 ], "runtime_boolean": [ true ], "runtime.day": [ "Monday" ], "labels.a": [ 1 ], "labels.b": [ 2 ], "responses.runtime_group_boolean": [ false ], "runtime.epoch_milli": [ 1688981589000 ] } } ] ... ``` ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

rw-access pushed a commit to rw-access/package-spec that referenced this issue Mar 23, 2021

Fix: create build dir if it's not present (elastic#39)

d4609f6

* Fix: create missing build directory * Fix

ruflin added the Team:Integrations Label for the Integrations team label May 31, 2021

ruflin mentioned this issue May 31, 2021

[Discuss] Default mappings for dynamic fields #178

Open

nchaulet mentioned this issue Aug 17, 2021

[Fleet] Update Elastic Agent dashboard, package icon, and data stream mappings elastic/integrations#1529

Merged

jsoriano mentioned this issue Mar 18, 2022

Discuss strategies to reduce number of fields elastic/integrations#2839

Closed

jlind23 added the enhancement New feature or request label May 24, 2022

ruflin mentioned this issue Dec 23, 2022

Remove event.duration and event.ingested from metric events elastic/integrations#4894

Open

kpollich assigned mrodm Apr 4, 2023

This was referenced Apr 19, 2023

[Fleet] Add support for runtime fields in packages elastic/kibana#155255

Closed

Add support to define runtime fields #505

Merged

mrodm mentioned this issue Apr 20, 2023

Add support for runtime fields elastic/elastic-package#1229

Closed

mrodm mentioned this issue Jun 15, 2023

Update minimum Kibana version required for runtime fields #543

Merged

2 tasks

mrodm mentioned this issue Jul 10, 2023

[Fleet] Add support for Runtime Fields elastic/kibana#161129

Merged

1 task

mrodm closed this as completed in elastic/kibana#161129 Jul 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for runtime fields in packages #39

Add support for runtime fields in packages #39

ruflin commented Aug 26, 2020

ruflin commented Aug 26, 2020

ruflin commented May 31, 2021

simitt commented Jul 6, 2021

ruflin commented Dec 23, 2022

amitkanfer commented Dec 23, 2022

joshdover commented Jan 11, 2023

mrodm commented Apr 18, 2023

jsoriano commented Apr 19, 2023

kpollich commented Apr 19, 2023

mrodm commented Apr 21, 2023

jen-huang commented Apr 21, 2023

jlind23 commented May 5, 2023

mrodm commented May 5, 2023

ruflin commented May 5, 2023

jlind23 commented May 5, 2023

Add support for runtime fields in packages #39

Add support for runtime fields in packages #39

Comments

ruflin commented Aug 26, 2020

ruflin commented Aug 26, 2020

ruflin commented May 31, 2021

simitt commented Jul 6, 2021

ruflin commented Dec 23, 2022

amitkanfer commented Dec 23, 2022

joshdover commented Jan 11, 2023

mrodm commented Apr 18, 2023

jsoriano commented Apr 19, 2023

kpollich commented Apr 19, 2023

mrodm commented Apr 21, 2023

jen-huang commented Apr 21, 2023

jlind23 commented May 5, 2023

mrodm commented May 5, 2023

ruflin commented May 5, 2023

jlind23 commented May 5, 2023