Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ML] Add support for search-time runtime fields to data frame analytics #65056

Closed
droberts195 opened this issue Nov 16, 2020 · 2 comments · Fixed by #69183
Closed

[ML] Add support for search-time runtime fields to data frame analytics #65056

droberts195 opened this issue Nov 16, 2020 · 2 comments · Fixed by #69183
Assignees
@dimitris-athanasiou
Labels
>enhancement :ml Machine learning

Comments

@droberts195
Copy link
Contributor

Since #64374 Elasticsearch has the option to have search-time runtime fields.

ML should allow these to be used in data frame analytics.

The most obvious way to do it seems to be to add an optional runtime_mappings section to the data frame analytics config with the same structure as the runtime_mappings section of searches (as defined in #64374), then add this into the search the data frame analytics job does when reindexing source data to the destination index.

However, after this initial step there is a choice of approaches. For the destination index we could:

  1. Make the search-time runtime fields on the source indices into indexed fields
  2. Make the search-time runtime fields on the source indices into runtime fields stored in the destination index mappings
  3. Keep the search-time runtime fields on the source indices as search-time runtime fields on the destination index

Some analysis of the pros and cons is needed before deciding which option is best.
@droberts195 droberts195 added >enhancement :ml Machine learning labels Nov 16, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (:ml)

@davidkyle davidkyle self-assigned this Nov 16, 2020
@dimitris-athanasiou
Copy link
Contributor

I am working on this following approach 2: allowing the user to provide search-time runtime fields to the config which are then becoming mapped runtime fields in the destination index.

@dimitris-athanasiou dimitris-athanasiou mentioned this issue Feb 18, 2021
Merged
dimitris-athanasiou added a commit that referenced this issue Feb 19, 2021
@dimitris-athanasiou
[ML] Add runtime mappings to data frame analytics source config (#69183)
7fb98c0 
Users can now specify runtime mappings as part of the source config
of a data frame analytics job. Those runtime mappings become part of
the mapping of the destination index. This ensures the fields are
accessible in the destination index even if the relevant data frame
analytics job gets deleted.

Closes #65056
@dimitris-athanasiou dimitris-athanasiou mentioned this issue Feb 19, 2021
Merged
dimitris-athanasiou added a commit that referenced this issue Feb 19, 2021
@dimitris-athanasiou
[7.x][ML] Add runtime mappings to data frame analytics source config … (
98c69ce 
#69284)

Users can now specify runtime mappings as part of the source config
of a data frame analytics job. Those runtime mappings become part of
the mapping of the destination index. This ensures the fields are
accessible in the destination index even if the relevant data frame
analytics job gets deleted.

Closes #65056

Backport of #69183
@stevejgordon stevejgordon mentioned this issue Apr 21, 2021
Closed
62 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dimitris-athanasiou dimitris-athanasiou

Labels
>enhancement :ml Machine learning
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ML] Add runtime mappings to data frame analytics source config dimitris-athanasiou/elasticsearch
4 participants
@davidkyle @droberts195 @dimitris-athanasiou @elasticmachine