Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow alias fields to target unavailable fields #104744

Open
felixbarny opened this issue Jan 25, 2024 · 4 comments
Open

Allow alias fields to target unavailable fields #104744

felixbarny opened this issue Jan 25, 2024 · 4 comments
Labels
>enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types :StorageEngine/Logs You know, for Logs Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch Team:StorageEngine

Comments

@felixbarny
Copy link
Member

To ease the transition between ECS and OpenTelemetry semantic convention (SemConv) mappings, we're planning to store the two different types of data in different data streams. With the help of alias fields, we'd like to make it possible to re-use existing queries that target ECS fields on OTel data streams. Similarly, we want to add SemConv aliases to ECS-based data streams so the queries in our own UIs and in custom dashboards can target SemConv data while still making it possible to incorporate data from ECS-based data streams.

More on how we'd like to map OTel data in this PR: #104455.

One of the challenges is this requirement of alias fields:

The target field must exist at the time the alias is created.

We'd like to move away from explicitly defining fields in the mapping and instead rely on dynamic templates (such as ecs@mappings, so that the mappings only contain fields that are actually used.

However, this is incompatible with the requirement that the target for alias fields must already exist.

cc @dakrone

@felixbarny felixbarny added >enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types :StorageEngine/Logs You know, for Logs labels Jan 25, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-storage-engine (Team:StorageEngine)

@javanna
Copy link
Member

javanna commented Jan 29, 2024

This reminds of the need to move alias fields to be implemented as runtime fields, see #87969 . I don't think we should make further changes to the alias field implementation, as many of the problems we encounter with it, would be solved by re-implementing them as runtime fields. We could take this additional requirement around targeting unavailable fields into account perhaps.

gmarouli added a commit to gmarouli/elasticsearch that referenced this issue Feb 2, 2024
elasticsearchmachine pushed a commit that referenced this issue Feb 2, 2024
Easy fix, there was a typo in the warning instead of checking for the
correct index patterns `other-*` it was checking for `ds-*`.

Closes #104774
@javanna javanna added Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch and removed Team:Search Meta label for search team labels Jul 16, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-search-foundations (Team:Search Foundations)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types :StorageEngine/Logs You know, for Logs Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch Team:StorageEngine
Projects
None yet
Development

No branches or pull requests

3 participants