Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Auto-flatten mappings when subobjects is set to false #99860

Closed
felixbarny opened this issue Sep 25, 2023 · 1 comment · Fixed by #103542
Closed

Auto-flatten mappings when subobjects is set to false #99860

felixbarny opened this issue Sep 25, 2023 · 1 comment · Fixed by #103542
Labels
>enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch

Comments

@felixbarny
Copy link
Member

In Elastic observability, we want to set subobjects: false at the root level in logs, metrics, and traces data streams.

Auto-flattening documents (#88934) played a huge role in making this a backwards compatible change.

However, there's another hurdle to adoption: when subobjects are disabled, mappings need to be specified in a flattened way, where the full path is denoted as a dotted string, instead of using properties, which implies the object field type which is not supported when subobjects are disabled.

While we can change the mappings in our integrations, we can't change user-defined mappings that are mixed into the integration mappings via <integration>@custom component templates.

Therefore, changing subobjects to false would be a breaking change that results in errors when updating integrations.

To mitigate this, we'd want to automatically flatten mappings:

When given an object mappings like this:

mapping:
  subobjects: false
  properties:
    foo:
      properties:
        bar:
          type: keyword

We'll turn it into a flat mapping like this:

mapping:
  subobjects: false
  properties:
    foo.bar:
      type: keyword

There will be some limitations, however. For example, this doesn't work when mapping parameters on the object field type are used (such as enabled and dynamic). These can't simply be pushed down to the children as they also affect non-explicitly mapped children that are added via dynamic mapping. There may be something we can do in the future to support some kind of wildcard matching (such as "foo.*" { "enabled": false }) but this won't be part of the first iteration and we'll need to get more quantitative data on whether it is worth the development effort and added complexity.

Also, the nested field type is not supported with subobjects: true.
@felixbarny felixbarny added >enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types labels Sep 25, 2023
@elasticsearchmachine elasticsearchmachine added the Team:Search Meta label for search team label Sep 25, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@felixbarny felixbarny mentioned this issue Dec 1, 2023
Merged
2 tasks
zmoog added a commit to elastic/kibana that referenced this issue Dec 12, 2023
@zmoog @kibanamachine
Fleet: add support for subobjects: false (#171826)
e6e10b5 
## Summary

Update the Fleet plugin to support the
[subobjects](https://www.elastic.co/guide/en/elasticsearch/reference/current/subobjects.html)
setting on the `object` type mapping.

This PR supports the `subobjects` setting on a per-field basis. We will
add support for `subobjects` setting at the data stream level when 
Elasticsearch will automatically flatten the mappings in elastic/elasticsearch#99860.

The PR add deals with the following [user
cases](elastic/package-spec#349 (comment))
found in the integration packages and add support for a few of them.

### ~~Case A~~

```yaml
- name: a.labels
  type: object
  subobjects: false
```

The use case A is invalid on package-spec v3 and it's not supported.


### Case B

```yaml
- name: b.labels.*
  type: object
  object_type: keyword
  subobjects: false
```

that `_generateMappings()` should map to:

```js
{
  dynamic_templates: [{
    "b.labels.*": {
      path_match: "b.labels.*",
      match_mapping_type: "string",
      mapping: {
        type: "keyword"
      }
    }
  }],
  properties: {
    b: {
      type: 'object',
      dynamic: true,
      properties: {
        labels: {
          dynamic: true,
          type: 'object',
          subobjects: false,
        }
      }
    },
  },
}
```

### ~~Case C~~

```yaml
- name: prometheus.c.labels.*
  type: object
  subobjects: false
```

The use case C is considered invalid and it's not supported.

### Case D

```yaml
- name: d.labels
  type: object
  object_type: keyword
  subobjects: false
```

that `_generateMappings()` should map to:

```js
{
  dynamic_templates: [{
    "d.labels": {
      path_match: "d.labels.*",
      match_mapping_type: "string",
      mapping: {
        type: "keyword"
      }
    }
  }],
  properties: {
    d: {
      type: 'object',
      dynamic: true,
      properties: {
        labels: {
          dynamic: true,
          type: 'object',
          subobjects: false,
        }
      }
    },
  },
}
```

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
@zmoog zmoog mentioned this issue Dec 12, 2023
Merged
3 tasks
@felixbarny felixbarny mentioned this issue Dec 19, 2023
Merged
@felixbarny felixbarny mentioned this issue Jan 3, 2024
Closed
@zmoog zmoog mentioned this issue Mar 11, 2024
Merged
2 tasks
ruflin pushed a commit to elastic/package-spec that referenced this issue Mar 26, 2024
@zmoog @mrodm
Add subobjects definition (at the data stream level) (#727)
93da9b5 
## What does this PR do?

<!-- Mandatory
Explain here WHAT changes you made in the PR.
-->

Add support for `subobjects: false` at the data stream level. 

Here is an example:

```yaml
# From /packages/good_v3/data_stream/subobjects/manifest.yml
title: my-data-stream
type: logs
elasticsearch:
  index_template:
    mappings:
      subobjects: true
```

## Why is it important?

<!-- Mandatory
Explain here the WHY, or the rationale/motivation for the changes.
-->

Give integration developers (per data stream) access to the [subobjects](https://www.elastic.co/guide/en/elasticsearch/reference/current/subobjects.html) option in the integration's index template mappings.

Since we added the `subobjects` option in stack version 8.3, users could customize how Elasticsearch handles fields that contain dots in their names from `true` (expanded, current default) to `false` (not expanded). However, integration developers could not set this up in the integrations.

Note on per filed option: the `subobjects` option [has been available](#573) at the field level since package-spec 3.1.0. However, to make this happen at the data stream level, we needed elastic/elasticsearch#99860 to land in Elasticsearch.

## Checklist

<!-- Mandatory
Add a checklist of things that are required to be reviewed in order to have the PR approved

List here all the items you have verified BEFORE sending this PR. Please DO NOT remove any item, striking through those that do not apply. (Just in case, strikethrough uses two tildes. ~~Scratch this.~~)
-->

- [x] I have added test packages to [`test/packages`](https://github.com/elastic/package-spec/tree/main/test/packages) that prove my change is effective.
- [x] I have added an entry in [`spec/changelog.yml`](https://github.com/elastic/package-spec/blob/main/spec/changelog.yml).

## Related issues

<!-- Recommended
Link related issues below. Insert the issue link or reference after the word "Closes" if merging this should automatically close it.

- Closes #123
- Relates #123
- Requires #123
- Supersedes #123
-->
- #349
- #573
- elastic/elasticsearch#99860 (requirement)


---------

Co-authored-by: Mario Rodriguez Molins <mario.rodriguez@elastic.co>
@javanna javanna added Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch and removed Team:Search Meta label for search team labels Jul 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Flatten object mappings when subobjects is false felixbarny/elasticsearch
4 participants
@javanna @felixbarny @eyalkoren @elasticsearchmachine