Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Misleading implementation of mappings of nested object subfields #784

Closed
Tracked by #10919
jsoriano opened this issue Aug 9, 2024 · 0 comments · Fixed by elastic/integrations#11016
Closed
Tracked by #10919

Misleading implementation of mappings of nested object subfields #784

jsoriano opened this issue Aug 9, 2024 · 0 comments · Fixed by elastic/integrations#11016
Assignees
@jsoriano
Labels
bug Something isn't working

Comments

@jsoriano
Copy link
Member

jsoriano commented Aug 9, 2024 •
edited
Loading

We have found that fields with type: nested and subfields don't get the mappings for the subfields in the final index template, this can lead to unexpected mappings.

That is that for example for a mapping with several subfields, this empty nested mapping is generated:

"whats": {
    "type": "nested"
}

In combination with dynamic mappings introduced by ecs@mappings this can lead subfields to have completely unexpected mappings.

While investigating we have found that the implementation in Fleet has two different codepaths for type: nested and for type: group-nested. For nested it ignores any subfieds, for group-nested it generates the expected mappings. We have also found that package spec only allows subfields with the group and nested fields, so any workaround based on the use of group-nested would not work at the moment.

So we need to align the behavior of Fleet, package-spec, and the current assumptions in existing integrations. Probably doing the following:

  • Assume that nested and group-nested are synonyms, even if they were not originally, we are in a situation now where nested is used as group-nested, and group-nested is not used.
  • Replace implementation of nested with the implementation of group-nested in Fleet, so they effectively behave the same.
  • Allow in package spec the use of group-nested in all cases where nested is allowed, this will allow to fix issues in packages for current versions of packages.
  • There are two additional parameters for nested objects called include_in_parent and include_in_root, we have only found uses of them in tests, and not on any actual package. We will need to confirm its expected behaviour and decide if we need to keep them.
    • Decided to keep them if possible, just in case they are needed, but no integration is using them at the moment.
  • Consider replacing nested with group-nested in affected packages.

Update: After further investigation the assumptions on group-nested were not correct. group-nested is only used internally in Kibana, and it is fine if it continues this way. So we won't change this. Plan changes to:

Thanks @mrodm for finding that some mappings were not being applied!
@jsoriano jsoriano added the bug Something isn't working label Aug 9, 2024
@jsoriano jsoriano mentioned this issue Aug 9, 2024
Closed
@mrodm mrodm mentioned this issue Aug 29, 2024
Closed
@jsoriano jsoriano mentioned this issue Aug 29, 2024
Closed
23 tasks
@jsoriano jsoriano self-assigned this Aug 29, 2024
This was referenced Aug 29, 2024
Closed
Merged
jsoriano added a commit to elastic/kibana that referenced this issue Aug 30, 2024
@jsoriano
[Fleet] Expand subfields of nested objects when generating template (#…
cdb1eb8 
…191730)

There are definitions of nested objects whose fields are defined as
subfields, like this:
```
  - name: a
    type: nested
    fields:
    - name: b
      type: keyword
```
This should generate a template with the subfields as subproperties:
```
      "properties": {
        ...
        "a": {
          "type": "nested",
          "properties": {
            "b": {
              "ignore_above": 1024,
              "type": "keyword",
            },
          },
        },
      },
```
This change adds support for this. Without it the nested object is
empty, without subfields, what is unexpected.

See elastic/package-spec#784 for more context.
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Aug 30, 2024
@jsoriano
[Fleet] Expand subfields of nested objects when generating template (e…
cc0f3eb 
…lastic#191730)

There are definitions of nested objects whose fields are defined as
subfields, like this:
```
  - name: a
    type: nested
    fields:
    - name: b
      type: keyword
```
This should generate a template with the subfields as subproperties:
```
      "properties": {
        ...
        "a": {
          "type": "nested",
          "properties": {
            "b": {
              "ignore_above": 1024,
              "type": "keyword",
            },
          },
        },
      },
```
This change adds support for this. Without it the nested object is
empty, without subfields, what is unexpected.

See elastic/package-spec#784 for more context.

(cherry picked from commit cdb1eb8)
kibanamachine added a commit to elastic/kibana that referenced this issue Aug 30, 2024
@kibanamachine @jsoriano
[8.15] [Fleet] Expand subfields of nested objects when generating tem…
ab6bb9c 
…plate (#191730) (#191879)

# Backport

This will backport the following commits from `main` to `8.15`:
- [[Fleet] Expand subfields of nested objects when generating template
(#191730)](#191730)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Jaime Soriano
Pastor","email":"jaime.soriano@elastic.co"},"sourceCommit":{"committedDate":"2024-08-30T19:25:15Z","message":"[Fleet]
Expand subfields of nested objects when generating template
(#191730)\n\nThere are definitions of nested objects whose fields are
defined as\r\nsubfields, like this:\r\n```\r\n - name: a\r\n type:
nested\r\n fields:\r\n - name: b\r\n type: keyword\r\n```\r\nThis should
generate a template with the subfields as subproperties:\r\n```\r\n
\"properties\": {\r\n ...\r\n \"a\": {\r\n \"type\": \"nested\",\r\n
\"properties\": {\r\n \"b\": {\r\n \"ignore_above\": 1024,\r\n \"type\":
\"keyword\",\r\n },\r\n },\r\n },\r\n },\r\n```\r\nThis change adds
support for this. Without it the nested object is\r\nempty, without
subfields, what is unexpected.\r\n\r\nSee
elastic/package-spec#784 for more
context.","sha":"cdb1eb8fe475c3f1dad0805586c06775538d2d40","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","Team:Fleet","backport:prev-minor","v8.16.0"],"title":"[Fleet]
Expand subfields of nested objects when generating
template","number":191730,"url":"https://github.com/elastic/kibana/pull/191730","mergeCommit":{"message":"[Fleet]
Expand subfields of nested objects when generating template
(#191730)\n\nThere are definitions of nested objects whose fields are
defined as\r\nsubfields, like this:\r\n```\r\n - name: a\r\n type:
nested\r\n fields:\r\n - name: b\r\n type: keyword\r\n```\r\nThis should
generate a template with the subfields as subproperties:\r\n```\r\n
\"properties\": {\r\n ...\r\n \"a\": {\r\n \"type\": \"nested\",\r\n
\"properties\": {\r\n \"b\": {\r\n \"ignore_above\": 1024,\r\n \"type\":
\"keyword\",\r\n },\r\n },\r\n },\r\n },\r\n```\r\nThis change adds
support for this. Without it the nested object is\r\nempty, without
subfields, what is unexpected.\r\n\r\nSee
elastic/package-spec#784 for more
context.","sha":"cdb1eb8fe475c3f1dad0805586c06775538d2d40"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/191730","number":191730,"mergeCommit":{"message":"[Fleet]
Expand subfields of nested objects when generating template
(#191730)\n\nThere are definitions of nested objects whose fields are
defined as\r\nsubfields, like this:\r\n```\r\n - name: a\r\n type:
nested\r\n fields:\r\n - name: b\r\n type: keyword\r\n```\r\nThis should
generate a template with the subfields as subproperties:\r\n```\r\n
\"properties\": {\r\n ...\r\n \"a\": {\r\n \"type\": \"nested\",\r\n
\"properties\": {\r\n \"b\": {\r\n \"ignore_above\": 1024,\r\n \"type\":
\"keyword\",\r\n },\r\n },\r\n },\r\n },\r\n```\r\nThis change adds
support for this. Without it the nested object is\r\nempty, without
subfields, what is unexpected.\r\n\r\nSee
elastic/package-spec#784 for more
context.","sha":"cdb1eb8fe475c3f1dad0805586c06775538d2d40"}}]}]
BACKPORT-->

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
This was referenced Sep 4, 2024
Merged
Merged
jsoriano added a commit to elastic/kibana that referenced this issue Sep 6, 2024
@jsoriano
Reapply "[Fleet] Expand subfields of nested objects when generating t…
fdbaff1 
…emplate (#191730)" (#191897) (#192246)

This change was released at the end in 8.15.1, so let's keep it the
branch.
    
There are definitions of nested objects whose fields are defined as
subfields, like this:
```
  - name: a
    type: nested
    fields:
    - name: b
      type: keyword
```
This should generate a template with the subfields as subproperties:
```
      "properties": {
        ...
        "a": {
          "type": "nested",
          "properties": {
            "b": {
              "ignore_above": 1024,
              "type": "keyword",
            },
          },
        },
      },
```
This change adds support for this. Without it the nested object is
empty, without subfields, what is unexpected.

See elastic/package-spec#784 for more context.

This change was originally reverted in 8.15 in  #191897

Release notes were manually added in elastic/ingest-docs#1292
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jsoriano jsoriano

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Adapt definitions of nested subfields to current Fleet implementation jsoriano/integrations
1 participant
@jsoriano