Add nested field support to KQL

[Bargs]

Summary

Closes #44554

This PR adds a new syntax to KQL for querying nested fields.

Nested fields can be queried in two different ways:

1. Parts of the query may only match a single nested doc (bool inside nested). This is what most people want when querying on a nested field.
2. Parts of the query may match different nested docs (nested inside bool). This is how a regular object field works but nested fields can be queried in the same way. Although generally less useful, there are occasions where one might want to query a nested field in this way.

The new KQL syntax supports both. Say we have the following document, where level1 and level2 are both nested fields:

{
  "level1": [
    {
      "level2": [
        {
          "foo": "qwe"
          "bar": "asd"
        },
        {
          "foo": "zxc"
          "bar": "rty"
        }
      ]
    },
    {
      "level2": [
        {
          "foo": "fgh"
          "bar": "vbn"
        },
        {
          "foo": "uio"
          "bar": "jkl"
        }
      ]
    }
  ]
}

If we wanted to match a single level2 document we could write a query like this:

level1.level2:{ foo:qwe and bar:asd }

The first part (level1.level2) is the nested path. Everything inside the curly braces (the "nested group") must match a single document. So for example, level1.level2:{foo:qwe and bar:rty} would not match because there isn't a single nested document that matches the entire query in the nested group. This is the first way to query a nested field listed above (bool inside nested).

What if we wanted to match a parent doc that contains a document where foo = qwe and a document where bar = rty, but it's ok if those are two different sub documents? This is the second way of querying a nested field (nested inside bool). We can do this:

level1.level2:{ foo:qwe } and level1.level2:{ bar:rty }

Since the individual query clauses are in separate nested groups they are allowed to match separate documents.

Common questions

• In the above scenario, why don't we just auto-detect that a nested field is being queried and save the user some typing?

We could, but it sets the user up for failure. Most users will want to use nested fields to find individual sub-documents that match a query. If we allow querying nested fields without the special syntax, many users will end up with something like this:

level1.level2.foo:qwe and level1.level2.bar:asd

Each query clause here could match a different sub-document, which is likely not what a user expects. By requiring the special syntax for any nested query we help the user understand that there is an extra decision to be made here. Furthermore, we selecting a nested field via the autocomplete we will automatically add the necessary path and curly braces, so there isn't really any extra typing to be done.

• Why do we require the path if we already know what it is for each field in the index pattern by reading field_caps?

The user will need this flexibility in the scenario where they have nested fields that contain other nested fields. They need the ability to control at what level the nested grouping occurs. Extending the previous example, let's say we want to match an individual document at level1, but we don't care if multiple documents match at level2. We can do this:

level1:{ level2:{ foo:qwe } and level2:{ bar:rty } }

This query would match our document, because the nested grouping is happening at level1. foo:qwe and bar:rty each exist in a single level1 document, even though they're in separate level2 documents. However the next query would fail, because the matching docs are not even in the same level1 document:

level1:{ level2:{ foo:qwe } and level2:{ bar:jkl } }

UI

Aside from the KQL syntax updates there are a few additions to the UI.

When selecting a nested field in the autocomplete for the first time, we display a helpful notification explaining that there is a special syntax for nested fields and point them to the docs explaining how it works:

When selecting a nested field in the autocomplete, we auto-add the required nested field syntax. The autocomplete also only suggests valid sub-fields inside of a nested group:

If you attempt to query a nested field without a nested group you'll get an error:

We also have helpful errors when there is a nested group but the path is not correct, and when a non-nested field is used inside a nested group.

TODO

• Update code that was already using parent/subType properties on index pattern
• Write migration for changes to parent/subType properties on index pattern
• Write docs
• Update and add new tests
  • autocomplete
  • getFullFieldNameNode

Checklist

Use strikethroughs to remove checklist items you don't feel are applicable to this PR.

• This was checked for cross-browser compatibility, including a check against IE11
• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• This was checked for keyboard-only and screenreader accessibility

For maintainers

• This was checked for breaking API changes and was labeled appropriately
• This includes a feature addition or change that requires a release note and was labeled appropriately

[gchaps]

@Bargs Thanks for the explanation. Will either of these work?

It looks like you're querying on a nested field. KQL has a special syntax for working with nested fields, and we filled in a query for you. For more suggestions, check out our docs.

It looks like you're querying on a nested field. You can construct KQL syntax for nested queries in different ways, depending on the results you want. Learn more in our docs.

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 4b589a5

[Bargs]

Thanks for all the helpful suggestions @gchaps! I updated the text in the help toast with your second suggestion and I pulled in all your inline suggestions except one which I had a tiny question about.

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: d34ccfd

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 898f76a

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 3db6b18

[timroes]

Maybe before starting with the more complex example here, let's explain the difference to what items:{ name:banana } and items:{ stock > 10 } actually is. I know that the next paragraph explains that behavior, but I think it might be a bit more clear if we use exactly the same example and show how this works in the "opposite" way, before then continue on an example with a more complex query? cc @gchaps

[gchaps]

@Bargs I agree with what Tim is saying. Happy to review the updated text when it is ready.

[timroes]

Just some minor code/doc comments. Tested on Chrome Linux everything I tested seems to work as expected. LGTM

[Bargs]

@timroes since the only comment left is a docs suggestion I am gonna merge this while it's green and if @gchaps has a suggestion I can update the docs later. Thank you again for the quick review! 🙇 🙇 🙇