[saved objects] add support for runtime fields in Saved Object find() requests

[pmuellr]

For several use cases in alerting, we have fields in our saved objects which are "dynamic" - the actual fields are determined by the type of alerting rule the saved object represents. As an example, we store alerting rules parameters currently as flattened objects which is a change from our original implementation which used type: object; enabled: false. flattened gives us some searchability, but alerting rule builders would like more, including numeric access.

Now that runtime fields are available, another thought is to allow runtime field access to these, presumably via the source. We'd have to go through another effort like Make alert params searchable to see if runtime fields would actually end up working out here - it's not clear it provides a better answer than flattened in the end, but we are thinking it might. And understand there will be caveats, like probably not being able to get the runtime field values back out directly - which is probably fine.

To do that though, we'd need support in the Saved Object client to support passing in the runtime field mappings, which as near as I can tell, it doesn't currently support. Presumably that would be code that would pass in runtime_mappings in the ES search API request body, and so would need a similar field in the SavedObjectsFindOptions of the SavedObjectsClient.find() API.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[rudolf]

Since runtime fields take precedence over mapped fields with the same name we would have to add basic validation to this parameter to ensure consumers can't override fields used for security i.e. type, namespace & namespaces.

To fit in with the rest of the Saved Objects API's we'd want to rename fields so that find with a runtime mapping of dashboard.attributes.description will add a dashboard.description runtime field when searching over the dashboard saved object type. It becomes a lot harder to rename references to fields in scripted runtime fields and here I think we should just accept that we can't expose a consistent API.

Because KQL filters are strictly validated to only be able to filter over known fields we wouldn't be able to filter over runtime fields. This KQL filters validation was added to help catch bugs where a filter is defined over fields which don't exist, so we could relax this validation when runtime fields are defined (or even consider all the runtime fields when doing the validation, though this would be a bit more complex).

I would estimate this to be a medium to large effort and unless we're pressed for time it would be better to rather expose this functionality through the KPT search API #102177

In the meantime, we should be able to validate if this is a viable solution for #112193 by experimenting with using the Elasticsearch API directly.

E.g. we could approximate the types of queries that are possible with saved objects by something like

GET .kibana/_search
{
  "runtime_mappings": {
    "day_of_week": {
      "type": "keyword",
      "script": {
        "source": "emit(doc['updated_at'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
      }
    }
  },
  "query": {
    "bool": {
      "filter": [
        { "term": { "day_of_week": "Tuesday"   }}
      ],
      "should": {
    "simple_query_string" : {
        "query": "Tuesday",
        "fields": ["title^5", "body", "day_of_week"],
        "default_operator": "and"
    }
      }
    }
  }
}

[cnasikas]

In Cases, there is a request to show the average resolve time (how long a case was open) of all cases. In our saved objects we save the created_at and closed_at date in ISO format. To provide the metric we need to do an aggregation over these properties. The problem is that the properties are strings in ISO format. To do the aggregation we need to transform the properties to UNIX timestamps so the aggregation can be over integers. Runtime fields would help to provide that feature without having to duplicate the information in ES.

I think having the functionality in the KTP would be sufficient. About your proposal with the ES query, is it ok to do queries over the Kibana index? What about authorization? For metrics, it is more about anonymity (privacy-preserving) so maybe it is acceptable to leak this information (average resolve time).

[pmuellr]

Since runtime fields take precedence over mapped fields with the same name we would have to add basic validation to this parameter to ensure consumers can't override fields used for security i.e. type, namespace & namespaces.

Just taking a peek at this issue again. Rudolf raises a number of valid issues in his comment above, but none of those issues are relevant to my use case. I'm not actually suggesting we let users make use of a runtime field - we'd like to use it internally. We'd be perfectly happy with the requirement that we'd have to know how to do those mappings with the attributes property explicitly, etc. All I really want is a way to pass a valid ES runtime mappings object into the SO find API as an option. Perhaps we could call it rawRuntimeMappings or such, to suggest that it's the nice mapping that Rudolf is suggesting.

It seems fairly straight-forward, and would provide TONS of value.

[pgayvallet]

All I really want is a way to pass a valid ES runtime mappings object into the SO find API as an option

Just to make sure, is the only need to be able to define a runtime_mappings property to the underlying _search request to be able to create queries based on those runtime definitions, or would you also need to retrieve those value in the objects returned by the SoClient.find call that used these runtime mappings?

[pmuellr]

Ah ... I hadn't considered the conversion to pull the the data out of the es doc, to put in the response. And ... I'm honestly not sure, because the use case is primarily for querying, and we'd be getting the field we want in the original attributes anyway.

To be specific, we have SO attributes which are flattened and some which are "JSON blob stored as unindexed", and we'd like to be able to query against those (flattened doesn't currently allow numeric fields, everything's a keyword). So the thinking is we'd add a runtime mapping to extract that field from the source, and expose it as something queryable. We don't need the runtime field in the result, because the value already gets returned in the flattened field / JSON blob, where we are expecting it.

So I think that this would work for this use case - just allowing us to pass runtime_mappings in - however I also suspect there are folks WOULD like to have a "calculated field" which WAS returned in the result, and it's not clear how that would work.

We are actually looking at going down another route at this time, bypassing all of this - so may not be something we need any time soon. But I suspect there are a number of other flattened field / JSON blob as unindexed field use cases out there that could take advantage of something like this.

[pgayvallet]

however I also suspect there are folks WOULD like to have a "calculated field" which WAS returned in the result, and it's not clear how that would work.

Yea, that was going to be my point. Adapting the find response structure to support such runtime mappings would be more complex than just allowing API consumers to pass the mappings in the query, and I'm unsure how exactly we should be handling that if we were to allow it.

But I suspect there are a number of other flattened field / JSON blob as unindexed field use cases out there that could take advantage of something like this

Without any doubt. It's just a question of do we need this in the SO APIs now, how is it acceptable to wait until consumers will be allowed to do it directly using KPT.

[rudolf]

This would also help us reduce the field count and could be a great solution for plugins that might be prototyping to be able to iterate quickly with runtime fields and once their data model is mature they can "commit" to a set of mappings that should be more future proof.

Given the relatively lower priority of KPT we should consider if this might be worth introducing into the Saved Objects Find API.

[pgayvallet]

we should consider if this might be worth introducing into the Saved Objects Find API.

We would also need to understand the implications regarding downward / bidirectional migrations.