Pre-save modification hooks for Saved Objects

[timroes]

It would be nice to have something like "derived/computed fields" in a saved object. Those fields are meant to be derived from another field in the same saved object and the saved object infrastructure should automatically calculate them on every update of the saved object.

Use-Case

Inside visualizations we store (already previously, but that's getting more obvious with a new format) the whole config in one keyword field as a canvas expression. We now need parts of that information available in different places, like the visualization type, that is sitting within that long field in some place.

There are places where we could calculate it ad-hoc, e.g. to show the visualization type icon in the visualization list. We could parse the AST of the expression for every page when showing it to extract that information. It might just be slow doing this on-the-fly, but we don't know that yet. There are places where we currently work around not having that information in a single field, e.g. telemetry. We want to report the amount of each visualization type. That means we need to parse potentially thousands of expressions to get their type and then report this. Since that can take a couple of seconds (also with enough visualizations easily above 10s), it doesn't make sense to do this every telemetry reporting cycle ad-hoc. So we're (planning to) use the task manager to sum this information up once a day (which is more then enough for that use-case) and then report that summed up data. It would already be nicer not needing the task manager for that.

Last but not least we have unfortunately the category for which we need that information in a separate field. Mainly because we need to query/filter/aggregate directly on it. The easiest example here is providing a Visualization Type filter in the visualization loading screen, where you can filter visualizations down by their type. To achieve that we need that information in a separate field we can query.

Solutions

So why putting that in the saved object infrastructure in general? Why can't the editor who updates that expression, just update those fields too. The problem here is, that there are multiple ways to update a saved object, that we know about: the editor updates it, the user imports it, the users uses the saved object UI to update it, the user export, batch process and reimport them, ...

Since those fields would always need to have the correct computed/derived data from the other fields, I think the most reasonable place is to put this into the saved object infrastructure in general as a pre-save hook. That way no matter in what way that object got updated, we can make sure these fields are up to date.

@spalger already outlined some nice ideas how that could look like by extending the mappings with scripts to derive those data.

From our current use-case we would only have a 1:N mapping for the derived fields, so every derived field can be computed from exactly one "real" field, which should make the process around partial updates easier.

[elasticmachine]

Pinging @elastic/kibana-platform