[Discuss] Referential integrity with Saved Objects

[streamich]

TLDR; This thread is to discuss extending Saved Objects service to add referential integrity.

---

Right now referential integrity in Kibana is not enforced. For example, you can delete a visualization, which is used by one or more dashboards, and those dashboards will render with empty panels for that visualization, as the visualization is not removed from the dashboards. Another example is you can have alerts that reference non-existing actions. One more example, is you can have a Dashboard drilldown that references a deleted dashboard.

Some ideas:

• Saved Object service could provide a delete method which checks if it is "safe to delete" that saved object, i.e. it has no other objects referencing it. Delete would fail if a saved object is referenced by one or more other saved objects.
• Saved Objects service could provide a first class way to retrieve all objects referencing your object. (Currently, it is also possible but is not part of Saved Objects service API).
• Not sure if we need it in Kibana, but there could be an option to execute cascading delete — it would delete the object you are trying to delete and all other objects referencing it, if any, recursively.

Related:

• Alerting needs referential integrity

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[rudolf]

I believe the security team is looking to implement this as part of #27004 although it's unclear from that issue exactly when that's planned.

[legrego]

Referential integrity checks would be part of phase 3 if I'm not mistaken (we are currently on Phase 1.5). It's important to note that this will be a best-effort check. It won't be possible to guarantee that references will always be correct.

[jportner]

Our planned work for "referential integrity checks" is only to support converting (existing) single-namespace types to (new) multi-namespace types.
So that means ensuring references stay intact during the conversion process itself, along with during import and copy.
I view the ideas described above as complementary to that.

• Saved Object service could provide a delete method which checks if it is "safe to delete" that saved object, i.e. it has no other objects referencing it. Delete would fail if a saved object is referenced by one or more other saved objects.

WRT multi-namespace objects: fortunately, we changed "Delete" so it only removes the object from the current Space. In a nutshell, this check is not more complex for multi-namespace objects, because we only have to check references in the current Space 👍

The other way an object may get deleted is when an entire Space is deleted (deleteByNamespace operation), but that uses a different deleteByQuery cluster call so it should be unaffected.

• Saved Objects service could provide a first class way to retrieve all objects referencing your object. (Currently, it is also possible but is not part of Saved Objects service API).

I like this idea (I've been calling these "inbound references" while working with them)

[pgayvallet]

I think this all makes a lot of sense.

Saved Object service could provide a delete method which checks if it is "safe to delete" that saved object, i.e. it has no other objects referencing it. Delete would fail if a saved object is referenced by one or more other saved objects.

We could simply add a safeDelete=false option to SavedObjectsDeleteOptions. When true, it would check for potential parent references before performing the delete and fails if any is found.

Saved Objects service could provide a first class way to retrieve all objects referencing your object. (Currently, it is also possible but is not part of Saved Objects service API).

AFAIK it's pretty simple, at least for single namespaced SO, it's a plain

client.find({
    hasReference: { type, id }
})

So exposing this a public API should be easy. Just need to be cautious regarding namespace scope: Do we want/need to be able to retrieve referencing objects from all namespaces, or always ONLY from the current one?

Not sure if we need it in Kibana, but there could be an option to execute cascading delete — it would delete the object you are trying to delete and all other objects referencing it, if any, recursively.

Ah, the famous cascade delete in a non-transactional world. Been there, done that with mongo. Would wait to have an actual use case to open this discussion imho, as error handling in the middle of the cascade is something I wouldn't want my worse enemy to have to implement.

[ppisljar]

Thank you for contributing to this issue, however, we are closing this issue due to inactivity as part of a backlog grooming effort. If you believe this feature/bug should still be considered, please reopen with a comment.