Delete saved objects in bulk

[mohitagrawa1]

Hii,
I pushing several visualizations, dashboards, savedsearches and scripted fields using a python script. They altogether fulfil one big functionality, thus I refer them as an application.
Now what I want is to update the whole application.
For this, I have written another python script which deletes each and every object previously created and put the newer ones. This script takes a lot of time as there are several delete rest calls ( one for each component ). Also if it fails before complete execution, there are tracing back and rolling back issues.
Thus is there a way in which I can delete these visualizations, dashboards, savedsearches and scripted at once(in other words, upgrade the application in a single go!).

I am using kibana's Delete Object API and Create Object API for deleting and saving the objects.

If I am able to delete the objects using a bulk delete api, it would be very helpful.

[elasticmachine]

Pinging @elastic/kibana-app

[elasticmachine]

Pinging @elastic/kibana-platform

[njd5475]

This seems like a very reasonable request, since there are already bulkCreate, and bulkGet APIs one would assume that there might also be a bulkDelete API. I am guessing that there is not one already because it has not been needed? @epixa Maybe you have some thoughts about this.

[spalger]

@mohitagrawa1 is it important to delete and recreate the objects? Why not just update the existing ones?

[mohitagrawa1]

Yes @spalger, update is an option, using query parameter 'overwrite'.
But what if I want to delete the dashboards/visualisations permanently or uninstall the application in one go, thus bulk delete would be helpful.

[spalger]

I agree, it would be a nice feature to add, just suggesting an alternate solution until a bulk delete is available.

[mohitagrawa1]

Thanks :)

[abdalians]

+1

[Glenrun]

+1

[lukasolson]

We have a use case in search sessions (see #89570).

[spalger]

@wayneseymour will also need this for changes he's planning to interact with the Saved Objects API directly for functional tests.

[pmuellr]

We have an issue open for bulk delete of alerts, part of which would be bulk delete of the alert SOs - #53144 . Though I suspect we won't be needing this for a few more minor releases ...

[Bamieh]

@joshdover I believe we have enough use cases to prioratize this. We have a couple of use cases in telemetry that would also greatly benefit from having this api.

[pgayvallet]

The signature of the delete API is:

async delete(type: string, id: string, {refresh, force, namespace}: SavedObjectsDeleteOptions): Promise<{}>

To introduce a new bulkDelete API, we would need to have an API similar to what we did for bulkCreate or bulkGet

One of the notable difference between delete and bulkDelete is that delete throws in case of error, as it's targeting a single object, where bulkDelete will need to return per-object status, as it's done for bulkCreate or bulkGet

A draft of the API could look like:

interface SavedObjectsBulkDeleteObject {
  type: string;
  id: string;
}

interface SavedObjectsBulkDeleteOptions extends SavedObjectsBaseOptions {
  refresh?: MutatingOperationRefreshSetting;
  force?: boolean;
}

interface SavedObjectsBulkDeleteStatus {
   success: boolean;
   error?: SavedObjectError;
}

interface SavedObjectsBulkDeleteResponse {
   statuses: SavedObjectsBulkDeleteStatus[];
}

async bulkDelete(objects: SavedObjectsBulkDeleteObject[], options: SavedObjectsBulkDeleteOptions): Promise<SavedObjectsBulkDeleteResponse>

• I don't think allowing per-object force option really makes sense
• as bulkGet does not support retrieving objects from distinct namespaces with a single query, it probably makes sense to do the same for this new bulkDelete API and only have namespace in the global options

WDYT?

cc @elastic/kibana-security do you have any idea of the cost of implementing this API in the security and spaces wrapper?

[legrego]

I don't think allowing per-object force option really makes sense

I agree, this feels unnecessary.

as bulkGet does not support retrieving objects from distinct namespaces with a single query, it probably makes sense to do the same for this new bulkDelete API and only have namespace in the global options

I'm torn. The only reason we don't have it for bulkGet yet is because nobody has asked for it. That said, it's easier to omit this if we don't have a legitimate need. It's confusing as a consumer of the SO client because we have an inconsistent story around multi-space support: some functions support it, while others don't.

do you have any idea of the cost of implementing this API in the security and spaces wrapper?

Spaces wrapper is trivial, and the security wrapper would be a small amount of effort. The largest piece of work would be augmenting the functional test suite to cover this new feature.

[Bamieh]

@pgayvallet thanks for drafting this out! I agree to both points; no need for force, and +1 for consistency across the apis

[kfirpeled]

+1

[spalger]

#136562 is a case where we're trying to use the saved obejcts API instead of the esArchiver in some functional tests but the issue is that the tests create 10k objects for some capacity tests, making this limitation really problematic when we need to cleanup after the test.

[pgayvallet]

We've been postponing this issue for literally years because we never had any high enough reasons to prioritize it. Seems like #136562 may actually be the excuse opportunity to finally tackle this bulkDelete API.

[TinaHeiligers]

ATM, there's refactoring in progress that might put a pause on this work:
Migrating the saved object's domain to packages (core) and refactoring the client wrappers

[TinaHeiligers]

Update: Core's going to go ahead with adding the API as soon as we've migrated the relevant part of the SO domain to packages.

@elastic/kibana-security I assume we need to take care of any legacy aliases that may exist for multi-namespace type objects being deleted, as we do in the single delete request, or am I completely off the mark here?

[legrego]

I assume we need to take care of any legacy aliases that may exist for multi-namespace type objects being deleted, as we do in the single delete request, or am I completely off the mark here?

@TinaHeiligers Yes, bulk delete should behave the same as delete with respect to legacy aliases.