Need a replacement for esArchiver.emptyKibanaIndex();

[bhavyarm]

I am migrating test/api_integration/fixtures/es_archiver/management/saved_objects/scroll_count to kbnArchiver (kibana/test/api_integration/apis/saved_objects_management/scroll_count.ts) And this test is using await esArchiver.emptyKibanaIndex();.

I tried to replace it with await kibanaServer.savedObjects.cleanStandardList(); but the test failed.
This is the test run with kibanaServer.savedObjects.cleanStandardList();

scroll_count with more than 10k objects
         └-> "before all" hook: beforeTestSuite.trigger for "returns the correct count for each included types"
         └-> "before all" hook for "returns the correct count for each included types"
         └-> returns the correct count for each included types
           └-> "before each" hook: global before each for "returns the correct count for each included types"
           └- ✓ pass  (493ms)
         └-> "after all" hook for "returns the correct count for each included types"
           │ debg Cleaning all saved objects { space: undefined }
           │ info deleting batch of 1000 objects
           │ info deleting batch of 1000 objects
           │ info deleting batch of 1000 objects
           │ info deleting batch of 1000 objects
           │ info deleting batch of 1000 objects
           │ info deleting batch of 1000 objects
           │ info deleting batch of 1000 objects
           └- ✖ fail: apis saved objects management apis scroll_count scroll_count with more than 10k objects "after all" hook for "returns the correct count for each included types"
           │      Error: Timeout of 360000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/bhavyarajumandya/Desktop/kibana_dev/kibana/test/api_integration/apis/saved_objects_management/scroll_count.ts)
           │       at listOnTimeout (node:internal/timers:559:17)
           │       at processTimers (node:internal/timers:502:7)
           │ 
           │ 
         └-> "after all" hook: afterTestSuite.trigger in "scroll_count"
       └-> "after all" hook: afterTestSuite.trigger in "saved objects management apis"
     └-> "after all" hook: afterTestSuite.trigger in "apis"
 │
 │5 passing (6.0m)
 │1 failing
 │
 │1)    apis
 │       saved objects management apis
 │         scroll_count
 │           scroll_count with more than 10k objects
 │             "after all" hook for "returns the correct count for each included types":
 │
 │      Error: Timeout of 360000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/bhavyarajumandya/Desktop/kibana_dev/kibana/test/api_integration/apis/saved_objects_management/scroll_count.ts)
 │       at listOnTimeout (node:internal/timers:559:17)
 │       at processTimers (node:internal/timers:502:7)
 │ 
 │ 

And this is the test run with await esArchiver.emptyKibanaIndex();

> "after all" hook for "includes all requested types even when none match the search"
           │ debg unloading docs from archive at /Users/bhavyarajumandya/Desktop/kibana_dev/kibana/test/api_integration/fixtures/kbn_archiver/saved_objects/scroll_count.json
           │ info deleting 6 objects { space: undefined }
           │ succ 6 saved objects deleted
           │ debg Cleaning all saved objects { space: undefined }
           │ info deleting batch of 0 objects
           │ succ deleted 0 objects
         └-> "after all" hook: afterTestSuite.trigger for "includes all requested types even when none match the search"
       └-: scroll_count with more than 10k objects
         └-> "before all" hook: beforeTestSuite.trigger for "returns the correct count for each included types"
         └-> "before all" hook for "returns the correct count for each included types"
         └-> returns the correct count for each included types
           └-> "before each" hook: global before each for "returns the correct count for each included types"
           └- ✓ pass  (550ms)
         └-> "after all" hook for "returns the correct count for each included types"
           │ warn since spaces are enabled, all objects other than the default space were deleted from .kibana rather than deleting the whole index
           │ info [emptyKibanaIndex] Deleted existing index ".kibana"
           │ debg Migrating saved objects
           │ info [emptyKibanaIndex] Created index ".kibana"
         └-> "after all" hook: afterTestSuite.trigger for "returns the correct count for each included types"
       └-> "after all" hook: afterTestSuite.trigger in "scroll_count"
     └-> "after all" hook: afterTestSuite.trigger in "saved objects management apis"
   └-> "after all" hook: afterTestSuite.trigger in "apis"

Creating an issue because we want to get rid of esArchiver.

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[spalger]

I'm pretty sure the problem is that there are so many saved objects to delete, and there isn't any bulk delete API. @elastic/kibana-core can we discuss the option of closing #30503 again?

[TinaHeiligers]

can we discuss the option of closing #30503 again

I've added it to our sync.

[pgayvallet]

I'm pretty sure the problem is that there are so many saved objects to delete, and there isn't any bulk delete API.

I tend to agree that this is the issue here. Deleting 10k objects without bulk APIs will likely take quite some time.

can we discuss the option of closing #30503 again?

I can close the issue for sure, but you probably meant to resolve it ?

[tylersmalley]

Realize this is from a few weeks ago... but could an alternative solution for this test be to use a space and delete it?

[spalger]

Does deleting a space clean up their saved objects in bulk? That sounds like a reasonable idea to me

[rudolf]

@tylersmalley @spalger Yes, deleting a space will delete all its documents too.

One edge case is creating an object and then sharing it to another space. In this case, removing the original space won't delete the object. But outside of the spaces test suites I doubt this happens often.

[TinaHeiligers]

FYI folks, #139680 merged in time for 8.5 FF.
A few caveats with the API that are worth mentioning:

1. hidden objects cannot be deleted using the API, these are filtered out from the permissible SO types we act on
2. Deleting shared saved objects that exist in more than one space has to be done by force. It's a similar option as to single delete except that the behavior applies to the whole request. If you want to be selective over which objects to force delete, you'll have to delete them one by one (or use the space-delete "hack")
3. Deleting many shared saved objects in one request might block the event loop because we have to delete legacy url aliases for each object that's deleted. ATM, these are done one at a time and we're using a pMap to minimize the number of concurrent requests. If anyone wants to help out and feels up to modifying the deleteByQuery script, feel free to do so, otherwise, we'll try to get there as priorities allow.

[LeeDr]

While discussing another issue, I think it was @pmuellr that mentioned that we really shouldn't delete alerts saved objects through the saved object API because it orphaned the task manager tasks related to it. So to really "clean" or "tear-down" after a test we should include things outside the .kibana index. Maybe we develop a function or service that takes all these things into account like;

For each space:

• find any alerts and delete them using the proper API so that it cleans up any tasks related to them
• for other types of saved objects, use the proper APIs to remove them
• for any remaining saved objects, use the saved object API to delete them (like the kibanaServer.savedObjects.cleanStandardList() method currently does

[pmuellr]

While discussing another issue, I think it was @pmuellr that mentioned that we really shouldn't delete alerts saved objects through the saved object API because it orphaned the task manager tasks related to it.

correct. We also have a parallel set of APIs for connectors - they can be deleted as plain old SO's today, but that could change in the future, so probably connectors should be looped through and deleted through their APIs as well.

For each space:

• find any alerts and delete them using the proper API so that it cleans up any tasks related to them
• for other types of saved objects, use the proper APIs to remove them
• for any remaining saved objects, use the saved object API to delete them (like the kibanaServer.savedObjects.cleanStandardList() method currently does

Or, for each space, for each saved object, if it's a "special" type call it's special delete, otherwise call the SO delete. I believe we have or will have bulk delete for rules (maybe not connectors), and plain old SO's as well, so probably want to batch those up instead of deleting one at a time.

[TinaHeiligers]

we have or will have bulk delete

@pmuellr there is a bulkDelete saved objects API in main: #139680.
I also added docs for that: #141164

[watson]

This bug report hasn't received any updates for a year. Can someone please verify if this is still relevant?