Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Storage] Make sky storage delete parallel #1990

Closed
romilbhardwaj opened this issue May 27, 2023 · 2 comments
Closed

[Storage] Make sky storage delete parallel #1990

romilbhardwaj opened this issue May 27, 2023 · 2 comments
Labels
good first issue Good for newcomers
Milestone
Storage

Comments

@romilbhardwaj
Copy link
Collaborator

Currently sky storage delete sequentially deletes storage objects, which can take a long time if there are many storage objects:

skypilot/sky/cli.py

Lines 3223 to 3224 in f4d29ba

for name in names:
sky.storage_delete(name)

We should see if this can be run in parallel.
@romilbhardwaj romilbhardwaj added the good first issue Good for newcomers label May 27, 2023
@romilbhardwaj romilbhardwaj added this to the Storage milestone May 27, 2023
@Iamhero337
Copy link

Hey, Can you guide me a little What do I have to do here?

ksasi added a commit to ksasi/skypilot that referenced this issue Jun 9, 2023
@ksasi
Fix for issue - [Storage] Make sky storage delete parallel skypilot-o…
b097c9c 
…rg#1990
ksasi added a commit to ksasi/skypilot that referenced this issue Jun 11, 2023
@ksasi
[Storage] Make sky storage delete parallel skypilot-org#1990 skypilot…
7b0f4ca 
…-org#2058
romilbhardwaj pushed a commit that referenced this issue Jun 28, 2023
@ksasi
[Storage] Make sky storage delete parallel #1990 (#2058)
beac2d5 
* Fix for issue - [Storage] Make sky storage delete parallel #1990

* [Storage] Make sky storage delete parallel #1990 #2058

* Updated test for deleting multiple buckets

* Fix for issue #2015

* Revert "Fix for issue #2015"

This reverts commit c28e4ec.

* Moved storage object creation to a fixture (tmp_multiple_scratch_storage_obj)

* Updated test_multiple_buckets_creation_and_deletion, tmp_multiple_scratch_storage_obj to account for multiple store types

* Added pytest.mark.xdist_group to test_multiple_buckets_creation_and_deletion test to run all parameterized tests serially
@romilbhardwaj
Copy link
Collaborator Author

Closed by #2058.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
good first issue Good for newcomers
Projects
None yet
Milestone
Storage
Development

No branches or pull requests
2 participants
@romilbhardwaj @Iamhero337