Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Concurrent downloads of remote media can lead to orphaned files in storage providers. #8692

Open
erikjohnston opened this issue Oct 30, 2020 · 0 comments
Labels
z-bug (Deprecated Label) z-p3 (Deprecated Label)

Comments

@erikjohnston
Copy link
Member

When running multiple media workers it is possible to end up with both fetching and persisting the same media. #8682 fixed it such that doing so a) didn't throw an error to the client and b) deleted the duplicated media from disk, however by this point the media is already queued up to be uploaded to s3.

We could change it to only upload to storage providers after persisting to the DB, however that runs the risk of having media in the DB that hasn't been uploaded to a storage provider if e.g. the worker dies half way through upload.

I think the "fix" here is to add a delete_file function to the storage provider interface that can be called to delete a file (or cancel its upload if it hasn't finished uploading).

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
z-bug (Deprecated Label) z-p3 (Deprecated Label)
Projects
None yet
Development

No branches or pull requests

1 participant