File upload: Investigate cleaning up /temp directory more proactively through app.

[kcondon]

Recently we found the /temp directory under the files storage directory was consuming a large amount of space and seemed to be in part related to failed file uploads and ingests.

Can we investigate cleaning this up more proactively so we don't use up a lot of disk space?

We have put a cleaner job in place in the meantime for files older than 1 day.

[bencomp]

Definitely related to #2125, as the caching may have caused this. There may be a difference between the directory for uploaded files before they are saved and the temporary directory in which processing of files takes place.

[tomck]

While a cleaner job works, I've noticed that after files have been moved from the first state, currently-being-uploaded, to the second state, uploaded-but-not-saved, they lose the 'tmp' and 'upload' head and tail. But, after being saved, they're copied (not moved) to their destination directory, so they sit in temp taking up space. I would think at least files that have been successfully placed in their final location would be automatically removed.

[pdurbin]

This is related to #2125 and #2848.

[donsizemore]

Am I reading correctly that old files in files.dir/temp should be safe to delete? By old, I mean a month or more.

[kcondon]

Don, my experience with watching this directory is that files that are successfully uploaded and saved are removed from that directory and files that remain were orphaned due to some failure. I am not sure about the behavior tomck is describing -I have not scrutinized it that carefully but I think old files are safe to delete. @landreev may have additional info.

[mheppler]

@kcondon in #5089 @qqmyers suggests his PR #5091 which was merged in 4.9.3 is "probably" a solution for this issue.

Working on behalf of TDL, I've found one case for shapefile zips where a successful upload leaves temporary files in the defined temp directory and several ways that user actions to delete or cancel when uploading can cause temp file to remain. I've gone through the code and have changes to submit.

The only cases I'm aware of where 'persistent' temp files can still be created would be 1) where network errors break the communication with the browser and neither a save or cancel is ever received, and 2) some code that writes directly to subdirs under /tmp (eg. some of the R code) which is nominally cleaned up by the operating system.

[pdurbin]

Related:

• feat(upload): make upload file storage path configurable #6656 #8983

[cmbz]

To focus on the most important features and bugs, we are closing issues created before 2020 (version 5.0) that are not new feature requests with the label 'Type: Feature'.

If you created this issue and you feel the team should revisit this decision, please reopen the issue and leave a comment.