-
Notifications
You must be signed in to change notification settings - Fork 659
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cleanup of work directory without losing command logs #668
Comments
Your proposal is to delete all task files but the |
My proposal is to replace all task files with empty file stubs, to show that they were created. |
A follow up on this, here is a
I saw that Nextflow has a built-in 'clean' feature though I was not clear on what exactly it was cleaning. Some of the items listed here might be beyond the scope of Nextflow, though in general some kind of reclamation of the storage space used by unnecessary files in the 'work' subdirs without losing complete execution records is my end goal. |
Closing this in favour of #452. |
After completing a pipeline, the 'work' directory tends to consume a lot of storage space. But deleting the work directory is not ideal, because it also contains the files such as '.command.run', '.command.log', etc., along with the files produced by your processes, which are useful to preserve for debugging, troubleshooting, record keeping, etc. It would be very useful to have some sort of feature that allows you to clear out the storage used by the 'work' directory without losing these items.
I came up with a script here to attempt to implement this by:
removing 'work' subdirectories not included in the 'trace' file (i.e. the results of the most recent pipeline execution)
replacing all files that do not match the pattern
'.command*'
or'.exitcode'
with "file stubs" (empty files of the same name)However a native Nextflow implementation of something similar might be helpful.
Alternatively, these aspects could be instead integrated into the 'trace' output (stdout & stderr logs, directory structure & contents, etc.).
The text was updated successfully, but these errors were encountered: