-
Notifications
You must be signed in to change notification settings - Fork 72
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pipeline fails for large studies #236
Comments
alexblaessle
pushed a commit
to alexblaessle/fetchngs
that referenced
this issue
Nov 7, 2023
Prefetching large studies resulted in error stated in issue nf-core#236 Resolved this by handing over text file with filepaths instead of list of filepaths. Rewrote sra_merge_samplesheet accordingly. Tested with study with ~25k samples.
10 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description of the bug
Trying to download a SRP project with 25000 crashes the pipeline at SRA_MERGE_SAMPLESHEET
The reason for this is that the list of files handed over is too long for bash commands. Probably chunking or handling it by reading in a list of files in a python script is probably a good way to circumvent this.
Command used and terminal output
Relevant files
No response
System information
NF version: 23.04.2
Hardware: HPC
Executor: local
Container: Singularity
OS: CentOS
Version: 1.11
The text was updated successfully, but these errors were encountered: