-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
--conda-create-envs-only results in errors - CreateCondaEnvironmentException and container creation failed #68
Comments
@Redmar-van-den-Berg this is a tricky issue, it's a case of Snakemake wanting to create a Singularity image using Anaconda inside the image to build an environment and it's trying to mount the somalier data directory as part of the process. Can you tell me if any part of the path Perhaps you could run realpath ~ and tell us if the path it returns is the same as username@login004:~$ pwd
/home/username
username@login004:~$ realpath ~
/data/user/home/username Knowing that info would be helpful in determining if moving the repo to a non-home directory location would help. In the past I've seen issues with Singularity and mounting paths in a user's home directory on top of the typical default mount that Singularity does where it auto-mounts a users home directory automatically. It shouldn't be an issue but for reasons I never figured out in the past it ended up being resolved by working out of a non-home directory location. I'm not sure that this is the case here but might be something to try. |
As far as I can tell, my home directory is a regular path: $ realpath ~
/home/username Rather than trying to debug this, would it be possible for QuaC to switch over to using singularity images instead of conda? Conda environments tend to become uninstallable over time, which means pipelines using them are not fully reproducible. I tend to use the Biocontainers hosted on quay.io, which are built automatically for all conda packages (e.g. https://quay.io/repository/biocontainers/goleft, which is available as |
That's a good idea, I know this is the preference of setup for Nextflow NF-Core based pipelines and @ManavalanG had dealt with a scenario in #52 that had prompted the thought to move move away completely from conda + singularity to just singularity. I'm reviewing #70 right now as part of this effort. I'm still really perplexed by this issue where Singularity can't mount a directory into the container. @Redmar-van-den-Berg if you have time before #70 is reviewed and merged (and this is totally at your discretion b/c it's more debugging and more potentially unnecessary work on you) could you sanity check the mounting setup for Singularity? If you have time could you quickly check that the The command would be cd /home/username/devel/quac
singularity run --cleanenv --bind /home/username/devel/quac/scratch/tmp/quac/tmp:/tmp --bind /home/username/devel/quac/.test/ngs-data/test_project/analysis,/home/username/devel/quac/scratch/tmp/quac/logs,/home/username/devel/quac/scratch/tmp/quac/results/test_project_2samples_exome-no_priorQC/analysis,data/external/dependency_datasets/somalier,data/external/dependency_datasets/verifyBamID,data/external/dependency_datasets/reference_genome,/home/username/devel/quac/.test/configs/no_priorQC,/home/username/devel/quac/configs/quac_watch/exome_quac_watch_config.yaml,data/external/dependency_datasets/verifyBamID/exome docker://sylabsio/lolcow The only thing I could think is something is preventing the singularity command from correctly binding relative paths, in particular
then I have no idea the issue and hope that just moving strictly to containers resolves whatever issue is coming up with the combo of conda + singularity 😂 . I completely understand if your preference is to just wait for the PR switching to full containers is merged and test that setup directly. |
@Redmar-van-den-Berg I would like to inform that we have refactored QuaC to move away from creation of conda environment within singularity containers. QuaC now simply depends on webserver-hosted container images, retrieves them automatically as part of the pipeline and uses them for snakemake-initiated jobs (#70). I hope this makes it easier to execute the pipeline. Please let us know if you run into any issues :) |
The issue still happens for me, even if I run the latest version of the pipeline that does not use Conda. I think it is related to the snakemake command attempting to bind mount a whole bunch of folders that do not exist: --singularity-args '--cleanenv --bind /data/lumc/devel/quac/data/quac/tmp:/tmp --bind /data/lumc/devel/quac/configs/quac_watch/wgs_quac_watch_config.yaml,data/external/dependency_datasets/somalier,/data/lumc/devel/quac/.test/configs/no_priorQC,data/external/dependency_datasets/reference_genome,data/external/dependency_datasets/verifyBamID/exome,/data/lumc/devel/quac/data/quac/results/test_project_2samples_wgs-no_priorQC/analysis,/data/lumc/devel/quac/data/quac/logs,/data/lumc/devel/quac/.test/ngs-data/test_project/analysis,data/external/dependency_datasets/verifyBamID |
@Redmar-van-den-Berg If you are still running into issues, could you provide the following info? It would greatly help us debug the issue.
PS- Please be sure to use |
Copy/pasting from source: openjournals/joss-reviews#5313 (comment)
Thanks for pointing me to the script, I was able to download the required reference files!
However, I run into a crash when I try to run the pipeline itself, following along to the documentation.
The text was updated successfully, but these errors were encountered: