Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nf-core download does not save some singularity images with the correct file name #2644

Closed
Aratz opened this issue Jan 9, 2024 · 2 comments
Assignees
Labels
bug Something isn't working download nf-core download

Comments

@Aratz
Copy link
Contributor

Aratz commented Jan 9, 2024

Description of the bug

This is a follow up of nf-core/demultiplex#164. I think I've nailed down the problem to nf-core/tools not saving singularity images with the right name when using nf-core download:

If the container is specified with e.g. container "nf-core/bclconvert:4.2.4", tools will download it as nf-core-bclconvert-4.2.4.img but nextflow expects quay.io-nf-core-bclconvert-4.2.4.img.

Command used and terminal output

$ nf-core download demultiplex

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 2.11.1 - https://nf-co.re


WARNING  Could not find GitHub authentication token. Some API requests may fail.                                                                                                                                   
? Select release / branch: 1.4.0  [release]
? Include the nf-core's default institutional configuration files into the download? Yes

In addition to the pipeline code, this tool can download software containers.
? Download software container images: singularity

Nextflow and nf-core can use an environment variable called $NXF_SINGULARITY_CACHEDIR that is a path to a directory where remote Singularity images are stored. This allows downloaded images to be cached in a 
central location.
? Define $NXF_SINGULARITY_CACHEDIR for a shared Singularity image download folder? [y/n]: n

If transferring the downloaded files to another system, it can be convenient to have everything compressed in a single file.
This is not recommended when downloading Singularity images, as it can take a long time and saves very little space.
? Choose compression type: none
INFO     Saving 'nf-core/demultiplex'                                                                                                                                                                              
          Pipeline revision: '1.4.0'                                                                                                                                                                               
          Use containers: 'singularity'                                                                                                                                                                            
          Container library: 'quay.io'                                                                                                                                                                             
          Output directory: 'nf-core-demultiplex_1.4.0'                                                                                                                                                            
          Include default institutional configuration: 'True'                                                                                                                                                      
INFO     Downloading centralised configs from GitHub                                                                                                                                                               
INFO     Downloading workflow files from GitHub                                                                                                                                                                    
INFO     Processing workflow revision 1.4.0, found 10 container images in total.                                                                                                                                   
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% • 10/10 completed
$ cd nf-core-demultiplex_1.4.0
$ ls singularity-images
depot.galaxyproject.org-singularity-falco-1.2.1--h867801b_3.img
depot.galaxyproject.org-singularity-fastp-0.23.4--h5f740d0_0.img
depot.galaxyproject.org-singularity-fqtk-0.2.1--h9f5acd7_0.img
depot.galaxyproject.org-singularity-multiqc-1.17--pyhdfd78af_0.img
depot.galaxyproject.org-singularity-multiqc-1.18--pyhdfd78af_0.img
depot.galaxyproject.org-singularity-sgdemux-1.1.1--ha982bd6_0.img
depot.galaxyproject.org-singularity-ubuntu-20.04.img
nf-core-bases2fastq-1.1.0.img
nf-core-bcl2fastq-2.20.0.422.img
nf-core-bclconvert-4.2.4.img
$ NXF_ANSI_LOG=false NXF_SINGULARITY_CACHEDIR=singularity-images nextflow run 1_4_0/main.nf -profile singularity,test --outdir /tmp/test_demux
N E X T F L O W  ~  version 23.04.3
Launching `1_4_0/main.nf` [special_neumann] DSL2 - revision: a8c81121bd


------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/demultiplex v1.4.0
------------------------------------------------------
Core Nextflow options
  runName                   : special_neumann
  containerEngine           : singularity
  launchDir                 : /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0
  workDir                   : /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0/work
  projectDir                : /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0/1_4_0
  userName                  : adrien
  profile                   : singularity,test
  configFiles               : 

Workflow options
  skip_tools                : []

Input/output options
  input                     : https://raw.githubusercontent.com/nf-core/test-datasets/demultiplex/samplesheet/1.3.0/flowcell_input.csv
  outdir                    : /tmp/test_demux

Demultiplexing options
  demultiplexer             : bclconvert

Institutional config options
  custom_config_base        : /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0/1_4_0/../configs/
  config_profile_name       : Test profile
  config_profile_description: Minimal test dataset to check pipeline function

Max job request options
  max_cpus                  : 2
  max_memory                : 6.GB
  max_time                  : 6.h

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/demultiplex for your analysis please cite:

* The pipeline
  https://doi.org/10.5281/zenodo.7153103

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/nf-core/demultiplex/blob/master/CITATIONS.md

Pulling Singularity image docker://quay.io/nf-core/bclconvert:4.2.4 [cache /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0/singularity-images/quay.io-nf-core-bclconvert-4.2.4.img]
[28/c0e23c] Submitted process > NFCORE_DEMULTIPLEX:DEMULTIPLEX:BCL_DEMULTIPLEX:BCLCONVERT (220422_M11111_0222_000000000-K9H97.1)
...

System information

nf-core, version 2.11.1
nextflow, version 23.04.3 build 5875
nf-core/demultixplex, version 1.4.0

@Aratz Aratz added the bug Something isn't working label Jan 9, 2024
@Aratz Aratz self-assigned this Jan 9, 2024
@MatthiasZepper
Copy link
Member

MatthiasZepper commented Feb 15, 2024

Also see #2751 and the lengthy discussion in Slack.

Bottom line: It is a feature, not a bug, but went out of sync with updates to Nextflow. As of now, we can only reintroduce the library or one needs to manually rename some images :-\

@MatthiasZepper
Copy link
Member

Should be fixed with the nf-core tools release 2.13.1 today! 👌🎉

(I would still appreciate seeing an update to Nextflow as well at some point, but for now symlinks are created so that the images are found nonetheless.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working download nf-core download
Projects
None yet
Development

No branches or pull requests

3 participants