nf-core should support private repos that are not on GitHub

[stevekm]

Description of the bug

as per the details here #2406

and the docs here https://nf-co.re/tools#private-remote-repositories

currently private custom modules repo's only work if you are hosting them on GitHub

There are tons of remote git hosting platforms, it seems really crazy that the nf-core tools would be hard-coded to only support GitHub

I confirmed this bug by attempting to pull from a public GitHub repo with custom modules

nf-core --verbose modules --git-remote git@github.com:mskcc-omics-workflows/modules.git list remote

It works just fine.

I then made an exact clone of the repo, hosted it on a different git remote platform, and I get the error

nf-core --verbose modules --git-remote ssh://git-codecommit....modules list remote



                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 2.13.1 - https://nf-co.re


Cloning from '/v1/repos/mskcc-modules' (ssh://git-codecommit...modules) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Waiting for responseDEBUG    Popen(['git', 'version'], cwd=/Users/steve/projects/test-custom-pipeline-modules, stdin=None, shell=False, universal_newlines=False)                                    cmd.py:1253
DEBUG    Popen(['git', 'clone', '-v', '--progress', '--', 'ssh://git-codecommit...modules', '/v1/repos/mskcc-modules'],                      cmd.py:1253
         cwd=/Users/steve/projects/test-custom-pipeline-modules, stdin=None, shell=False, universal_newlines=True)
DEBUG    AutoInterrupt wait stderr: b''                                                                                                                                            cmd.py:833
CRITICAL Failed to clone from the remote: `ssh://git-codecommit...modules`                                                               __main__.py:740

The fact that only GitHub is supported (?) seems weird and arbitrarily restrictive. It also does not seem to be mentioned in the documentation. Can we please fix this so that its not restricted to only one git platform? Thanks.

Command used and terminal output

No response

System information

No response

[ewels]

Hi @stevekm,

We built and continue to build the nf-core tooling primarily for the nf-core community and it's pipelines, which reside only on GitHub. However, we are doing our best to slowly move the codebase towards more general use for the wider Nextflow community. This includes providing support for other Git providers.

currently private custom modules repo's only work if you are hosting them on GitHub
The fact that only GitHub is supported (?) seems weird and arbitrarily restrictive.

Your assertions here are incorrect. We explicitly do support other Git providers for modules. In fact we have code tests in the CI that work with an nf-core organisation on GitLab specifically to ensure this compatibility: https://gitlab.com/nf-core

If you've found a case where this doesn't work for whatever reason, please open an issue specifically for that error with a reproducible example. Or better still, submit a PR to fix the problem.

Issue #2406 that you linked to is about downloading pipelines, not working with modules. That's a much older part of the nf-core/tools codebase and much of it dates to before we aimed to support anything other than nf-core pipelines. It also includes complexities such as fetching the nf-core shared configs which are not generalisable. We could look into extending the relative recent --tower functionality that uses git clone if needed, but let's take that discussion on that thread.

It's probably not surprising that the vast majority of active contributors to nf-core, especially the core tooling, are people who are use nf-core pipelines in their day to day work. We would love more contributors to nf-core/tools who work with pipelines outside of that scope. You're welcome to join us :)

[stevekm]

thanks for the clarification!