From 18ce08033aecbb6810b5c9fa23fb43f65718bd74 Mon Sep 17 00:00:00 2001 From: Erik Danielsson Date: Mon, 2 Aug 2021 15:04:50 +0200 Subject: [PATCH 001/110] Run rst2myst and update conf.py --- .../api/{bump_version.rst => bump_version.md} | 5 ++-- docs/api/_src/api/{create.rst => create.md} | 5 ++-- .../_src/api/{download.rst => download.md} | 5 ++-- docs/api/_src/api/index.md | 9 +++++++ docs/api/_src/api/index.rst | 9 ------- docs/api/_src/api/{launch.rst => launch.md} | 5 ++-- .../_src/api/{licences.rst => licences.md} | 5 ++-- docs/api/_src/api/{lint.rst => lint.md} | 11 ++++++--- docs/api/_src/api/{list.rst => list.md} | 5 ++-- docs/api/_src/api/{modules.rst => modules.md} | 5 ++-- docs/api/_src/api/{schema.rst => schema.md} | 5 ++-- docs/api/_src/api/{sync.rst => sync.md} | 5 ++-- docs/api/_src/api/{utils.rst => utils.md} | 5 ++-- docs/api/_src/conf.py | 2 +- docs/api/_src/index.md | 24 +++++++++++++++++++ docs/api/_src/index.rst | 24 ------------------- docs/api/_src/module_lint_tests/index.md | 9 +++++++ .../_src/module_lint_tests/modules_command.md | 7 ++++++ ...awsfulltest.rst => actions_awsfulltest.md} | 5 ++-- ...actions_awstest.rst => actions_awstest.md} | 5 ++-- .../{actions_ci.rst => actions_ci.md} | 5 ++-- ...ation.rst => actions_schema_validation.md} | 5 ++-- .../{files_exist.rst => files_exist.md} | 6 ++--- ...files_unchanged.rst => files_unchanged.md} | 5 ++-- docs/api/_src/pipeline_lint_tests/index.md | 9 +++++++ docs/api/_src/pipeline_lint_tests/index.rst | 9 ------- .../{merge_markers.rst => merge_markers.md} | 5 ++-- .../{modules_json.rst => modules_json.md} | 5 ++-- ...nextflow_config.rst => nextflow_config.md} | 5 ++-- ...tions.rst => pipeline_name_conventions.md} | 5 ++-- .../{pipeline_todos.rst => pipeline_todos.md} | 5 ++-- .../{readme.rst => readme.md} | 5 ++-- ..._description.rst => schema_description.md} | 5 ++-- .../{schema_lint.rst => schema_lint.md} | 5 ++-- .../{schema_params.rst => schema_params.md} | 5 ++-- ...mplate_strings.rst => template_strings.md} | 5 ++-- ...consistency.rst => version_consistency.md} | 5 ++-- 37 files changed, 148 insertions(+), 101 deletions(-) rename docs/api/_src/api/{bump_version.rst => bump_version.md} (73%) rename docs/api/_src/api/{create.rst => create.md} (76%) rename docs/api/_src/api/{download.rst => download.md} (75%) create mode 100644 docs/api/_src/api/index.md delete mode 100644 docs/api/_src/api/index.rst rename docs/api/_src/api/{launch.rst => launch.md} (76%) rename docs/api/_src/api/{licences.rst => licences.md} (75%) rename docs/api/_src/api/{lint.rst => lint.md} (63%) rename docs/api/_src/api/{list.rst => list.md} (76%) rename docs/api/_src/api/{modules.rst => modules.md} (75%) rename docs/api/_src/api/{schema.rst => schema.md} (76%) rename docs/api/_src/api/{sync.rst => sync.md} (76%) rename docs/api/_src/api/{utils.rst => utils.md} (76%) create mode 100644 docs/api/_src/index.md delete mode 100644 docs/api/_src/index.rst create mode 100644 docs/api/_src/module_lint_tests/index.md create mode 100644 docs/api/_src/module_lint_tests/modules_command.md rename docs/api/_src/pipeline_lint_tests/{actions_awsfulltest.rst => actions_awsfulltest.md} (61%) rename docs/api/_src/pipeline_lint_tests/{actions_awstest.rst => actions_awstest.md} (62%) rename docs/api/_src/pipeline_lint_tests/{actions_ci.rst => actions_ci.md} (63%) rename docs/api/_src/pipeline_lint_tests/{actions_schema_validation.rst => actions_schema_validation.md} (57%) rename docs/api/_src/pipeline_lint_tests/{files_exist.rst => files_exist.md} (63%) rename docs/api/_src/pipeline_lint_tests/{files_unchanged.rst => files_unchanged.md} (62%) create mode 100644 docs/api/_src/pipeline_lint_tests/index.md delete mode 100644 docs/api/_src/pipeline_lint_tests/index.rst rename docs/api/_src/pipeline_lint_tests/{merge_markers.rst => merge_markers.md} (62%) rename docs/api/_src/pipeline_lint_tests/{modules_json.rst => modules_json.md} (60%) rename docs/api/_src/pipeline_lint_tests/{nextflow_config.rst => nextflow_config.md} (62%) rename docs/api/_src/pipeline_lint_tests/{pipeline_name_conventions.rst => pipeline_name_conventions.md} (57%) rename docs/api/_src/pipeline_lint_tests/{pipeline_todos.rst => pipeline_todos.md} (62%) rename docs/api/_src/pipeline_lint_tests/{readme.rst => readme.md} (64%) rename docs/api/_src/pipeline_lint_tests/{schema_description.rst => schema_description.md} (61%) rename docs/api/_src/pipeline_lint_tests/{schema_lint.rst => schema_lint.md} (63%) rename docs/api/_src/pipeline_lint_tests/{schema_params.rst => schema_params.md} (62%) rename docs/api/_src/pipeline_lint_tests/{template_strings.rst => template_strings.md} (61%) rename docs/api/_src/pipeline_lint_tests/{version_consistency.rst => version_consistency.md} (61%) diff --git a/docs/api/_src/api/bump_version.rst b/docs/api/_src/api/bump_version.md similarity index 73% rename from docs/api/_src/api/bump_version.rst rename to docs/api/_src/api/bump_version.md index aca9f76210..54c3774408 100644 --- a/docs/api/_src/api/bump_version.rst +++ b/docs/api/_src/api/bump_version.md @@ -1,8 +1,9 @@ -nf_core.bump_version -==================== +# nf_core.bump_version +```{eval-rst} .. automodule:: nf_core.bump_version :members: :undoc-members: :show-inheritance: :private-members: +``` diff --git a/docs/api/_src/api/create.rst b/docs/api/_src/api/create.md similarity index 76% rename from docs/api/_src/api/create.rst rename to docs/api/_src/api/create.md index 10803a8313..5d5f6a62d1 100644 --- a/docs/api/_src/api/create.rst +++ b/docs/api/_src/api/create.md @@ -1,8 +1,9 @@ -nf_core.create -============== +# nf_core.create +```{eval-rst} .. automodule:: nf_core.create :members: :undoc-members: :show-inheritance: :private-members: +``` diff --git a/docs/api/_src/api/download.rst b/docs/api/_src/api/download.md similarity index 75% rename from docs/api/_src/api/download.rst rename to docs/api/_src/api/download.md index b7f8940f85..18ab51376e 100644 --- a/docs/api/_src/api/download.rst +++ b/docs/api/_src/api/download.md @@ -1,8 +1,9 @@ -nf_core.download -================ +# nf_core.download +```{eval-rst} .. automodule:: nf_core.download :members: :undoc-members: :show-inheritance: :private-members: +``` diff --git a/docs/api/_src/api/index.md b/docs/api/_src/api/index.md new file mode 100644 index 0000000000..9050280fd3 --- /dev/null +++ b/docs/api/_src/api/index.md @@ -0,0 +1,9 @@ +# API Reference + +```{toctree} +:caption: 'Tests:' +:glob: true +:maxdepth: 2 + +* +``` diff --git a/docs/api/_src/api/index.rst b/docs/api/_src/api/index.rst deleted file mode 100644 index 4ddec5ecbe..0000000000 --- a/docs/api/_src/api/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -API Reference -============= - -.. toctree:: - :maxdepth: 2 - :caption: Tests: - :glob: - - * diff --git a/docs/api/_src/api/launch.rst b/docs/api/_src/api/launch.md similarity index 76% rename from docs/api/_src/api/launch.rst rename to docs/api/_src/api/launch.md index 060d97009e..eef777ca87 100644 --- a/docs/api/_src/api/launch.rst +++ b/docs/api/_src/api/launch.md @@ -1,8 +1,9 @@ -nf_core.launch -============== +# nf_core.launch +```{eval-rst} .. automodule:: nf_core.launch :members: :undoc-members: :show-inheritance: :private-members: +``` diff --git a/docs/api/_src/api/licences.rst b/docs/api/_src/api/licences.md similarity index 75% rename from docs/api/_src/api/licences.rst rename to docs/api/_src/api/licences.md index 8d061741d9..95b5f9768b 100644 --- a/docs/api/_src/api/licences.rst +++ b/docs/api/_src/api/licences.md @@ -1,8 +1,9 @@ -nf_core.licences -================ +# nf_core.licences +```{eval-rst} .. automodule:: nf_core.licences :members: :undoc-members: :show-inheritance: :private-members: +``` diff --git a/docs/api/_src/api/lint.rst b/docs/api/_src/api/lint.md similarity index 63% rename from docs/api/_src/api/lint.rst rename to docs/api/_src/api/lint.md index 8e0cfdff97..a0e3a6d5c9 100644 --- a/docs/api/_src/api/lint.rst +++ b/docs/api/_src/api/lint.md @@ -1,14 +1,19 @@ -nf_core.lint -============ +# nf_core.lint -.. seealso:: See the `Lint Tests <../lint_tests/index.html>`_ docs for information about specific linting functions. +:::{seealso} +See the [Lint Tests](../lint_tests/index.html) docs for information about specific linting functions. +::: +```{eval-rst} .. automodule:: nf_core.lint :members: run_linting :undoc-members: :show-inheritance: +``` +```{eval-rst} .. autoclass:: nf_core.lint.PipelineLint :members: _lint_pipeline :private-members: _print_results, _get_results_md, _save_json_results, _wrap_quotes, _strip_ansi_codes :show-inheritance: +``` diff --git a/docs/api/_src/api/list.rst b/docs/api/_src/api/list.md similarity index 76% rename from docs/api/_src/api/list.rst rename to docs/api/_src/api/list.md index ea4a406ffa..35c819bc56 100644 --- a/docs/api/_src/api/list.rst +++ b/docs/api/_src/api/list.md @@ -1,8 +1,9 @@ -nf_core.list -============ +# nf_core.list +```{eval-rst} .. automodule:: nf_core.list :members: :undoc-members: :show-inheritance: :private-members: +``` diff --git a/docs/api/_src/api/modules.rst b/docs/api/_src/api/modules.md similarity index 75% rename from docs/api/_src/api/modules.rst rename to docs/api/_src/api/modules.md index 6bb6e0547d..e3f1e39bed 100644 --- a/docs/api/_src/api/modules.rst +++ b/docs/api/_src/api/modules.md @@ -1,8 +1,9 @@ -nf_core.modules -=============== +# nf_core.modules +```{eval-rst} .. automodule:: nf_core.modules :members: :undoc-members: :show-inheritance: :private-members: +``` diff --git a/docs/api/_src/api/schema.rst b/docs/api/_src/api/schema.md similarity index 76% rename from docs/api/_src/api/schema.rst rename to docs/api/_src/api/schema.md index d2d346c28c..a702d28058 100644 --- a/docs/api/_src/api/schema.rst +++ b/docs/api/_src/api/schema.md @@ -1,8 +1,9 @@ -nf_core.schema -============== +# nf_core.schema +```{eval-rst} .. automodule:: nf_core.schema :members: :undoc-members: :show-inheritance: :private-members: +``` diff --git a/docs/api/_src/api/sync.rst b/docs/api/_src/api/sync.md similarity index 76% rename from docs/api/_src/api/sync.rst rename to docs/api/_src/api/sync.md index edb9c28e31..8cc02209da 100644 --- a/docs/api/_src/api/sync.rst +++ b/docs/api/_src/api/sync.md @@ -1,8 +1,9 @@ -nf_core.sync -============== +# nf_core.sync +```{eval-rst} .. automodule:: nf_core.sync :members: :undoc-members: :show-inheritance: :private-members: +``` diff --git a/docs/api/_src/api/utils.rst b/docs/api/_src/api/utils.md similarity index 76% rename from docs/api/_src/api/utils.rst rename to docs/api/_src/api/utils.md index f3077f4cbb..1353f97ef5 100644 --- a/docs/api/_src/api/utils.rst +++ b/docs/api/_src/api/utils.md @@ -1,8 +1,9 @@ -nf_core.utils -============= +# nf_core.utils +```{eval-rst} .. automodule:: nf_core.utils :members: :undoc-members: :show-inheritance: :private-members: +``` diff --git a/docs/api/_src/conf.py b/docs/api/_src/conf.py index 36807cf340..4d8ae661d5 100644 --- a/docs/api/_src/conf.py +++ b/docs/api/_src/conf.py @@ -39,7 +39,7 @@ # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions = ["sphinx.ext.autodoc", "sphinx.ext.napoleon"] +extensions = ["myst_parser", "sphinx.ext.autodoc", "sphinx.ext.napoleon"] # Add any paths that contain templates here, relative to this directory. templates_path = ["./_templates"] diff --git a/docs/api/_src/index.md b/docs/api/_src/index.md new file mode 100644 index 0000000000..d7ff705e22 --- /dev/null +++ b/docs/api/_src/index.md @@ -0,0 +1,24 @@ +# nf-core/tools documentation + +```{toctree} +:caption: 'Contents:' +:glob: true +:hidden: true +:maxdepth: 2 + +pipeline_lint_tests/index.rst +module_lint_tests/index.rst +api/index.rst +``` + +This documentation is for the `nf-core/tools` package. + +Primarily, it describes the different [code lint tests](lint_tests/index.html) +run by `nf-core lint` (typically visited by a developer when their pipeline fails a given +test), and also reference for the `nf_core` [Python package API](api/index.html). + +# Indices and tables + +- {ref}`genindex` +- {ref}`modindex` +- {ref}`search` diff --git a/docs/api/_src/index.rst b/docs/api/_src/index.rst deleted file mode 100644 index 8561e2757e..0000000000 --- a/docs/api/_src/index.rst +++ /dev/null @@ -1,24 +0,0 @@ -nf-core/tools documentation -=========================== - -.. toctree:: - :hidden: - :maxdepth: 2 - :caption: Contents: - :glob: - - pipeline_lint_tests/index.rst - api/index.rst - -This documentation is for the ``nf-core/tools`` package. - -Primarily, it describes the different `code lint tests `_ -run by ``nf-core lint`` (typically visited by a developer when their pipeline fails a given -test), and also reference for the ``nf_core`` `Python package API `_. - -Indices and tables -================== - -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` diff --git a/docs/api/_src/module_lint_tests/index.md b/docs/api/_src/module_lint_tests/index.md new file mode 100644 index 0000000000..f889abf73c --- /dev/null +++ b/docs/api/_src/module_lint_tests/index.md @@ -0,0 +1,9 @@ +# Module lint tests + +```{toctree} +:caption: 'Tests:' +:glob: true +:maxdepth: 2 + +* +``` diff --git a/docs/api/_src/module_lint_tests/modules_command.md b/docs/api/_src/module_lint_tests/modules_command.md new file mode 100644 index 0000000000..642d3591a7 --- /dev/null +++ b/docs/api/_src/module_lint_tests/modules_command.md @@ -0,0 +1,7 @@ +ModulesCommand +=============== +This is the base class for all `nf-core modules` commands +```{eval-rst} +.. automodule:: nf_core.modules.module_utils + :members: +``` diff --git a/docs/api/_src/pipeline_lint_tests/actions_awsfulltest.rst b/docs/api/_src/pipeline_lint_tests/actions_awsfulltest.md similarity index 61% rename from docs/api/_src/pipeline_lint_tests/actions_awsfulltest.rst rename to docs/api/_src/pipeline_lint_tests/actions_awsfulltest.md index daf414a1b7..c0f0aef5a4 100644 --- a/docs/api/_src/pipeline_lint_tests/actions_awsfulltest.rst +++ b/docs/api/_src/pipeline_lint_tests/actions_awsfulltest.md @@ -1,4 +1,5 @@ -actions_awsfulltest -=================== +# actions_awsfulltest +```{eval-rst} .. automethod:: nf_core.lint.PipelineLint.actions_awsfulltest +``` diff --git a/docs/api/_src/pipeline_lint_tests/actions_awstest.rst b/docs/api/_src/pipeline_lint_tests/actions_awstest.md similarity index 62% rename from docs/api/_src/pipeline_lint_tests/actions_awstest.rst rename to docs/api/_src/pipeline_lint_tests/actions_awstest.md index b27c830285..42441c8ac5 100644 --- a/docs/api/_src/pipeline_lint_tests/actions_awstest.rst +++ b/docs/api/_src/pipeline_lint_tests/actions_awstest.md @@ -1,4 +1,5 @@ -actions_awstest -=============== +# actions_awstest +```{eval-rst} .. automethod:: nf_core.lint.PipelineLint.actions_awstest +``` diff --git a/docs/api/_src/pipeline_lint_tests/actions_ci.rst b/docs/api/_src/pipeline_lint_tests/actions_ci.md similarity index 63% rename from docs/api/_src/pipeline_lint_tests/actions_ci.rst rename to docs/api/_src/pipeline_lint_tests/actions_ci.md index 28bf91cce5..78ea3aea0d 100644 --- a/docs/api/_src/pipeline_lint_tests/actions_ci.rst +++ b/docs/api/_src/pipeline_lint_tests/actions_ci.md @@ -1,4 +1,5 @@ -actions_ci -========== +# actions_ci +```{eval-rst} .. automethod:: nf_core.lint.PipelineLint.actions_ci +``` diff --git a/docs/api/_src/pipeline_lint_tests/actions_schema_validation.rst b/docs/api/_src/pipeline_lint_tests/actions_schema_validation.md similarity index 57% rename from docs/api/_src/pipeline_lint_tests/actions_schema_validation.rst rename to docs/api/_src/pipeline_lint_tests/actions_schema_validation.md index d7d2b4c13d..f0c98eb213 100644 --- a/docs/api/_src/pipeline_lint_tests/actions_schema_validation.rst +++ b/docs/api/_src/pipeline_lint_tests/actions_schema_validation.md @@ -1,4 +1,5 @@ -actions_schema_validation -========================= +# actions_schema_validation +```{eval-rst} .. automethod:: nf_core.lint.PipelineLint.actions_schema_validation +``` diff --git a/docs/api/_src/pipeline_lint_tests/files_exist.rst b/docs/api/_src/pipeline_lint_tests/files_exist.md similarity index 63% rename from docs/api/_src/pipeline_lint_tests/files_exist.rst rename to docs/api/_src/pipeline_lint_tests/files_exist.md index 5d4700af00..309ea62f0b 100644 --- a/docs/api/_src/pipeline_lint_tests/files_exist.rst +++ b/docs/api/_src/pipeline_lint_tests/files_exist.md @@ -1,5 +1,5 @@ -files_exist -=========== +# files_exist +```{eval-rst} .. automethod:: nf_core.lint.PipelineLint.files_exist - +``` diff --git a/docs/api/_src/pipeline_lint_tests/files_unchanged.rst b/docs/api/_src/pipeline_lint_tests/files_unchanged.md similarity index 62% rename from docs/api/_src/pipeline_lint_tests/files_unchanged.rst rename to docs/api/_src/pipeline_lint_tests/files_unchanged.md index 5ec1de6492..2f3b04fe1b 100644 --- a/docs/api/_src/pipeline_lint_tests/files_unchanged.rst +++ b/docs/api/_src/pipeline_lint_tests/files_unchanged.md @@ -1,4 +1,5 @@ -files_unchanged -=============== +# files_unchanged +```{eval-rst} .. automethod:: nf_core.lint.PipelineLint.files_unchanged +``` diff --git a/docs/api/_src/pipeline_lint_tests/index.md b/docs/api/_src/pipeline_lint_tests/index.md new file mode 100644 index 0000000000..d10f8421c5 --- /dev/null +++ b/docs/api/_src/pipeline_lint_tests/index.md @@ -0,0 +1,9 @@ +# Pipline lint tests + +```{toctree} +:caption: 'Tests:' +:glob: true +:maxdepth: 2 + +* +``` diff --git a/docs/api/_src/pipeline_lint_tests/index.rst b/docs/api/_src/pipeline_lint_tests/index.rst deleted file mode 100644 index 6b6d2d9717..0000000000 --- a/docs/api/_src/pipeline_lint_tests/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -Pipline lint tests -============================================ - -.. toctree:: - :maxdepth: 2 - :caption: Tests: - :glob: - - * diff --git a/docs/api/_src/pipeline_lint_tests/merge_markers.rst b/docs/api/_src/pipeline_lint_tests/merge_markers.md similarity index 62% rename from docs/api/_src/pipeline_lint_tests/merge_markers.rst rename to docs/api/_src/pipeline_lint_tests/merge_markers.md index ea5e3be84b..b52fdba1a8 100644 --- a/docs/api/_src/pipeline_lint_tests/merge_markers.rst +++ b/docs/api/_src/pipeline_lint_tests/merge_markers.md @@ -1,4 +1,5 @@ -merge_markers -============== +# merge_markers +```{eval-rst} .. automethod:: nf_core.lint.PipelineLint.merge_markers +``` diff --git a/docs/api/_src/pipeline_lint_tests/modules_json.rst b/docs/api/_src/pipeline_lint_tests/modules_json.md similarity index 60% rename from docs/api/_src/pipeline_lint_tests/modules_json.rst rename to docs/api/_src/pipeline_lint_tests/modules_json.md index 33fb139bb4..0c0ba71a8f 100644 --- a/docs/api/_src/pipeline_lint_tests/modules_json.rst +++ b/docs/api/_src/pipeline_lint_tests/modules_json.md @@ -1,4 +1,5 @@ -nextflow_config -=============== +# nextflow_config +```{eval-rst} .. automethod:: nf_core.lint.PipelineLint.modules_json +``` diff --git a/docs/api/_src/pipeline_lint_tests/nextflow_config.rst b/docs/api/_src/pipeline_lint_tests/nextflow_config.md similarity index 62% rename from docs/api/_src/pipeline_lint_tests/nextflow_config.rst rename to docs/api/_src/pipeline_lint_tests/nextflow_config.md index 68fe8708e7..e23e966143 100644 --- a/docs/api/_src/pipeline_lint_tests/nextflow_config.rst +++ b/docs/api/_src/pipeline_lint_tests/nextflow_config.md @@ -1,4 +1,5 @@ -nextflow_config -=============== +# nextflow_config +```{eval-rst} .. automethod:: nf_core.lint.PipelineLint.nextflow_config +``` diff --git a/docs/api/_src/pipeline_lint_tests/pipeline_name_conventions.rst b/docs/api/_src/pipeline_lint_tests/pipeline_name_conventions.md similarity index 57% rename from docs/api/_src/pipeline_lint_tests/pipeline_name_conventions.rst rename to docs/api/_src/pipeline_lint_tests/pipeline_name_conventions.md index 8a63f9759a..0034319d3d 100644 --- a/docs/api/_src/pipeline_lint_tests/pipeline_name_conventions.rst +++ b/docs/api/_src/pipeline_lint_tests/pipeline_name_conventions.md @@ -1,4 +1,5 @@ -pipeline_name_conventions -========================= +# pipeline_name_conventions +```{eval-rst} .. automethod:: nf_core.lint.PipelineLint.pipeline_name_conventions +``` diff --git a/docs/api/_src/pipeline_lint_tests/pipeline_todos.rst b/docs/api/_src/pipeline_lint_tests/pipeline_todos.md similarity index 62% rename from docs/api/_src/pipeline_lint_tests/pipeline_todos.rst rename to docs/api/_src/pipeline_lint_tests/pipeline_todos.md index 259cc693e2..08e456ea22 100644 --- a/docs/api/_src/pipeline_lint_tests/pipeline_todos.rst +++ b/docs/api/_src/pipeline_lint_tests/pipeline_todos.md @@ -1,4 +1,5 @@ -pipeline_todos -============== +# pipeline_todos +```{eval-rst} .. automethod:: nf_core.lint.PipelineLint.pipeline_todos +``` diff --git a/docs/api/_src/pipeline_lint_tests/readme.rst b/docs/api/_src/pipeline_lint_tests/readme.md similarity index 64% rename from docs/api/_src/pipeline_lint_tests/readme.rst rename to docs/api/_src/pipeline_lint_tests/readme.md index dca8a32d11..9583a56e0a 100644 --- a/docs/api/_src/pipeline_lint_tests/readme.rst +++ b/docs/api/_src/pipeline_lint_tests/readme.md @@ -1,4 +1,5 @@ -readme -====== +# readme +```{eval-rst} .. automethod:: nf_core.lint.PipelineLint.readme +``` diff --git a/docs/api/_src/pipeline_lint_tests/schema_description.rst b/docs/api/_src/pipeline_lint_tests/schema_description.md similarity index 61% rename from docs/api/_src/pipeline_lint_tests/schema_description.rst rename to docs/api/_src/pipeline_lint_tests/schema_description.md index 8733e203e5..14f756acd2 100644 --- a/docs/api/_src/pipeline_lint_tests/schema_description.rst +++ b/docs/api/_src/pipeline_lint_tests/schema_description.md @@ -1,4 +1,5 @@ -schema_description -=========== +# schema_description +```{eval-rst} .. automethod:: nf_core.lint.PipelineLint.schema_description +``` diff --git a/docs/api/_src/pipeline_lint_tests/schema_lint.rst b/docs/api/_src/pipeline_lint_tests/schema_lint.md similarity index 63% rename from docs/api/_src/pipeline_lint_tests/schema_lint.rst rename to docs/api/_src/pipeline_lint_tests/schema_lint.md index 7d9697c8e9..39be2ea65c 100644 --- a/docs/api/_src/pipeline_lint_tests/schema_lint.rst +++ b/docs/api/_src/pipeline_lint_tests/schema_lint.md @@ -1,4 +1,5 @@ -schema_lint -=========== +# schema_lint +```{eval-rst} .. automethod:: nf_core.lint.PipelineLint.schema_lint +``` diff --git a/docs/api/_src/pipeline_lint_tests/schema_params.rst b/docs/api/_src/pipeline_lint_tests/schema_params.md similarity index 62% rename from docs/api/_src/pipeline_lint_tests/schema_params.rst rename to docs/api/_src/pipeline_lint_tests/schema_params.md index 0997774c50..80a626e880 100644 --- a/docs/api/_src/pipeline_lint_tests/schema_params.rst +++ b/docs/api/_src/pipeline_lint_tests/schema_params.md @@ -1,4 +1,5 @@ -schema_params -============= +# schema_params +```{eval-rst} .. automethod:: nf_core.lint.PipelineLint.schema_params +``` diff --git a/docs/api/_src/pipeline_lint_tests/template_strings.rst b/docs/api/_src/pipeline_lint_tests/template_strings.md similarity index 61% rename from docs/api/_src/pipeline_lint_tests/template_strings.rst rename to docs/api/_src/pipeline_lint_tests/template_strings.md index 9599a1c26b..3d03bfb256 100644 --- a/docs/api/_src/pipeline_lint_tests/template_strings.rst +++ b/docs/api/_src/pipeline_lint_tests/template_strings.md @@ -1,4 +1,5 @@ -template_strings -================ +# template_strings +```{eval-rst} .. automethod:: nf_core.lint.PipelineLint.template_strings +``` diff --git a/docs/api/_src/pipeline_lint_tests/version_consistency.rst b/docs/api/_src/pipeline_lint_tests/version_consistency.md similarity index 61% rename from docs/api/_src/pipeline_lint_tests/version_consistency.rst rename to docs/api/_src/pipeline_lint_tests/version_consistency.md index f0b334fc1c..e8038f8531 100644 --- a/docs/api/_src/pipeline_lint_tests/version_consistency.rst +++ b/docs/api/_src/pipeline_lint_tests/version_consistency.md @@ -1,4 +1,5 @@ -version_consistency -=================== +# version_consistency +```{eval-rst} .. automethod:: nf_core.lint.PipelineLint.version_consistency +``` From ebbfcd6b702d68eafc22ce4e2c92b4895c8eb5d0 Mon Sep 17 00:00:00 2001 From: Erik Danielsson Date: Mon, 2 Aug 2021 15:36:14 +0200 Subject: [PATCH 002/110] Linting of .md and fix failing pytest --- docs/api/_src/index.md | 2 +- .../_src/module_lint_tests/modules_command.md | 7 ++++--- tests/test_lint.py | 18 +++++++++--------- 3 files changed, 14 insertions(+), 13 deletions(-) diff --git a/docs/api/_src/index.md b/docs/api/_src/index.md index d7ff705e22..a6fb767122 100644 --- a/docs/api/_src/index.md +++ b/docs/api/_src/index.md @@ -17,7 +17,7 @@ Primarily, it describes the different [code lint tests](lint_tests/index.html) run by `nf-core lint` (typically visited by a developer when their pipeline fails a given test), and also reference for the `nf_core` [Python package API](api/index.html). -# Indices and tables +## Indices and tables - {ref}`genindex` - {ref}`modindex` diff --git a/docs/api/_src/module_lint_tests/modules_command.md b/docs/api/_src/module_lint_tests/modules_command.md index 642d3591a7..211a473911 100644 --- a/docs/api/_src/module_lint_tests/modules_command.md +++ b/docs/api/_src/module_lint_tests/modules_command.md @@ -1,7 +1,8 @@ -ModulesCommand -=============== +# ModulesCommand + This is the base class for all `nf-core modules` commands + ```{eval-rst} .. automodule:: nf_core.modules.module_utils - :members: + :members: ``` diff --git a/tests/test_lint.py b/tests/test_lint.py index c6d5dc351d..c33d5edc85 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -149,29 +149,29 @@ def test_strip_ansi_codes(self): stripped = self.lint_obj._strip_ansi_codes("ls \x1b[00m\x1b[01;31mexamplefile.zip\x1b[00m\x1b[01;31m") assert stripped == "ls examplefile.zip" - def test_sphinx_rst_files(self): - """Check that we have .rst files for all lint module code, + def test_sphinx_md_files(self): + """Check that we have .md files for all lint module code, and that there are no unexpected files (eg. deleted lint tests)""" docs_basedir = os.path.join( os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "docs", "api", "_src", "pipeline_lint_tests" ) - # Get list of existing .rst files + # Get list of existing .md files existing_docs = [] for fn in os.listdir(docs_basedir): - if fnmatch.fnmatch(fn, "*.rst") and not fnmatch.fnmatch(fn, "index.rst"): + if fnmatch.fnmatch(fn, "*.md") and not fnmatch.fnmatch(fn, "index.md"): existing_docs.append(os.path.join(docs_basedir, fn)) - # Check .rst files against each test name + # Check .md files against each test name lint_obj = nf_core.lint.PipelineLint("", True) for test_name in lint_obj.lint_tests: - fn = os.path.join(docs_basedir, "{}.rst".format(test_name)) - assert os.path.exists(fn), "Could not find lint docs .rst file: {}".format(fn) + fn = os.path.join(docs_basedir, "{}.md".format(test_name)) + assert os.path.exists(fn), "Could not find lint docs .md file: {}".format(fn) existing_docs.remove(fn) - # Check that we have no remaining .rst files that we didn't expect - assert len(existing_docs) == 0, "Unexpected lint docs .rst files found: {}".format(", ".join(existing_docs)) + # Check that we have no remaining .md files that we didn't expect + assert len(existing_docs) == 0, "Unexpected lint docs .md files found: {}".format(", ".join(existing_docs)) ####################### # SPECIFIC LINT TESTS # From f530318283dac141b284d62f9a5b98fe642b1e72 Mon Sep 17 00:00:00 2001 From: Erik Danielsson Date: Mon, 2 Aug 2021 16:46:07 +0200 Subject: [PATCH 003/110] Remove test files, update automatic docs creation --- docs/api/_src/module_lint_tests/index.md | 9 ----- .../_src/module_lint_tests/modules_command.md | 8 ----- docs/api/make_lint_rst.py | 16 +++++---- ...st.py.22f9371628f16666a1214bbd4decbfd7.tmp | 34 +++++++++++++++++++ 4 files changed, 43 insertions(+), 24 deletions(-) delete mode 100644 docs/api/_src/module_lint_tests/index.md delete mode 100644 docs/api/_src/module_lint_tests/modules_command.md create mode 100644 docs/api/make_lint_rst.py.22f9371628f16666a1214bbd4decbfd7.tmp diff --git a/docs/api/_src/module_lint_tests/index.md b/docs/api/_src/module_lint_tests/index.md deleted file mode 100644 index f889abf73c..0000000000 --- a/docs/api/_src/module_lint_tests/index.md +++ /dev/null @@ -1,9 +0,0 @@ -# Module lint tests - -```{toctree} -:caption: 'Tests:' -:glob: true -:maxdepth: 2 - -* -``` diff --git a/docs/api/_src/module_lint_tests/modules_command.md b/docs/api/_src/module_lint_tests/modules_command.md deleted file mode 100644 index 211a473911..0000000000 --- a/docs/api/_src/module_lint_tests/modules_command.md +++ /dev/null @@ -1,8 +0,0 @@ -# ModulesCommand - -This is the base class for all `nf-core modules` commands - -```{eval-rst} -.. automodule:: nf_core.modules.module_utils - :members: -``` diff --git a/docs/api/make_lint_rst.py b/docs/api/make_lint_rst.py index 67d2bc5ef7..2e80e384ac 100644 --- a/docs/api/make_lint_rst.py +++ b/docs/api/make_lint_rst.py @@ -6,27 +6,29 @@ docs_basedir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "_src", "pipeline_lint_tests") -# Get list of existing .rst files +# Get list of existing .md files existing_docs = [] for fn in os.listdir(docs_basedir): - if fnmatch.fnmatch(fn, "*.rst") and not fnmatch.fnmatch(fn, "index.rst"): + if fnmatch.fnmatch(fn, "*.md") and not fnmatch.fnmatch(fn, "index.md"): existing_docs.append(os.path.join(docs_basedir, fn)) -# Make .rst file for each test name +# Make .md file for each test name lint_obj = nf_core.lint.PipelineLint("", True) -rst_template = """{0} -{1} +md_template = """# {0} +```{eval-rst} .. automethod:: nf_core.lint.PipelineLint.{0} +``` + """ for test_name in lint_obj.lint_tests: - fn = os.path.join(docs_basedir, "{}.rst".format(test_name)) + fn = os.path.join(docs_basedir, "{}.md".format(test_name)) if os.path.exists(fn): existing_docs.remove(fn) else: with open(fn, "w") as fh: - fh.write(rst_template.format(test_name, len(test_name) * "=")) + fh.write(md_template.format(test_name)) for fn in existing_docs: os.remove(fn) diff --git a/docs/api/make_lint_rst.py.22f9371628f16666a1214bbd4decbfd7.tmp b/docs/api/make_lint_rst.py.22f9371628f16666a1214bbd4decbfd7.tmp new file mode 100644 index 0000000000..2e80e384ac --- /dev/null +++ b/docs/api/make_lint_rst.py.22f9371628f16666a1214bbd4decbfd7.tmp @@ -0,0 +1,34 @@ +#!/usr/bin/env python + +import fnmatch +import os +import nf_core.lint + +docs_basedir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "_src", "pipeline_lint_tests") + +# Get list of existing .md files +existing_docs = [] +for fn in os.listdir(docs_basedir): + if fnmatch.fnmatch(fn, "*.md") and not fnmatch.fnmatch(fn, "index.md"): + existing_docs.append(os.path.join(docs_basedir, fn)) + +# Make .md file for each test name +lint_obj = nf_core.lint.PipelineLint("", True) +md_template = """# {0} + +```{eval-rst} +.. automethod:: nf_core.lint.PipelineLint.{0} +``` + +""" + +for test_name in lint_obj.lint_tests: + fn = os.path.join(docs_basedir, "{}.md".format(test_name)) + if os.path.exists(fn): + existing_docs.remove(fn) + else: + with open(fn, "w") as fh: + fh.write(md_template.format(test_name)) + +for fn in existing_docs: + os.remove(fn) From f09e9ae487d42c1fa256158763b1158644dfde84 Mon Sep 17 00:00:00 2001 From: Erik Danielsson Date: Mon, 2 Aug 2021 16:47:19 +0200 Subject: [PATCH 004/110] Update CHANGELOG.md --- CHANGELOG.md | 1 + ...st.py.22f9371628f16666a1214bbd4decbfd7.tmp | 34 ------------------- 2 files changed, 1 insertion(+), 34 deletions(-) delete mode 100644 docs/api/make_lint_rst.py.22f9371628f16666a1214bbd4decbfd7.tmp diff --git a/CHANGELOG.md b/CHANGELOG.md index e9f518293f..7f37986284 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ ### General * Changed `questionary` `ask()` to `unsafe_ask()` to not catch `KeyboardInterupts` ([#1237](https://github.com/nf-core/tools/issues/1237)) +* Convert documentation to MyST. ### Modules diff --git a/docs/api/make_lint_rst.py.22f9371628f16666a1214bbd4decbfd7.tmp b/docs/api/make_lint_rst.py.22f9371628f16666a1214bbd4decbfd7.tmp deleted file mode 100644 index 2e80e384ac..0000000000 --- a/docs/api/make_lint_rst.py.22f9371628f16666a1214bbd4decbfd7.tmp +++ /dev/null @@ -1,34 +0,0 @@ -#!/usr/bin/env python - -import fnmatch -import os -import nf_core.lint - -docs_basedir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "_src", "pipeline_lint_tests") - -# Get list of existing .md files -existing_docs = [] -for fn in os.listdir(docs_basedir): - if fnmatch.fnmatch(fn, "*.md") and not fnmatch.fnmatch(fn, "index.md"): - existing_docs.append(os.path.join(docs_basedir, fn)) - -# Make .md file for each test name -lint_obj = nf_core.lint.PipelineLint("", True) -md_template = """# {0} - -```{eval-rst} -.. automethod:: nf_core.lint.PipelineLint.{0} -``` - -""" - -for test_name in lint_obj.lint_tests: - fn = os.path.join(docs_basedir, "{}.md".format(test_name)) - if os.path.exists(fn): - existing_docs.remove(fn) - else: - with open(fn, "w") as fh: - fh.write(md_template.format(test_name)) - -for fn in existing_docs: - os.remove(fn) From cb0d2903f51aa20f6c7aa4db90f3de4ebd07b003 Mon Sep 17 00:00:00 2001 From: Erik Danielsson Date: Wed, 4 Aug 2021 15:48:05 +0200 Subject: [PATCH 005/110] Update lint docs creation script to include modules --- docs/api/make_lint_md.py | 54 +++++++++++++++++++++++++++++++++++++++ docs/api/make_lint_rst.py | 34 ------------------------ 2 files changed, 54 insertions(+), 34 deletions(-) create mode 100644 docs/api/make_lint_md.py delete mode 100644 docs/api/make_lint_rst.py diff --git a/docs/api/make_lint_md.py b/docs/api/make_lint_md.py new file mode 100644 index 0000000000..6b54dc8682 --- /dev/null +++ b/docs/api/make_lint_md.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python + +import fnmatch +import os +import nf_core.lint +import nf_core.modules.lint + + +def make_docs(docs_basedir, lint_tests, md_template): + # Get list of existing .md files + existing_docs = [] + for fn in os.listdir(docs_basedir): + if fnmatch.fnmatch(fn, "*.md") and not fnmatch.fnmatch(fn, "index.md"): + existing_docs.append(os.path.join(docs_basedir, fn)) + + for test_name in lint_tests: + fn = os.path.join(docs_basedir, "{}.md".format(test_name)) + if os.path.exists(fn): + existing_docs.remove(fn) + else: + with open(fn, "w") as fh: + fh.write(md_template.format(test_name)) + + for fn in existing_docs: + os.remove(fn) + + +# Create the pipeline docs +pipeline_docs_basedir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "_src", "pipeline_lint_tests") +make_docs( + pipeline_docs_basedir, + nf_core.lint.PipelineLint._get_all_lint_tests(), + """# {0} + + ```{eval-rst} + .. automethod:: nf_core.lint.PipelineLint.{0} + ``` + + """, +) + +# Create the modules lint docs +modules_docs_basedir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "_src", "modules_lint_tests") +make_docs( + modules_docs_basedir, + nf_core.modules.lint.ModuleLint._get_all_lint_tests(), + """# {0} + + ```{eval-rst} + .. automethod:: nf_core.modules.lint.ModuleLint.{0} + ``` + + """, +) diff --git a/docs/api/make_lint_rst.py b/docs/api/make_lint_rst.py deleted file mode 100644 index 2e80e384ac..0000000000 --- a/docs/api/make_lint_rst.py +++ /dev/null @@ -1,34 +0,0 @@ -#!/usr/bin/env python - -import fnmatch -import os -import nf_core.lint - -docs_basedir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "_src", "pipeline_lint_tests") - -# Get list of existing .md files -existing_docs = [] -for fn in os.listdir(docs_basedir): - if fnmatch.fnmatch(fn, "*.md") and not fnmatch.fnmatch(fn, "index.md"): - existing_docs.append(os.path.join(docs_basedir, fn)) - -# Make .md file for each test name -lint_obj = nf_core.lint.PipelineLint("", True) -md_template = """# {0} - -```{eval-rst} -.. automethod:: nf_core.lint.PipelineLint.{0} -``` - -""" - -for test_name in lint_obj.lint_tests: - fn = os.path.join(docs_basedir, "{}.md".format(test_name)) - if os.path.exists(fn): - existing_docs.remove(fn) - else: - with open(fn, "w") as fh: - fh.write(md_template.format(test_name)) - -for fn in existing_docs: - os.remove(fn) From 8805422edb65d1b31d6ae40fb38de2abdff6bc37 Mon Sep 17 00:00:00 2001 From: Erik Danielsson Date: Wed, 4 Aug 2021 17:34:26 +0200 Subject: [PATCH 006/110] Flesh out module lint tests descriptions, and create .md files --- .../_src/module_lint_tests/functions_nf.md | 5 ++++ docs/api/_src/module_lint_tests/index.md | 9 +++++++ docs/api/_src/module_lint_tests/main_nf.md | 6 +++++ docs/api/_src/module_lint_tests/meta_yml.md | 7 ++++++ .../_src/module_lint_tests/module_changes.md | 6 +++++ .../_src/module_lint_tests/module_todos.md | 5 ++++ docs/api/_src/pipeline_lint_tests/index.md | 2 +- docs/api/make_lint_md.py | 9 +++---- nf_core/lint/__init__.py | 8 +++---- nf_core/modules/lint/functions_nf.py | 9 ++++++- nf_core/modules/lint/main_nf.py | 16 ++++++++++--- nf_core/modules/lint/meta_yml.py | 14 ++++++++++- nf_core/modules/lint/module_changes.py | 7 ++++-- nf_core/modules/lint/module_tests.py | 9 +++++-- nf_core/modules/lint/module_todos.py | 24 +++++++++++++++++-- nf_core/modules/lint/module_version.py | 8 +++++-- 16 files changed, 122 insertions(+), 22 deletions(-) create mode 100644 docs/api/_src/module_lint_tests/functions_nf.md create mode 100644 docs/api/_src/module_lint_tests/index.md create mode 100644 docs/api/_src/module_lint_tests/main_nf.md create mode 100644 docs/api/_src/module_lint_tests/meta_yml.md create mode 100644 docs/api/_src/module_lint_tests/module_changes.md create mode 100644 docs/api/_src/module_lint_tests/module_todos.md diff --git a/docs/api/_src/module_lint_tests/functions_nf.md b/docs/api/_src/module_lint_tests/functions_nf.md new file mode 100644 index 0000000000..c89d1d862c --- /dev/null +++ b/docs/api/_src/module_lint_tests/functions_nf.md @@ -0,0 +1,5 @@ +# functions_nf + +```{eval-rst} +.. automethod:: nf_core.modules.lint.ModuleLint.functions_nf +``` diff --git a/docs/api/_src/module_lint_tests/index.md b/docs/api/_src/module_lint_tests/index.md new file mode 100644 index 0000000000..f889abf73c --- /dev/null +++ b/docs/api/_src/module_lint_tests/index.md @@ -0,0 +1,9 @@ +# Module lint tests + +```{toctree} +:caption: 'Tests:' +:glob: true +:maxdepth: 2 + +* +``` diff --git a/docs/api/_src/module_lint_tests/main_nf.md b/docs/api/_src/module_lint_tests/main_nf.md new file mode 100644 index 0000000000..452520d705 --- /dev/null +++ b/docs/api/_src/module_lint_tests/main_nf.md @@ -0,0 +1,6 @@ +# main_nf + +```{eval-rst} +.. automethod:: nf_core.modules.lint.ModuleLint.main_nf +``` + diff --git a/docs/api/_src/module_lint_tests/meta_yml.md b/docs/api/_src/module_lint_tests/meta_yml.md new file mode 100644 index 0000000000..56e1bc8b7b --- /dev/null +++ b/docs/api/_src/module_lint_tests/meta_yml.md @@ -0,0 +1,7 @@ +# meta_yml + +```{eval-rst} +.. automethod:: nf_core.modules.lint.ModuleLint.meta_yml +``` + + diff --git a/docs/api/_src/module_lint_tests/module_changes.md b/docs/api/_src/module_lint_tests/module_changes.md new file mode 100644 index 0000000000..afcd71b07b --- /dev/null +++ b/docs/api/_src/module_lint_tests/module_changes.md @@ -0,0 +1,6 @@ +# module_changes + +```{eval-rst} +.. automethod:: nf_core.modules.lint.ModuleLint.module_changes +``` + diff --git a/docs/api/_src/module_lint_tests/module_todos.md b/docs/api/_src/module_lint_tests/module_todos.md new file mode 100644 index 0000000000..d592addd06 --- /dev/null +++ b/docs/api/_src/module_lint_tests/module_todos.md @@ -0,0 +1,5 @@ +# module_todos + +```{eval-rst} +.. automethod:: nf_core.modules.lint.ModuleLint.module_todos +``` diff --git a/docs/api/_src/pipeline_lint_tests/index.md b/docs/api/_src/pipeline_lint_tests/index.md index d10f8421c5..0cf9bc1d21 100644 --- a/docs/api/_src/pipeline_lint_tests/index.md +++ b/docs/api/_src/pipeline_lint_tests/index.md @@ -1,4 +1,4 @@ -# Pipline lint tests +# Pipeline lint tests ```{toctree} :caption: 'Tests:' diff --git a/docs/api/make_lint_md.py b/docs/api/make_lint_md.py index 6b54dc8682..25e5539f67 100644 --- a/docs/api/make_lint_md.py +++ b/docs/api/make_lint_md.py @@ -20,6 +20,7 @@ def make_docs(docs_basedir, lint_tests, md_template): else: with open(fn, "w") as fh: fh.write(md_template.format(test_name)) + print(test_name) for fn in existing_docs: os.remove(fn) @@ -29,10 +30,10 @@ def make_docs(docs_basedir, lint_tests, md_template): pipeline_docs_basedir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "_src", "pipeline_lint_tests") make_docs( pipeline_docs_basedir, - nf_core.lint.PipelineLint._get_all_lint_tests(), + nf_core.lint.PipelineLint._get_all_lint_tests(True), """# {0} - ```{eval-rst} + ```{{eval-rst}} .. automethod:: nf_core.lint.PipelineLint.{0} ``` @@ -40,13 +41,13 @@ def make_docs(docs_basedir, lint_tests, md_template): ) # Create the modules lint docs -modules_docs_basedir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "_src", "modules_lint_tests") +modules_docs_basedir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "_src", "module_lint_tests") make_docs( modules_docs_basedir, nf_core.modules.lint.ModuleLint._get_all_lint_tests(), """# {0} - ```{eval-rst} + ```{{eval-rst}} .. automethod:: nf_core.modules.lint.ModuleLint.{0} ``` diff --git a/nf_core/lint/__init__.py b/nf_core/lint/__init__.py index b61765b162..1f6d8e9253 100644 --- a/nf_core/lint/__init__.py +++ b/nf_core/lint/__init__.py @@ -20,8 +20,8 @@ import nf_core.utils import nf_core.lint_utils +import nf_core.modules.lint from nf_core.lint_utils import console -from nf_core.modules.lint import ModuleLint log = logging.getLogger(__name__) @@ -44,7 +44,7 @@ def run_linting( # Verify that the requested tests exist if key: - all_tests = set(PipelineLint._get_all_lint_tests(release_mode)).union(set(ModuleLint._get_all_lint_tests())) + all_tests = set(PipelineLint._get_all_lint_tests(release_mode)).union(set(nf_core.modules.lint.ModuleLint._get_all_lint_tests())) bad_keys = [k for k in key if k not in all_tests] if len(bad_keys) > 0: raise AssertionError( @@ -66,7 +66,7 @@ def run_linting( lint_obj._list_files() # Create the modules lint object - module_lint_obj = ModuleLint(pipeline_dir) + module_lint_obj = nf_core.modules.lint.ModuleLint(pipeline_dir) # Verify that the pipeline is correctly configured try: @@ -77,7 +77,7 @@ def run_linting( # Run only the tests we want if key: # Select only the module lint tests - module_lint_tests = list(set(key).intersection(set(ModuleLint._get_all_lint_tests()))) + module_lint_tests = list(set(key).intersection(set(nf_core.modules.lint.ModuleLint._get_all_lint_tests()))) else: # If no key is supplied, run the default modules tests module_lint_tests = ("module_changes", "module_version") diff --git a/nf_core/modules/lint/functions_nf.py b/nf_core/modules/lint/functions_nf.py index 600a1ae7fd..97be84560c 100644 --- a/nf_core/modules/lint/functions_nf.py +++ b/nf_core/modules/lint/functions_nf.py @@ -9,7 +9,14 @@ def functions_nf(module_lint_object, module): """ Lint a functions.nf file - Verifies that the file exists and contains all necessary functions + + Verifies that the file exists and contains the following functions:: + + getSoftwareName + initOptions + getPathFromList + saveFiles + """ local_copy = None template_copy = None diff --git a/nf_core/modules/lint/main_nf.py b/nf_core/modules/lint/main_nf.py index 018dc99af2..9fc6f63f7f 100644 --- a/nf_core/modules/lint/main_nf.py +++ b/nf_core/modules/lint/main_nf.py @@ -10,10 +10,20 @@ def main_nf(module_lint_object, module): """ Lint a single main.nf module file + Can also be used to lint local module files, - in which case failures should be interpreted - as warnings + in which case failures will be reported as + warnings + + The test checks that software versions and + containers are correct. It warns if that the + module does not have a process label or if it + is not among the standard ones. + + In the modules script section it checks that + ``def software`` and ``def prefix`` are defined. """ + inputs = [] outputs = [] @@ -116,7 +126,7 @@ def main_nf(module_lint_object, module): def check_script_section(self, lines): """ Lint the script section - Checks whether 'def sotware' and 'def prefix' are defined + Checks whether 'def software' and 'def prefix' are defined """ script = "".join(lines) diff --git a/nf_core/modules/lint/meta_yml.py b/nf_core/modules/lint/meta_yml.py index be265a16ec..60e4832823 100644 --- a/nf_core/modules/lint/meta_yml.py +++ b/nf_core/modules/lint/meta_yml.py @@ -7,7 +7,19 @@ def meta_yml(module_lint_object, module): - """Lint a meta yml file""" + """ + Lint a meta.yml file + + The lint test checks that the module has + a ``meta.yml`` file and that it contains + the required keys: 'name', input' and + 'output'. + + In addition it checks that the module name + and module input is consistent between the + ``meta.yml`` and the ``main.nf``. + + """ required_keys = ["name", "input", "output"] required_keys_lists = ["input", "output"] try: diff --git a/nf_core/modules/lint/module_changes.py b/nf_core/modules/lint/module_changes.py index 57f3b9a03f..34354f1795 100644 --- a/nf_core/modules/lint/module_changes.py +++ b/nf_core/modules/lint/module_changes.py @@ -11,11 +11,14 @@ def module_changes(module_lint_object, module): """ Checks whether installed nf-core modules have changed compared to the original repository - Downloads the 'main.nf', 'functions.nf' and 'meta.yml' files for every module + + Downloads the ``main.nf``, ``functions.nf`` and ``meta.yml`` files for every module and compares them to the local copies - If the module has a 'git_sha', the file content is checked against this sha + If the module has a commit SHA entry in the ``modules.json``, the file content is + checked against this SHA """ + files_to_check = ["main.nf", "functions.nf", "meta.yml"] # Loop over nf-core modules diff --git a/nf_core/modules/lint/module_tests.py b/nf_core/modules/lint/module_tests.py index fad25d20ac..78cc18e063 100644 --- a/nf_core/modules/lint/module_tests.py +++ b/nf_core/modules/lint/module_tests.py @@ -3,7 +3,6 @@ """ import os import logging -import sys import yaml log = logging.getLogger(__name__) @@ -11,7 +10,13 @@ def module_tests(module_lint_object, module): """ - Lint module tests + Lint the tests of a module in nf-core/modules + + It verifies that the test directory exists + and contains a ``main.nf`` and a ``test.yml``, + and that the module is present in the ``pytest_modules.yml`` + file. + """ if os.path.exists(module.test_dir): diff --git a/nf_core/modules/lint/module_todos.py b/nf_core/modules/lint/module_todos.py index 57c73ebbcd..bbd1c4d6f7 100644 --- a/nf_core/modules/lint/module_todos.py +++ b/nf_core/modules/lint/module_todos.py @@ -8,8 +8,28 @@ def module_todos(module_lint_object, module): """ Look for TODO statements in the module files - Slight modification of the "nf_core.lint.pipeline_todos" function to make it work - for a single module + + The nf-core module template contains a number of comment lines to help developers + of new modules know where they need to edit files and add content. + They typically have the following format: + + .. code-block:: groovy + + // TODO nf-core: Make some kind of change to the workflow here + + ..or in markdown: + + .. code-block:: html + + + + This lint test runs through all files in the module and searches for these lines. + If any are found they will throw a warning. + + .. tip:: Note that many GUI code editors have plugins to list all instances of *TODO* + in a given project directory. This is a very quick and convenient way to get + started on your pipeline! + """ module.wf_path = module.module_dir results = pipeline_todos(module) diff --git a/nf_core/modules/lint/module_version.py b/nf_core/modules/lint/module_version.py index cc3e01c8be..03ea4f248c 100644 --- a/nf_core/modules/lint/module_version.py +++ b/nf_core/modules/lint/module_version.py @@ -18,9 +18,13 @@ def module_version(module_lint_object, module): """ - Verify that the module has a version (git_sha) specified in the - modules.json file and checks whether a new version is available + Verifies that the module has a version specified in the modules.json + + It checks whether the module has an entry in the ``modules.json`` file + containing a commit SHA. If that is true, it verifies that there are no + newer version of the module available. """ + modules_json_path = os.path.join(module_lint_object.dir, "modules.json") # Verify that a git_sha exists in the `modules.json` file for this module From 3bb958423a63cbd5f29a86076b271037606a11f4 Mon Sep 17 00:00:00 2001 From: Erik Danielsson Date: Thu, 5 Aug 2021 10:28:23 +0200 Subject: [PATCH 007/110] Small fixes --- nf_core/lint/modules_json.py | 2 +- nf_core/modules/lint/main_nf.py | 23 +++++++++++++---------- nf_core/modules/lint/meta_yml.py | 4 ++-- nf_core/modules/lint/module_changes.py | 8 ++++---- 4 files changed, 20 insertions(+), 17 deletions(-) diff --git a/nf_core/lint/modules_json.py b/nf_core/lint/modules_json.py index 07abf94987..4c5923d508 100644 --- a/nf_core/lint/modules_json.py +++ b/nf_core/lint/modules_json.py @@ -8,7 +8,7 @@ def modules_json(self): """Make sure all modules described in the ``modules.json`` file are actually installed Every module installed from ``nf-core/modules`` must have an entry in the ``modules.json`` file - with an associated version git_sha hash. + with an associated version commit hash. * Failure: If module entries are found in ``modules.json`` for modules that are not installed """ diff --git a/nf_core/modules/lint/main_nf.py b/nf_core/modules/lint/main_nf.py index 9fc6f63f7f..9b6d266728 100644 --- a/nf_core/modules/lint/main_nf.py +++ b/nf_core/modules/lint/main_nf.py @@ -13,15 +13,18 @@ def main_nf(module_lint_object, module): Can also be used to lint local module files, in which case failures will be reported as - warnings - - The test checks that software versions and - containers are correct. It warns if that the - module does not have a process label or if it - is not among the standard ones. - - In the modules script section it checks that - ``def software`` and ``def prefix`` are defined. + warnings. + + The test checks for the following: + + * Software versions and containers are valid + * The module has a process label and it is among + the standard ones. + * If a ``meta`` map is defined as one of the modules + inputs it should be defined as one of the outputs, + and be correctly configured in the ``saveAs`` function. + * The module script section should contain definitions + of ``software`` and ``prefix`` """ inputs = [] @@ -162,7 +165,7 @@ def check_process_section(self, lines): if all([x.upper() for x in self.process_name]): self.passed.append(("process_capitals", "Process name is in capital letters", self.main_nf)) else: - self.failed.append(("process_capitals", "Process name is not in captial letters", self.main_nf)) + self.failed.append(("process_capitals", "Process name is not in capital letters", self.main_nf)) # Check that process labels are correct correct_process_labels = ["process_low", "process_medium", "process_high", "process_long"] diff --git a/nf_core/modules/lint/meta_yml.py b/nf_core/modules/lint/meta_yml.py index 60e4832823..561f230c91 100644 --- a/nf_core/modules/lint/meta_yml.py +++ b/nf_core/modules/lint/meta_yml.py @@ -12,8 +12,8 @@ def meta_yml(module_lint_object, module): The lint test checks that the module has a ``meta.yml`` file and that it contains - the required keys: 'name', input' and - 'output'. + the required keys: ``name``, input`` and + ``output``. In addition it checks that the module name and module input is consistent between the diff --git a/nf_core/modules/lint/module_changes.py b/nf_core/modules/lint/module_changes.py index 34354f1795..b75b2e0067 100644 --- a/nf_core/modules/lint/module_changes.py +++ b/nf_core/modules/lint/module_changes.py @@ -10,13 +10,13 @@ def module_changes(module_lint_object, module): """ Checks whether installed nf-core modules have changed compared to the - original repository + original repository. - Downloads the ``main.nf``, ``functions.nf`` and ``meta.yml`` files for every module - and compares them to the local copies + Downloads the ``main.nf``, ``functions.nf`` and ``meta.yml`` files from the remote + for every module and compares them to the local copies. If the module has a commit SHA entry in the ``modules.json``, the file content is - checked against this SHA + compared against the files in the remote at this SHA. """ files_to_check = ["main.nf", "functions.nf", "meta.yml"] From fee0040a9a25a07de6f431e43185d40e87010627 Mon Sep 17 00:00:00 2001 From: Erik Danielsson Date: Thu, 5 Aug 2021 10:33:37 +0200 Subject: [PATCH 008/110] Minor changes --- nf_core/modules/lint/functions_nf.py | 2 +- nf_core/modules/lint/main_nf.py | 2 +- nf_core/modules/lint/meta_yml.py | 2 +- nf_core/modules/lint/module_tests.py | 2 +- nf_core/modules/lint/module_version.py | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/nf_core/modules/lint/functions_nf.py b/nf_core/modules/lint/functions_nf.py index 97be84560c..f927e55ffc 100644 --- a/nf_core/modules/lint/functions_nf.py +++ b/nf_core/modules/lint/functions_nf.py @@ -8,7 +8,7 @@ def functions_nf(module_lint_object, module): """ - Lint a functions.nf file + Lint a ``functions.nf`` file Verifies that the file exists and contains the following functions:: diff --git a/nf_core/modules/lint/main_nf.py b/nf_core/modules/lint/main_nf.py index 9b6d266728..555fcedc5a 100644 --- a/nf_core/modules/lint/main_nf.py +++ b/nf_core/modules/lint/main_nf.py @@ -9,7 +9,7 @@ def main_nf(module_lint_object, module): """ - Lint a single main.nf module file + Lint a ``main.nf`` module file Can also be used to lint local module files, in which case failures will be reported as diff --git a/nf_core/modules/lint/meta_yml.py b/nf_core/modules/lint/meta_yml.py index 561f230c91..3c59c774d4 100644 --- a/nf_core/modules/lint/meta_yml.py +++ b/nf_core/modules/lint/meta_yml.py @@ -8,7 +8,7 @@ def meta_yml(module_lint_object, module): """ - Lint a meta.yml file + Lint a ``meta.yml`` file The lint test checks that the module has a ``meta.yml`` file and that it contains diff --git a/nf_core/modules/lint/module_tests.py b/nf_core/modules/lint/module_tests.py index 78cc18e063..f76d45d69f 100644 --- a/nf_core/modules/lint/module_tests.py +++ b/nf_core/modules/lint/module_tests.py @@ -10,7 +10,7 @@ def module_tests(module_lint_object, module): """ - Lint the tests of a module in nf-core/modules + Lint the tests of a module in ``nf-core/modules`` It verifies that the test directory exists and contains a ``main.nf`` and a ``test.yml``, diff --git a/nf_core/modules/lint/module_version.py b/nf_core/modules/lint/module_version.py index 03ea4f248c..29d7327490 100644 --- a/nf_core/modules/lint/module_version.py +++ b/nf_core/modules/lint/module_version.py @@ -18,7 +18,7 @@ def module_version(module_lint_object, module): """ - Verifies that the module has a version specified in the modules.json + Verifies that the module has a version specified in the ``modules.json`` file It checks whether the module has an entry in the ``modules.json`` file containing a commit SHA. If that is true, it verifies that there are no From 2aad12159eb34132572aac0baba3e54e0ec88f50 Mon Sep 17 00:00:00 2001 From: Erik Danielsson Date: Thu, 5 Aug 2021 10:50:06 +0200 Subject: [PATCH 009/110] run markdownlint --- docs/api/_src/module_lint_tests/main_nf.md | 1 - docs/api/_src/module_lint_tests/meta_yml.md | 2 -- docs/api/_src/module_lint_tests/module_changes.md | 1 - 3 files changed, 4 deletions(-) diff --git a/docs/api/_src/module_lint_tests/main_nf.md b/docs/api/_src/module_lint_tests/main_nf.md index 452520d705..5a96785733 100644 --- a/docs/api/_src/module_lint_tests/main_nf.md +++ b/docs/api/_src/module_lint_tests/main_nf.md @@ -3,4 +3,3 @@ ```{eval-rst} .. automethod:: nf_core.modules.lint.ModuleLint.main_nf ``` - diff --git a/docs/api/_src/module_lint_tests/meta_yml.md b/docs/api/_src/module_lint_tests/meta_yml.md index 56e1bc8b7b..33920a4b9b 100644 --- a/docs/api/_src/module_lint_tests/meta_yml.md +++ b/docs/api/_src/module_lint_tests/meta_yml.md @@ -3,5 +3,3 @@ ```{eval-rst} .. automethod:: nf_core.modules.lint.ModuleLint.meta_yml ``` - - diff --git a/docs/api/_src/module_lint_tests/module_changes.md b/docs/api/_src/module_lint_tests/module_changes.md index afcd71b07b..ce2d428ca9 100644 --- a/docs/api/_src/module_lint_tests/module_changes.md +++ b/docs/api/_src/module_lint_tests/module_changes.md @@ -3,4 +3,3 @@ ```{eval-rst} .. automethod:: nf_core.modules.lint.ModuleLint.module_changes ``` - From 1ea015e68fa1e4e92526373fe582998a48106817 Mon Sep 17 00:00:00 2001 From: Erik Danielsson Date: Thu, 5 Aug 2021 10:57:46 +0200 Subject: [PATCH 010/110] Update CHANGELOG.md --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7f37986284..cc296788a5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ * Changed `questionary` `ask()` to `unsafe_ask()` to not catch `KeyboardInterupts` ([#1237](https://github.com/nf-core/tools/issues/1237)) * Convert documentation to MyST. +* Add documentation for the `nf-core modules lint` tests. ### Modules From d3f1b85035d612d15e719f08eec1f8d7d5bae874 Mon Sep 17 00:00:00 2001 From: Erik Danielsson Date: Thu, 5 Aug 2021 11:05:21 +0200 Subject: [PATCH 011/110] Fix issues in md templates --- docs/api/make_lint_md.py | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/docs/api/make_lint_md.py b/docs/api/make_lint_md.py index 25e5539f67..a6ec98a944 100644 --- a/docs/api/make_lint_md.py +++ b/docs/api/make_lint_md.py @@ -33,11 +33,10 @@ def make_docs(docs_basedir, lint_tests, md_template): nf_core.lint.PipelineLint._get_all_lint_tests(True), """# {0} - ```{{eval-rst}} - .. automethod:: nf_core.lint.PipelineLint.{0} - ``` - - """, +```{{eval-rst}} +.. automethod:: nf_core.lint.PipelineLint.{0} +``` +""", ) # Create the modules lint docs @@ -47,9 +46,8 @@ def make_docs(docs_basedir, lint_tests, md_template): nf_core.modules.lint.ModuleLint._get_all_lint_tests(), """# {0} - ```{{eval-rst}} - .. automethod:: nf_core.modules.lint.ModuleLint.{0} - ``` - - """, +```{{eval-rst}} +.. automethod:: nf_core.modules.lint.ModuleLint.{0} +``` +""", ) From 1b151d22ecb4caf5bf58c6aa690ff04ee506261d Mon Sep 17 00:00:00 2001 From: Erik Danielsson Date: Thu, 5 Aug 2021 11:08:28 +0200 Subject: [PATCH 012/110] black --- nf_core/lint/__init__.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/nf_core/lint/__init__.py b/nf_core/lint/__init__.py index 1f6d8e9253..4838726928 100644 --- a/nf_core/lint/__init__.py +++ b/nf_core/lint/__init__.py @@ -44,7 +44,9 @@ def run_linting( # Verify that the requested tests exist if key: - all_tests = set(PipelineLint._get_all_lint_tests(release_mode)).union(set(nf_core.modules.lint.ModuleLint._get_all_lint_tests())) + all_tests = set(PipelineLint._get_all_lint_tests(release_mode)).union( + set(nf_core.modules.lint.ModuleLint._get_all_lint_tests()) + ) bad_keys = [k for k in key if k not in all_tests] if len(bad_keys) > 0: raise AssertionError( From 15718ae3446622fdecfd803911def47b31d9cca5 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 15 Mar 2022 22:26:47 +0100 Subject: [PATCH 013/110] Bump to v2.4dev --- .github/RELEASE_CHECKLIST.md | 4 +++- CHANGELOG.md | 10 +++++++++- setup.py | 2 +- 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/.github/RELEASE_CHECKLIST.md b/.github/RELEASE_CHECKLIST.md index 1f9a8b2eb4..e5b670c561 100644 --- a/.github/RELEASE_CHECKLIST.md +++ b/.github/RELEASE_CHECKLIST.md @@ -9,7 +9,9 @@ 7. Make sure all CI tests are passing again (additional tests are run on PRs to `master`) 8. Request review (2 approvals required) 9. Merge the PR into `master` -10. Once CI tests on commit have passed, create a new release copying the `CHANGELOG` for that release into the description section. +10. Wait for CI tests on the commit to passed +11. (Optional but a good idea) Run a manual sync on `nf-core/testpipeline` and check that CI is passing on the resulting PR. +12. Create a new release copying the `CHANGELOG` for that release into the description section. ## After release diff --git a/CHANGELOG.md b/CHANGELOG.md index 4f7a5ddd8f..87ea3eae15 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ # nf-core/tools: Changelog +## v2.4dev + +### Template + +### General + +### Modules + ## [v2.3 - Mercury Vulture](https://github.com/nf-core/tools/releases/tag/2.3) - [2022-03-15] ### Template @@ -17,7 +25,7 @@ * Add optional loading of of pipeline-specific institutional configs to `nextflow.config` * Make `--outdir` a mandatory parameter ([nf-core/tools#1415](https://github.com/nf-core/tools/issues/1415)) -## General +### General * Updated `nf-core download` to work with latest DSL2 syntax for containers ([#1379](https://github.com/nf-core/tools/issues/1379)) * Made `nf-core modules create` detect repository type with explicit `.nf-core.yml` instead of random readme stuff ([#1391](https://github.com/nf-core/tools/pull/1391)) diff --git a/setup.py b/setup.py index b4acd45832..5fda0ae5be 100644 --- a/setup.py +++ b/setup.py @@ -2,7 +2,7 @@ from setuptools import setup, find_packages -version = "2.3" +version = "2.4dev" with open("README.md") as f: readme = f.read() From 89a25473b23089dfff90c509e20b9a7840cf385a Mon Sep 17 00:00:00 2001 From: Harshil Patel Date: Tue, 15 Mar 2022 21:44:00 +0000 Subject: [PATCH 014/110] Minor fixes to template --- nf_core/pipeline-template/.github/PULL_REQUEST_TEMPLATE.md | 2 +- nf_core/pipeline-template/docs/usage.md | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/nf_core/pipeline-template/.github/PULL_REQUEST_TEMPLATE.md b/nf_core/pipeline-template/.github/PULL_REQUEST_TEMPLATE.md index a6eeaaccef..1ef9d3eaad 100644 --- a/nf_core/pipeline-template/.github/PULL_REQUEST_TEMPLATE.md +++ b/nf_core/pipeline-template/.github/PULL_REQUEST_TEMPLATE.md @@ -19,7 +19,7 @@ Learn more about contributing: [CONTRIBUTING.md](https://github.com/{{ name }}/t - [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/{{ name }}/tree/master/.github/CONTRIBUTING.md) - [ ] If necessary, also make a PR on the {{ name }} _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository. - [ ] Make sure your code lints (`nf-core lint`). -- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker` --outdir `). +- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker --outdir `). - [ ] Usage Documentation in `docs/usage.md` is updated. - [ ] Output Documentation in `docs/output.md` is updated. - [ ] `CHANGELOG.md` is updated. diff --git a/nf_core/pipeline-template/docs/usage.md b/nf_core/pipeline-template/docs/usage.md index 8d84911d67..00da1fca8c 100644 --- a/nf_core/pipeline-template/docs/usage.md +++ b/nf_core/pipeline-template/docs/usage.md @@ -57,7 +57,7 @@ An [example samplesheet](../assets/samplesheet.csv) has been provided with the p The typical command for running the pipeline is as follows: ```console -nextflow run {{ name }} --input samplesheet.csv --outdir --genome GRCh37 -profile docker +nextflow run {{ name }} --input samplesheet.csv --outdir --genome GRCh37 -profile docker ``` This will launch the pipeline with the `docker` configuration profile. See below for more information about profiles. @@ -186,6 +186,7 @@ process { ``` > **NB:** We specify the full process name i.e. `NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN` in the config file because this takes priority over the short name (`STAR_ALIGN`) and allows existing configuration using the full process name to be correctly overridden. +> > If you get a warning suggesting that the process selector isn't recognised check that the process name has been specified correctly. ### Updating containers From db5a7fa0ed4d083f3e0d3b49aacc4ea11e71ce39 Mon Sep 17 00:00:00 2001 From: jpfeuffer Date: Wed, 16 Mar 2022 12:56:28 +0000 Subject: [PATCH 015/110] allow using plain pytest --- .github/CONTRIBUTING.md | 2 +- pytest.ini | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index e9e98d457c..999cebd8a6 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -75,7 +75,7 @@ New features should also come with new tests, to keep the test-coverage high (we You can try running the tests locally before pushing code using the following command: ```bash -pytest --color=yes tests/ +pytest --color=yes ``` ### Lint Tests diff --git a/pytest.ini b/pytest.ini index 02a8acac0f..57209fee41 100644 --- a/pytest.ini +++ b/pytest.ini @@ -1,3 +1,5 @@ [pytest] filterwarnings = ignore::pytest.PytestRemovedIn8Warning:_pytest.nodes:140 +testpaths = + tests From 348b105fa99afcdc2ef419b466fdbf4260083ce2 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 16 Mar 2022 14:14:11 +0100 Subject: [PATCH 016/110] Add myst-parser to docs requirements --- docs/api/requirements.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/api/requirements.txt b/docs/api/requirements.txt index 3f9531ba3d..fc5184b95f 100644 --- a/docs/api/requirements.txt +++ b/docs/api/requirements.txt @@ -1,3 +1,4 @@ Sphinx>=3.3.1 sphinxcontrib-napoleon sphinx_rtd_theme>=0.5.0 +myst-parser From 9295189819f6fbf26f1d7ea93e9d4bbdb9c255d4 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 16 Mar 2022 14:22:50 +0100 Subject: [PATCH 017/110] CI: Always try building docs. Only deploy if pushed on dev of nf-core/tools --- .github/workflows/tools-api-docs-dev.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/tools-api-docs-dev.yml b/.github/workflows/tools-api-docs-dev.yml index c8ccf211d6..8a45922c7c 100644 --- a/.github/workflows/tools-api-docs-dev.yml +++ b/.github/workflows/tools-api-docs-dev.yml @@ -1,7 +1,6 @@ name: nf-core/tools dev API docs -on: - push: - branches: [dev] +# Run on push and PR to test that docs build +on: [pull_request, push] jobs: api-docs: @@ -27,7 +26,8 @@ jobs: run: make --directory ./docs/api html - name: Sync dev docs - if: github.repository == 'nf-core/tools' + # Only sync with the website if it was a push from nf-core/tools dev branch + if: github.repository == 'nf-core/tools' && github.event.type == 'push' && github.event.base_ref == 'refs/heads/dev' uses: SamKirkland/FTP-Deploy-Action@4.0.0 with: server: ${{ secrets.ftp_server }} From a520a187fcbbd96d7552f4349f5d37d6bc63c82d Mon Sep 17 00:00:00 2001 From: jpfeuffer Date: Wed, 16 Mar 2022 14:01:11 +0000 Subject: [PATCH 018/110] [FEATURE] flexible linting for actionsCI and multiqc_config --- .../pipeline_lint_tests/multiqc_config.rst | 4 + nf_core/lint/__init__.py | 2 + nf_core/lint/actions_ci.py | 8 +- nf_core/lint/files_unchanged.py | 3 - nf_core/lint/multiqc_config.py | 84 +++++++++++++++++++ 5 files changed, 96 insertions(+), 5 deletions(-) create mode 100644 docs/api/_src/pipeline_lint_tests/multiqc_config.rst create mode 100644 nf_core/lint/multiqc_config.py diff --git a/docs/api/_src/pipeline_lint_tests/multiqc_config.rst b/docs/api/_src/pipeline_lint_tests/multiqc_config.rst new file mode 100644 index 0000000000..1a68bc2c21 --- /dev/null +++ b/docs/api/_src/pipeline_lint_tests/multiqc_config.rst @@ -0,0 +1,4 @@ +multiqc_config +=============== + +.. automethod:: nf_core.lint.PipelineLint.multiqc_config diff --git a/nf_core/lint/__init__.py b/nf_core/lint/__init__.py index 14f3667e2c..a0dad185b8 100644 --- a/nf_core/lint/__init__.py +++ b/nf_core/lint/__init__.py @@ -153,6 +153,7 @@ class PipelineLint(nf_core.utils.Pipeline): from .files_unchanged import files_unchanged from .merge_markers import merge_markers from .modules_json import modules_json + from .multiqc_config import multiqc_config from .nextflow_config import nextflow_config from .pipeline_name_conventions import pipeline_name_conventions from .pipeline_todos import pipeline_todos @@ -205,6 +206,7 @@ def _get_all_lint_tests(release_mode): "actions_schema_validation", "merge_markers", "modules_json", + "multiqc_config", ] + (["version_consistency"] if release_mode else []) def _load(self): diff --git a/nf_core/lint/actions_ci.py b/nf_core/lint/actions_ci.py index e9adf706b7..52e92591f7 100644 --- a/nf_core/lint/actions_ci.py +++ b/nf_core/lint/actions_ci.py @@ -85,9 +85,13 @@ def actions_ci(self): # Check that the action is turned on for the correct events try: - expected = {"push": {"branches": ["dev"]}, "pull_request": None, "release": {"types": ["published"]}} # NB: YAML dict key 'on' is evaluated to a Python dict key True - assert ciwf[True] == expected + assert "dev" in ciwf[True]["push"]["branches"] + pr_subtree = ciwf[True]["pull_request"] + assert pr_subtree == None or \ + ("branches" in pr_subtree and "dev" in pr_subtree["branches"]) or \ + ("ignore_branches" in pr_subtree and not "dev" in pr_subtree["ignore_branches"]) + assert "published" in ciwf[True]["release"]["types"] except (AssertionError, KeyError, TypeError): failed.append("'.github/workflows/ci.yml' is not triggered on expected events") else: diff --git a/nf_core/lint/files_unchanged.py b/nf_core/lint/files_unchanged.py index 229f5bb94c..cd034fa24b 100644 --- a/nf_core/lint/files_unchanged.py +++ b/nf_core/lint/files_unchanged.py @@ -45,7 +45,6 @@ def files_unchanged(self): Files that can have additional content but must include the template contents:: .gitignore - assets/multiqc_config.yaml .. tip:: You can configure the ``nf-core lint`` tests to ignore any of these checks by setting the ``files_unchanged`` key as follows in your linting config file. For example: @@ -54,7 +53,6 @@ def files_unchanged(self): files_unchanged: - .github/workflows/branch.yml - - assets/multiqc_config.yaml """ @@ -102,7 +100,6 @@ def files_unchanged(self): ] files_partial = [ [".gitignore", "foo"], - [os.path.join("assets", "multiqc_config.yaml")], ] # Only show error messages from pipeline creation diff --git a/nf_core/lint/multiqc_config.py b/nf_core/lint/multiqc_config.py new file mode 100644 index 0000000000..84dd978bc0 --- /dev/null +++ b/nf_core/lint/multiqc_config.py @@ -0,0 +1,84 @@ +#!/usr/bin/env python + +import os +from logging import warn +from nf_core.modules.modules_command import ModuleCommand + + +def multiqc_config(self): + """Make sure basic multiQC plugins are installed and plots are exported + Basic template: + report_comment: > + This report has been generated by the nf-core/quantms + analysis pipeline. For information about how to interpret these results, please see the + documentation. + report_section_order: + software_versions: + order: -1000 + nf-core-quantms-summary: + order: -1001 + + export_plots: true + """ + passed = [] + failed = [] + + fn = os.path.join(self.wf_path, "assets", "multiqc_config.yaml") + + # Return a failed status if we can't find the file + if not os.path.isfile(fn): + return {"failed": ["'assets/multiqc_config.yaml' not found"]} + + try: + with open(fn, "r") as fh: + mqc_yml = yaml.safe_load(fh) + except Exception as e: + return {"failed": ["Could not parse yaml file: {}, {}".format(fn, e)]} + + # Check that the report_comment exists and matches + try: + assert "report_section_order" in mqc_yml + orders = defaultdict(int) + summary_plugin_name = "nf-core-" + self.pipeline_name + "-summary" + min_plugins = ["software_versions", summary_plugin_name] + for plugin in min_plugins: + assert plugin in mqc_yml["report_section_order"] + assert "order" in mqc_yml["report_section_order"][plugin] + plugin_order = mqc_yml["report_section_order"][plugin]["order"] + assert plugin_order < 0 + + for plugin in mqc_yml["report_section_order"]: + if "order" in mqc_yml["report_section_order"][plugin]: + orders[plugin] = mqc_yml["report_section_order"][plugin]["order"] + + assert orders[summary_plugin_name] == min(orders) + orders.pop(summary_plugin_name) + assert orders["software_versions"] == min(orders) + except (AssertionError, KeyError, TypeError): + failed.append("'assets/multiqc_config.yaml' does not follow the ordering scheme of the minimally required plugins.") + else: + passed.append("'assets/multiqc_config.yaml' contains a matching report_comment.") + + # Check that the minimum plugins exist and are coming first in the summary + try: + assert "report_comment" in mqc_yml + assert mqc_yml["report_comment"].trim() == \ + '''This report has been generated by the nf-core/%s + analysis pipeline. For information about how to interpret these results, please see the + documentation. + ''' % (self.pipeline_name) + except (AssertionError, KeyError, TypeError): + failed.append("'assets/multiqc_config.yaml' does not contain a matching report_comment.") + else: + passed.append("'assets/multiqc_config.yaml' contains a matching report_comment.") + + # Check that export_plots is activated + try: + assert "export_plots" in mqc_yml + assert mqc_yml["export_plots"] == True + except (AssertionError, KeyError, TypeError): + failed.append("'assets/multiqc_config.yaml' does not contain 'export_plots: true'.") + else: + passed.append("'assets/multiqc_config.yaml' contains 'export_plots: true'.") + + return {"passed": passed, "failed": failed} From 26643ab0c9c36d46641f68ca1677a533fba66ccf Mon Sep 17 00:00:00 2001 From: jpfeuffer Date: Wed, 16 Mar 2022 14:13:43 +0000 Subject: [PATCH 019/110] black reformat --- nf_core/lint/actions_ci.py | 8 +++++--- nf_core/lint/multiqc_config.py | 17 +++++++++++------ 2 files changed, 16 insertions(+), 9 deletions(-) diff --git a/nf_core/lint/actions_ci.py b/nf_core/lint/actions_ci.py index 52e92591f7..89d9becd04 100644 --- a/nf_core/lint/actions_ci.py +++ b/nf_core/lint/actions_ci.py @@ -88,9 +88,11 @@ def actions_ci(self): # NB: YAML dict key 'on' is evaluated to a Python dict key True assert "dev" in ciwf[True]["push"]["branches"] pr_subtree = ciwf[True]["pull_request"] - assert pr_subtree == None or \ - ("branches" in pr_subtree and "dev" in pr_subtree["branches"]) or \ - ("ignore_branches" in pr_subtree and not "dev" in pr_subtree["ignore_branches"]) + assert ( + pr_subtree == None + or ("branches" in pr_subtree and "dev" in pr_subtree["branches"]) + or ("ignore_branches" in pr_subtree and not "dev" in pr_subtree["ignore_branches"]) + ) assert "published" in ciwf[True]["release"]["types"] except (AssertionError, KeyError, TypeError): failed.append("'.github/workflows/ci.yml' is not triggered on expected events") diff --git a/nf_core/lint/multiqc_config.py b/nf_core/lint/multiqc_config.py index 84dd978bc0..8204f95224 100644 --- a/nf_core/lint/multiqc_config.py +++ b/nf_core/lint/multiqc_config.py @@ -46,27 +46,32 @@ def multiqc_config(self): assert "order" in mqc_yml["report_section_order"][plugin] plugin_order = mqc_yml["report_section_order"][plugin]["order"] assert plugin_order < 0 - + for plugin in mqc_yml["report_section_order"]: if "order" in mqc_yml["report_section_order"][plugin]: orders[plugin] = mqc_yml["report_section_order"][plugin]["order"] - + assert orders[summary_plugin_name] == min(orders) orders.pop(summary_plugin_name) assert orders["software_versions"] == min(orders) except (AssertionError, KeyError, TypeError): - failed.append("'assets/multiqc_config.yaml' does not follow the ordering scheme of the minimally required plugins.") + failed.append( + "'assets/multiqc_config.yaml' does not follow the ordering scheme of the minimally required plugins." + ) else: passed.append("'assets/multiqc_config.yaml' contains a matching report_comment.") # Check that the minimum plugins exist and are coming first in the summary try: assert "report_comment" in mqc_yml - assert mqc_yml["report_comment"].trim() == \ - '''This report has been generated by the nf-core/%s + assert ( + mqc_yml["report_comment"].trim() + == """This report has been generated by the nf-core/%s analysis pipeline. For information about how to interpret these results, please see the documentation. - ''' % (self.pipeline_name) + """ + % (self.pipeline_name) + ) except (AssertionError, KeyError, TypeError): failed.append("'assets/multiqc_config.yaml' does not contain a matching report_comment.") else: From 0926f0395d68f16e13b5677ef4ae53676f2161e3 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 16 Mar 2022 15:57:36 +0100 Subject: [PATCH 020/110] Fix github wf variable names in deploy workflow --- .github/workflows/tools-api-docs-dev.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/tools-api-docs-dev.yml b/.github/workflows/tools-api-docs-dev.yml index 8a45922c7c..34feec23f1 100644 --- a/.github/workflows/tools-api-docs-dev.yml +++ b/.github/workflows/tools-api-docs-dev.yml @@ -27,7 +27,7 @@ jobs: - name: Sync dev docs # Only sync with the website if it was a push from nf-core/tools dev branch - if: github.repository == 'nf-core/tools' && github.event.type == 'push' && github.event.base_ref == 'refs/heads/dev' + if: github.repository == 'nf-core/tools' && github.event_name == 'push' && github.event.ref == 'refs/heads/dev' uses: SamKirkland/FTP-Deploy-Action@4.0.0 with: server: ${{ secrets.ftp_server }} From 21d1b4fe881df8c20ea77a95a2b5d725976b9152 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 16 Mar 2022 16:08:38 +0100 Subject: [PATCH 021/110] Remove sphinx scaffold page for old functions_nf.md --- docs/api/_src/module_lint_tests/functions_nf.md | 5 ----- 1 file changed, 5 deletions(-) delete mode 100644 docs/api/_src/module_lint_tests/functions_nf.md diff --git a/docs/api/_src/module_lint_tests/functions_nf.md b/docs/api/_src/module_lint_tests/functions_nf.md deleted file mode 100644 index c89d1d862c..0000000000 --- a/docs/api/_src/module_lint_tests/functions_nf.md +++ /dev/null @@ -1,5 +0,0 @@ -# functions_nf - -```{eval-rst} -.. automethod:: nf_core.modules.lint.ModuleLint.functions_nf -``` From c3224a6429b50254bb27024a9347c369726b1a5c Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 16 Mar 2022 16:11:37 +0100 Subject: [PATCH 022/110] Ran make_lint_md.py --- docs/api/_src/module_lint_tests/module_changes.md | 5 ----- docs/api/_src/module_lint_tests/module_deprecations.md | 5 +++++ 2 files changed, 5 insertions(+), 5 deletions(-) delete mode 100644 docs/api/_src/module_lint_tests/module_changes.md create mode 100644 docs/api/_src/module_lint_tests/module_deprecations.md diff --git a/docs/api/_src/module_lint_tests/module_changes.md b/docs/api/_src/module_lint_tests/module_changes.md deleted file mode 100644 index ce2d428ca9..0000000000 --- a/docs/api/_src/module_lint_tests/module_changes.md +++ /dev/null @@ -1,5 +0,0 @@ -# module_changes - -```{eval-rst} -.. automethod:: nf_core.modules.lint.ModuleLint.module_changes -``` diff --git a/docs/api/_src/module_lint_tests/module_deprecations.md b/docs/api/_src/module_lint_tests/module_deprecations.md new file mode 100644 index 0000000000..f00b4dc54d --- /dev/null +++ b/docs/api/_src/module_lint_tests/module_deprecations.md @@ -0,0 +1,5 @@ +# module_deprecations + +```{eval-rst} +.. automethod:: nf_core.modules.lint.ModuleLint.module_deprecations +``` From 69d8b37a969a1bbb19ad13846a4fa75130932c00 Mon Sep 17 00:00:00 2001 From: jpfeuffer Date: Wed, 16 Mar 2022 15:13:59 +0000 Subject: [PATCH 023/110] refactor version to avoid install. fix multiqc_config lint --- nf_core/__init__.py | 4 +--- nf_core/lint/multiqc_config.py | 26 ++++++++++++++------------ nf_core/version.py | 1 + setup.py | 8 +++++++- 4 files changed, 23 insertions(+), 16 deletions(-) create mode 100644 nf_core/version.py diff --git a/nf_core/__init__.py b/nf_core/__init__.py index 42c3f188f3..a309d7690e 100644 --- a/nf_core/__init__.py +++ b/nf_core/__init__.py @@ -4,6 +4,4 @@ Shouldn't do much, as everything is under subcommands. """ -import pkg_resources - -__version__ = pkg_resources.get_distribution("nf_core").version +from .version import __version__ diff --git a/nf_core/lint/multiqc_config.py b/nf_core/lint/multiqc_config.py index 8204f95224..5d41542f36 100644 --- a/nf_core/lint/multiqc_config.py +++ b/nf_core/lint/multiqc_config.py @@ -1,8 +1,7 @@ #!/usr/bin/env python import os -from logging import warn -from nf_core.modules.modules_command import ModuleCommand +import yaml def multiqc_config(self): @@ -38,7 +37,7 @@ def multiqc_config(self): # Check that the report_comment exists and matches try: assert "report_section_order" in mqc_yml - orders = defaultdict(int) + orders = dict() summary_plugin_name = "nf-core-" + self.pipeline_name + "-summary" min_plugins = ["software_versions", summary_plugin_name] for plugin in min_plugins: @@ -51,26 +50,29 @@ def multiqc_config(self): if "order" in mqc_yml["report_section_order"][plugin]: orders[plugin] = mqc_yml["report_section_order"][plugin]["order"] - assert orders[summary_plugin_name] == min(orders) + assert orders[summary_plugin_name] == min(orders.values()) orders.pop(summary_plugin_name) - assert orders["software_versions"] == min(orders) + assert orders["software_versions"] == min(orders.values()) except (AssertionError, KeyError, TypeError): failed.append( "'assets/multiqc_config.yaml' does not follow the ordering scheme of the minimally required plugins." ) else: - passed.append("'assets/multiqc_config.yaml' contains a matching report_comment.") + passed.append("'assets/multiqc_config.yaml' follows the ordering scheme of the minimally required plugins.") # Check that the minimum plugins exist and are coming first in the summary try: assert "report_comment" in mqc_yml assert ( - mqc_yml["report_comment"].trim() - == """This report has been generated by the nf-core/%s - analysis pipeline. For information about how to interpret these results, please see the - documentation. - """ - % (self.pipeline_name) + mqc_yml["report_comment"].strip() + == 'This report has been generated by the nf-core/%s ' + "analysis pipeline. For information about how to interpret these results, please see the " + 'documentation.' + % ( + self.pipeline_name, + self.pipeline_name, + self.pipeline_name, + ) ) except (AssertionError, KeyError, TypeError): failed.append("'assets/multiqc_config.yaml' does not contain a matching report_comment.") diff --git a/nf_core/version.py b/nf_core/version.py new file mode 100644 index 0000000000..8214ab8925 --- /dev/null +++ b/nf_core/version.py @@ -0,0 +1 @@ +__version__ = "2.4dev" \ No newline at end of file diff --git a/setup.py b/setup.py index 5fda0ae5be..bd793cc528 100644 --- a/setup.py +++ b/setup.py @@ -2,7 +2,13 @@ from setuptools import setup, find_packages -version = "2.4dev" +versionfile = "autowrap/version.py" +try: + execfile(versionfile) +except: + exec(open(versionfile).read()) + +version = __version__ with open("README.md") as f: readme = f.read() From 9176ef4f8b38365ad29f937503ec08ee72e2a194 Mon Sep 17 00:00:00 2001 From: Harshil Patel Date: Wed, 16 Mar 2022 15:18:05 +0000 Subject: [PATCH 024/110] Run nf-core create manually and fix YAMLLint --- .../.github/workflows/awsfulltest.yml | 2 +- .../.github/workflows/awstest.yml | 2 +- .../.github/workflows/branch.yml | 2 +- .../.github/workflows/linting.yml | 2 +- .../.github/workflows/linting_comment.yml | 4 +- nf_core/pipeline-template/modules.json | 4 +- .../modules/nf-core/modules/fastqc/main.nf | 3 + .../modules/nf-core/modules/fastqc/meta.yml | 90 +++++++++---------- .../modules/nf-core/modules/multiqc/main.nf | 9 +- .../modules/nf-core/modules/multiqc/meta.yml | 66 +++++++------- 10 files changed, 95 insertions(+), 89 deletions(-) diff --git a/nf_core/pipeline-template/.github/workflows/awsfulltest.yml b/nf_core/pipeline-template/.github/workflows/awsfulltest.yml index d2ba27e3a2..874149a936 100644 --- a/nf_core/pipeline-template/.github/workflows/awsfulltest.yml +++ b/nf_core/pipeline-template/.github/workflows/awsfulltest.yml @@ -18,7 +18,7 @@ jobs: # TODO nf-core: You can customise AWS full pipeline tests as required # Add full size test data (but still relatively small datasets for few samples) # on the `test_full.config` test runs with only one set of parameters - {% raw %} + {%- raw %} with: workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }} access_token: ${{ secrets.TOWER_ACCESS_TOKEN }} diff --git a/nf_core/pipeline-template/.github/workflows/awstest.yml b/nf_core/pipeline-template/.github/workflows/awstest.yml index b5ceb32297..99f64ec34a 100644 --- a/nf_core/pipeline-template/.github/workflows/awstest.yml +++ b/nf_core/pipeline-template/.github/workflows/awstest.yml @@ -12,7 +12,7 @@ jobs: steps: - name: Launch workflow via tower uses: nf-core/tower-action@v3 - {% raw %} + {%- raw %} with: workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }} access_token: ${{ secrets.TOWER_ACCESS_TOKEN }} diff --git a/nf_core/pipeline-template/.github/workflows/branch.yml b/nf_core/pipeline-template/.github/workflows/branch.yml index 281d1af157..5281ada9cb 100644 --- a/nf_core/pipeline-template/.github/workflows/branch.yml +++ b/nf_core/pipeline-template/.github/workflows/branch.yml @@ -43,4 +43,4 @@ jobs: Thanks again for your contribution! repo-token: ${{ secrets.GITHUB_TOKEN }} allow-repeats: false -{% endraw %} +{%- endraw %} diff --git a/nf_core/pipeline-template/.github/workflows/linting.yml b/nf_core/pipeline-template/.github/workflows/linting.yml index 76449396ee..28b0c97095 100644 --- a/nf_core/pipeline-template/.github/workflows/linting.yml +++ b/nf_core/pipeline-template/.github/workflows/linting.yml @@ -138,4 +138,4 @@ jobs: lint_log.txt lint_results.md PR_number.txt -{% endraw %} +{%- endraw %} diff --git a/nf_core/pipeline-template/.github/workflows/linting_comment.yml b/nf_core/pipeline-template/.github/workflows/linting_comment.yml index 68a0feee18..edd997325e 100644 --- a/nf_core/pipeline-template/.github/workflows/linting_comment.yml +++ b/nf_core/pipeline-template/.github/workflows/linting_comment.yml @@ -1,4 +1,4 @@ -{% raw %} +{% raw -%} name: nf-core linting comment # This workflow is triggered after the linting action is complete # It posts an automated comment to the PR, even if the PR is coming from a fork @@ -27,4 +27,4 @@ jobs: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} number: ${{ steps.pr_number.outputs.pr_number }} path: linting-logs/lint_results.md -{% endraw %} +{%- endraw %} diff --git a/nf_core/pipeline-template/modules.json b/nf_core/pipeline-template/modules.json index 853ad6a1e3..2c377919a0 100644 --- a/nf_core/pipeline-template/modules.json +++ b/nf_core/pipeline-template/modules.json @@ -7,10 +7,10 @@ "git_sha": "20d8250d9f39ddb05dfb437603aaf99b5c0b2b41" }, "fastqc": { - "git_sha": "9d0cad583b9a71a6509b754fdf589cbfbed08961" + "git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d" }, "multiqc": { - "git_sha": "20d8250d9f39ddb05dfb437603aaf99b5c0b2b41" + "git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d" } } } diff --git a/nf_core/pipeline-template/modules/nf-core/modules/fastqc/main.nf b/nf_core/pipeline-template/modules/nf-core/modules/fastqc/main.nf index d250eca075..ed6b8c50b1 100644 --- a/nf_core/pipeline-template/modules/nf-core/modules/fastqc/main.nf +++ b/nf_core/pipeline-template/modules/nf-core/modules/fastqc/main.nf @@ -15,6 +15,9 @@ process FASTQC { tuple val(meta), path("*.zip") , emit: zip path "versions.yml" , emit: versions + when: + task.ext.when == null || task.ext.when + script: def args = task.ext.args ?: '' // Add soft-links to original FastQs for consistent naming in pipeline diff --git a/nf_core/pipeline-template/modules/nf-core/modules/fastqc/meta.yml b/nf_core/pipeline-template/modules/nf-core/modules/fastqc/meta.yml index b09553a3c3..4da5bb5a06 100644 --- a/nf_core/pipeline-template/modules/nf-core/modules/fastqc/meta.yml +++ b/nf_core/pipeline-template/modules/nf-core/modules/fastqc/meta.yml @@ -1,52 +1,52 @@ name: fastqc description: Run FastQC on sequenced reads keywords: - - quality control - - qc - - adapters - - fastq + - quality control + - qc + - adapters + - fastq tools: - - fastqc: - description: | - FastQC gives general quality metrics about your reads. - It provides information about the quality score distribution - across your reads, the per base sequence content (%A/C/G/T). - You get information about adapter contamination and other - overrepresented sequences. - homepage: https://www.bioinformatics.babraham.ac.uk/projects/fastqc/ - documentation: https://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/ - licence: ['GPL-2.0-only'] + - fastqc: + description: | + FastQC gives general quality metrics about your reads. + It provides information about the quality score distribution + across your reads, the per base sequence content (%A/C/G/T). + You get information about adapter contamination and other + overrepresented sequences. + homepage: https://www.bioinformatics.babraham.ac.uk/projects/fastqc/ + documentation: https://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/ + licence: ["GPL-2.0-only"] input: - - meta: - type: map - description: | - Groovy Map containing sample information - e.g. [ id:'test', single_end:false ] - - reads: - type: file - description: | - List of input FastQ files of size 1 and 2 for single-end and paired-end data, - respectively. + - meta: + type: map + description: | + Groovy Map containing sample information + e.g. [ id:'test', single_end:false ] + - reads: + type: file + description: | + List of input FastQ files of size 1 and 2 for single-end and paired-end data, + respectively. output: - - meta: - type: map - description: | - Groovy Map containing sample information - e.g. [ id:'test', single_end:false ] - - html: - type: file - description: FastQC report - pattern: "*_{fastqc.html}" - - zip: - type: file - description: FastQC report archive - pattern: "*_{fastqc.zip}" - - versions: - type: file - description: File containing software versions - pattern: "versions.yml" + - meta: + type: map + description: | + Groovy Map containing sample information + e.g. [ id:'test', single_end:false ] + - html: + type: file + description: FastQC report + pattern: "*_{fastqc.html}" + - zip: + type: file + description: FastQC report archive + pattern: "*_{fastqc.zip}" + - versions: + type: file + description: File containing software versions + pattern: "versions.yml" authors: - - "@drpatelh" - - "@grst" - - "@ewels" - - "@FelixKrueger" + - "@drpatelh" + - "@grst" + - "@ewels" + - "@FelixKrueger" diff --git a/nf_core/pipeline-template/modules/nf-core/modules/multiqc/main.nf b/nf_core/pipeline-template/modules/nf-core/modules/multiqc/main.nf index 3dceb162a3..1264aac1eb 100644 --- a/nf_core/pipeline-template/modules/nf-core/modules/multiqc/main.nf +++ b/nf_core/pipeline-template/modules/nf-core/modules/multiqc/main.nf @@ -1,10 +1,10 @@ process MULTIQC { label 'process_medium' - conda (params.enable_conda ? 'bioconda::multiqc=1.11' : null) + conda (params.enable_conda ? 'bioconda::multiqc=1.12' : null) container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ? - 'https://depot.galaxyproject.org/singularity/multiqc:1.11--pyhdfd78af_0' : - 'quay.io/biocontainers/multiqc:1.11--pyhdfd78af_0' }" + 'https://depot.galaxyproject.org/singularity/multiqc:1.12--pyhdfd78af_0' : + 'quay.io/biocontainers/multiqc:1.12--pyhdfd78af_0' }" input: path multiqc_files @@ -15,6 +15,9 @@ process MULTIQC { path "*_plots" , optional:true, emit: plots path "versions.yml" , emit: versions + when: + task.ext.when == null || task.ext.when + script: def args = task.ext.args ?: '' """ diff --git a/nf_core/pipeline-template/modules/nf-core/modules/multiqc/meta.yml b/nf_core/pipeline-template/modules/nf-core/modules/multiqc/meta.yml index 63c75a450a..6fa891efc2 100644 --- a/nf_core/pipeline-template/modules/nf-core/modules/multiqc/meta.yml +++ b/nf_core/pipeline-template/modules/nf-core/modules/multiqc/meta.yml @@ -1,40 +1,40 @@ name: MultiQC description: Aggregate results from bioinformatics analyses across many samples into a single report keywords: - - QC - - bioinformatics tools - - Beautiful stand-alone HTML report + - QC + - bioinformatics tools + - Beautiful stand-alone HTML report tools: - - multiqc: - description: | - MultiQC searches a given directory for analysis logs and compiles a HTML report. - It's a general use tool, perfect for summarising the output from numerous bioinformatics tools. - homepage: https://multiqc.info/ - documentation: https://multiqc.info/docs/ - licence: ['GPL-3.0-or-later'] + - multiqc: + description: | + MultiQC searches a given directory for analysis logs and compiles a HTML report. + It's a general use tool, perfect for summarising the output from numerous bioinformatics tools. + homepage: https://multiqc.info/ + documentation: https://multiqc.info/docs/ + licence: ["GPL-3.0-or-later"] input: - - multiqc_files: - type: file - description: | - List of reports / files recognised by MultiQC, for example the html and zip output of FastQC + - multiqc_files: + type: file + description: | + List of reports / files recognised by MultiQC, for example the html and zip output of FastQC output: - - report: - type: file - description: MultiQC report file - pattern: "multiqc_report.html" - - data: - type: dir - description: MultiQC data dir - pattern: "multiqc_data" - - plots: - type: file - description: Plots created by MultiQC - pattern: "*_data" - - versions: - type: file - description: File containing software versions - pattern: "versions.yml" + - report: + type: file + description: MultiQC report file + pattern: "multiqc_report.html" + - data: + type: dir + description: MultiQC data dir + pattern: "multiqc_data" + - plots: + type: file + description: Plots created by MultiQC + pattern: "*_data" + - versions: + type: file + description: File containing software versions + pattern: "versions.yml" authors: - - "@abhi18av" - - "@bunop" - - "@drpatelh" + - "@abhi18av" + - "@bunop" + - "@drpatelh" From 01597b7ca0cab3fb92ff124217794a60acf25ca7 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 16 Mar 2022 16:24:27 +0100 Subject: [PATCH 025/110] Fix broken links --- docs/api/_src/api/lint.md | 2 +- docs/api/_src/index.md | 10 ++++------ 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/docs/api/_src/api/lint.md b/docs/api/_src/api/lint.md index a0e3a6d5c9..cb75b0ecb5 100644 --- a/docs/api/_src/api/lint.md +++ b/docs/api/_src/api/lint.md @@ -1,7 +1,7 @@ # nf_core.lint :::{seealso} -See the [Lint Tests](../lint_tests/index.html) docs for information about specific linting functions. +See the [Lint Tests](../pipeline_lint_tests/index.md) docs for information about specific linting functions. ::: ```{eval-rst} diff --git a/docs/api/_src/index.md b/docs/api/_src/index.md index a6fb767122..de39e7244b 100644 --- a/docs/api/_src/index.md +++ b/docs/api/_src/index.md @@ -13,12 +13,10 @@ api/index.rst This documentation is for the `nf-core/tools` package. -Primarily, it describes the different [code lint tests](lint_tests/index.html) -run by `nf-core lint` (typically visited by a developer when their pipeline fails a given -test), and also reference for the `nf_core` [Python package API](api/index.html). - -## Indices and tables +## Contents +- [Pipeline code lint tests](pipeline_lint_tests/index.md) (run by `nf-core lint`) +- [Module code lint tests](module_lint_tests/index.md) (run by `nf-core modules lint`) +- [nf-core/tools Python package API reference](api/index.md) - {ref}`genindex` - {ref}`modindex` -- {ref}`search` From edbb9cf10ac8748b82a02b2c2fb4fa7efc5e95e1 Mon Sep 17 00:00:00 2001 From: jpfeuffer Date: Wed, 16 Mar 2022 15:31:17 +0000 Subject: [PATCH 026/110] black version.py --- nf_core/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/version.py b/nf_core/version.py index 8214ab8925..2c4485c6c0 100644 --- a/nf_core/version.py +++ b/nf_core/version.py @@ -1 +1 @@ -__version__ = "2.4dev" \ No newline at end of file +__version__ = "2.4dev" From dbc64982355b0a6d7d3fea5741a4b0576d08d47f Mon Sep 17 00:00:00 2001 From: jpfeuffer Date: Wed, 16 Mar 2022 15:33:02 +0000 Subject: [PATCH 027/110] fix copypaste error --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index bd793cc528..dc28528e26 100644 --- a/setup.py +++ b/setup.py @@ -2,7 +2,7 @@ from setuptools import setup, find_packages -versionfile = "autowrap/version.py" +versionfile = "nf_core/version.py" try: execfile(versionfile) except: From 487e807f926139fe23931c6eeaf5aeca728e0b21 Mon Sep 17 00:00:00 2001 From: jpfeuffer Date: Wed, 16 Mar 2022 15:43:27 +0000 Subject: [PATCH 028/110] wrong ending --- .../pipeline_lint_tests/{multiqc_config.rst => multiqc_config.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/api/_src/pipeline_lint_tests/{multiqc_config.rst => multiqc_config.md} (100%) diff --git a/docs/api/_src/pipeline_lint_tests/multiqc_config.rst b/docs/api/_src/pipeline_lint_tests/multiqc_config.md similarity index 100% rename from docs/api/_src/pipeline_lint_tests/multiqc_config.rst rename to docs/api/_src/pipeline_lint_tests/multiqc_config.md From 83da199150dd7128cec559156915cb61949f4fc2 Mon Sep 17 00:00:00 2001 From: Julianus Pfeuffer Date: Wed, 16 Mar 2022 17:18:53 +0100 Subject: [PATCH 029/110] [lint] spaces before comment --- nf_core/pipeline-template/.gitpod.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/.gitpod.yml b/nf_core/pipeline-template/.gitpod.yml index b7d4cee18b..2b9ef8bf95 100644 --- a/nf_core/pipeline-template/.gitpod.yml +++ b/nf_core/pipeline-template/.gitpod.yml @@ -1,7 +1,7 @@ image: nfcore/gitpod:latest vscode: - extensions: # based on nf-core.nf-core-extensionpack + extensions: # based on nf-core.nf-core-extensionpack - codezombiech.gitignore # Language support for .gitignore files # - cssho.vscode-svgviewer # SVG viewer - davidanson.vscode-markdownlint # Markdown/CommonMark linting and style checking for Visual Studio Code From bd39506081efcc3ffd7258674704067659bc333c Mon Sep 17 00:00:00 2001 From: Matthias Zepper Date: Wed, 16 Mar 2022 20:24:19 +0100 Subject: [PATCH 030/110] Added a new light color scheme --- docs/api/_src/_static/css/custom.css | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/docs/api/_src/_static/css/custom.css b/docs/api/_src/_static/css/custom.css index 9bd4341651..b9866f438e 100644 --- a/docs/api/_src/_static/css/custom.css +++ b/docs/api/_src/_static/css/custom.css @@ -21,7 +21,7 @@ code, .rst-content code.literal{ background-color: rgba(220,220,220,0.4); - color: #db9444; + color: #C03221; border: none; } .rst-content .note .admonition-title{ @@ -35,9 +35,6 @@ .rst-content .method>dt>code { color: #f5f6f7; } - .sig-paren{ - color: #db9444; - } .rst-content div[class^=highlight], .rst-content pre.literal-block{ border:none; } @@ -50,19 +47,26 @@ background-color: #585b60fc; } .highlight .k, .highlight .nt, .highlight .no{ - color: #6fb2e8; + color: #246EB9; } .highlight .s, .highlight .s1, .highlight .s2{ color: #32AD65; } .highlight .nb, .highlight .o{ - color: #db9444; + color: #FFBE0B; } .highlight .c1{ color: #88898afc; } - .highlight .nv{ - color: #f9d977; + .highlight .nv, .py>span{ + color: #C03221; + } + .py .sig-param, .py .sig-paren, .property .pre{ + color: #EEF0F2; + } + + .method > .py .sig-param, .method > .py .sig-paren{ + color: #32AD65; } .btn.btn-neutral { background-color: #e5e6e7d4 !important; @@ -80,8 +84,8 @@ } -.rst-content .admonition-todo .admonition-title, .rst-content .admonition-todo .wy-alert-title, .rst-content .attention .admonition-title, .rst-content .attention .wy-alert-title, .rst-content .caution .admonition-title, .rst-content .caution .wy-alert-title, .rst-content .warning .admonition-title, .rst-content .warning .wy-alert-title, .rst-content .wy-alert-warning.admonition .admonition-title, .rst-content .wy-alert-warning.admonition .wy-alert-title, .rst-content .wy-alert-warning.danger .admonition-title, .rst-content .wy-alert-warning.danger .wy-alert-title, .rst-content .wy-alert-warning.error .admonition-title, .rst-content .wy-alert-warning.error .wy-alert-title, .rst-content .wy-alert-warning.hint .admonition-title, .rst-content .wy-alert-warning.hint .wy-alert-title, .rst-content .wy-alert-warning.important .admonition-title, .rst-content .wy-alert-warning.important .wy-alert-title, .rst-content .wy-alert-warning.note .admonition-title, .rst-content .wy-alert-warning.note .wy-alert-title, .rst-content .wy-alert-warning.seealso .admonition-title, .rst-content .wy-alert-warning.seealso .wy-alert-title, .rst-content .wy-alert-warning.tip .admonition-title, .rst-content .wy-alert-warning.tip .wy-alert-title, .rst-content .wy-alert.wy-alert-warning .admonition-title, .wy-alert.wy-alert-warning .rst-content .admonition-title, .wy-alert.wy-alert-warning .wy-alert-title { - background-color: #f4a25b; +.rst-content .admonition-todo .admonition-title, .rst-content .admonition-todo .wy-alert-title, .rst-content .attention .admonition-title, .rst-content .attention .wy-alert-title, .rst-content .caution .admonition-title, .rst-content .caution .wy-alert-title, .rst-content .warning .admonition-title, .rst-content .warning .wy-alert-title, .rst-content .wy-alert-warning.admonition .admonition-title, .rst-content .wy-alert-warning.admonition .wy-alert-title, .rst-content .wy-alert-warning.danger .admonition-title, .rst-content .wy-alert-warning.danger .wy-alert-title, .rst-content .wy-alert-warning.error .admonition-title, .rst-content .wy-alert-warning.error .wy-alert-title, .rst-content .wy-alert-warning.hint .admonition-title, .rst-content .wy-alert-warning.hint .wy-alert-title, .rst-content .seealso .admonition-title, .rst-content .seealso .wy-alert-title, .rst-content .wy-alert-warning.important .admonition-title, .rst-content .wy-alert-warning.important .wy-alert-title, .rst-content .wy-alert-warning.note .admonition-title, .rst-content .wy-alert-warning.note .wy-alert-title, .rst-content .wy-alert-warning.seealso .admonition-title, .rst-content .wy-alert-warning.seealso .wy-alert-title, .rst-content .wy-alert-warning.tip .admonition-title, .rst-content .wy-alert-warning.tip .wy-alert-title, .rst-content .wy-alert.wy-alert-warning .admonition-title, .wy-alert.wy-alert-warning .rst-content .admonition-title, .wy-alert.wy-alert-warning .wy-alert-title { + background-color: #246EB9; } .rst-content .admonition-todo, .rst-content .attention, .rst-content .caution, .rst-content .warning, .rst-content .wy-alert-warning.admonition, .rst-content .wy-alert-warning.danger, .rst-content .wy-alert-warning.error, .rst-content .wy-alert-warning.hint, .rst-content .wy-alert-warning.important, .rst-content .wy-alert-warning.note, .rst-content .wy-alert-warning.seealso, .rst-content .wy-alert-warning.tip, .wy-alert.wy-alert-warning { color: #343434; @@ -189,7 +193,7 @@ border: none; } - .rst-content .admonition-todo .admonition-title, .rst-content .admonition-todo .wy-alert-title, .rst-content .attention .admonition-title, .rst-content .attention .wy-alert-title, .rst-content .caution .admonition-title, .rst-content .caution .wy-alert-title, .rst-content .warning .admonition-title, .rst-content .warning .wy-alert-title, .rst-content .wy-alert-warning.admonition .admonition-title, .rst-content .wy-alert-warning.admonition .wy-alert-title, .rst-content .wy-alert-warning.danger .admonition-title, .rst-content .wy-alert-warning.danger .wy-alert-title, .rst-content .wy-alert-warning.error .admonition-title, .rst-content .wy-alert-warning.error .wy-alert-title, .rst-content .wy-alert-warning.hint .admonition-title, .rst-content .wy-alert-warning.hint .wy-alert-title, .rst-content .wy-alert-warning.important .admonition-title, .rst-content .wy-alert-warning.important .wy-alert-title, .rst-content .wy-alert-warning.note .admonition-title, .rst-content .wy-alert-warning.note .wy-alert-title, .rst-content .wy-alert-warning.seealso .admonition-title, .rst-content .wy-alert-warning.seealso .wy-alert-title, .rst-content .wy-alert-warning.tip .admonition-title, .rst-content .wy-alert-warning.tip .wy-alert-title, .rst-content .wy-alert.wy-alert-warning .admonition-title, .wy-alert.wy-alert-warning .rst-content .admonition-title, .wy-alert.wy-alert-warning .wy-alert-title { + .rst-content .admonition-todo .admonition-title, .rst-content .admonition-todo .wy-alert-title, .rst-content .attention .admonition-title, .rst-content .attention .wy-alert-title, .rst-content .caution .admonition-title, .rst-content .caution .wy-alert-title, .rst-content .warning .admonition-title, .rst-content .warning .wy-alert-title, .rst-content .wy-alert-warning.admonition .admonition-title, .rst-content .wy-alert-warning.admonition .wy-alert-title, .rst-content .wy-alert-warning.danger .admonition-title, .rst-content .wy-alert-warning.danger .wy-alert-title, .rst-content .wy-alert-warning.error .admonition-title, .rst-content .wy-alert-warning.error .wy-alert-title, .rst-content .wy-alert-warning.hint .admonition-title, .rst-content .wy-alert-warning.hint .wy-alert-title, .rst-content .seealso .admonition-title, .rst-content .seealso .wy-alert-title, .rst-content .wy-alert-warning.important .admonition-title, .rst-content .wy-alert-warning.important .wy-alert-title, .rst-content .wy-alert-warning.note .admonition-title, .rst-content .wy-alert-warning.note .wy-alert-title, .rst-content .wy-alert-warning.seealso .admonition-title, .rst-content .wy-alert-warning.seealso .wy-alert-title, .rst-content .wy-alert-warning.tip .admonition-title, .rst-content .wy-alert-warning.tip .wy-alert-title, .rst-content .wy-alert.wy-alert-warning .admonition-title, .wy-alert.wy-alert-warning .rst-content .admonition-title, .wy-alert.wy-alert-warning .wy-alert-title { background-color: #f4a25b; } .rst-content .admonition-todo, .rst-content .attention, .rst-content .caution, .rst-content .warning, .rst-content .wy-alert-warning.admonition, .rst-content .wy-alert-warning.danger, .rst-content .wy-alert-warning.error, .rst-content .wy-alert-warning.hint, .rst-content .wy-alert-warning.important, .rst-content .wy-alert-warning.note, .rst-content .wy-alert-warning.seealso, .rst-content .wy-alert-warning.tip, .wy-alert.wy-alert-warning { From cbcb1b641d06f05d38cf1845bcc4247bc1b3b70d Mon Sep 17 00:00:00 2001 From: JoseEspinosa Date: Wed, 16 Mar 2022 22:10:20 +0100 Subject: [PATCH 031/110] Install tools from the repo itself & not from conda --- CHANGELOG.md | 1 + nf_core/gitpod/gitpod.Dockerfile | 8 +++++++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a63afa13b..7f2a47b616 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ * Convert nf-core/tools API / lint test documentation to MyST ([#1245](https://github.com/nf-core/tools/pull/1245)) * Build documentation for the `nf-core modules lint` tests ([#1250](https://github.com/nf-core/tools/pull/1250)) +* Install tools inside GitPod Docker using the repo itself and not from Conda. ### Modules diff --git a/nf_core/gitpod/gitpod.Dockerfile b/nf_core/gitpod/gitpod.Dockerfile index 2171bd01fc..aa8bfe682d 100644 --- a/nf_core/gitpod/gitpod.Dockerfile +++ b/nf_core/gitpod/gitpod.Dockerfile @@ -21,7 +21,6 @@ RUN conda update -n base -c defaults conda && \ conda install \ openjdk=11.0.13 \ nextflow=21.10.6 \ - nf-core=2.2 \ pytest-workflow=1.6.0 \ mamba=0.22.1 \ pip=22.0.4 \ @@ -30,3 +29,10 @@ RUN conda update -n base -c defaults conda && \ -n base && \ nextflow self-update && \ conda clean --all -f -y + +# Add the nf-core source files to the image +COPY . /usr/src/nf_core +WORKDIR /usr/src/nf_core + +# Install nf-core +RUN python -m pip install . \ No newline at end of file From f8afc0be5d3fb2198c2909c95fd63eb1a3a1d70b Mon Sep 17 00:00:00 2001 From: Julianus Pfeuffer Date: Thu, 17 Mar 2022 11:09:36 +0100 Subject: [PATCH 032/110] revert version handling --- nf_core/__init__.py | 4 +++- nf_core/version.py | 1 - 2 files changed, 3 insertions(+), 2 deletions(-) delete mode 100644 nf_core/version.py diff --git a/nf_core/__init__.py b/nf_core/__init__.py index a309d7690e..42c3f188f3 100644 --- a/nf_core/__init__.py +++ b/nf_core/__init__.py @@ -4,4 +4,6 @@ Shouldn't do much, as everything is under subcommands. """ -from .version import __version__ +import pkg_resources + +__version__ = pkg_resources.get_distribution("nf_core").version diff --git a/nf_core/version.py b/nf_core/version.py deleted file mode 100644 index 2c4485c6c0..0000000000 --- a/nf_core/version.py +++ /dev/null @@ -1 +0,0 @@ -__version__ = "2.4dev" From e271b0e4b77be0626ccc759674b29472413b9943 Mon Sep 17 00:00:00 2001 From: Julianus Pfeuffer Date: Thu, 17 Mar 2022 11:10:56 +0100 Subject: [PATCH 033/110] revert setup.py --- setup.py | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/setup.py b/setup.py index dc28528e26..5fda0ae5be 100644 --- a/setup.py +++ b/setup.py @@ -2,13 +2,7 @@ from setuptools import setup, find_packages -versionfile = "nf_core/version.py" -try: - execfile(versionfile) -except: - exec(open(versionfile).read()) - -version = __version__ +version = "2.4dev" with open("README.md") as f: readme = f.read() From 1cd2740e4826914d828bcb6bec1649f018d50231 Mon Sep 17 00:00:00 2001 From: Matthias Zepper Date: Thu, 17 Mar 2022 11:14:06 +0100 Subject: [PATCH 034/110] Fixed to the dark theme and small additions to the light theme. --- docs/api/_src/_static/css/custom.css | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/docs/api/_src/_static/css/custom.css b/docs/api/_src/_static/css/custom.css index b9866f438e..c892c07aa6 100644 --- a/docs/api/_src/_static/css/custom.css +++ b/docs/api/_src/_static/css/custom.css @@ -1,4 +1,10 @@ @media (prefers-color-scheme: light) { + a, a:visited { + color: #246EB9; + } + a:hover, a:focus, a:active { + color: #C03221; + } .wy-nav-side{ background-color: #ededed; } @@ -49,10 +55,10 @@ .highlight .k, .highlight .nt, .highlight .no{ color: #246EB9; } - .highlight .s, .highlight .s1, .highlight .s2{ + .highlight .s, .highlight .s1, .highlight .s2, .highlight .na{ color: #32AD65; } - .highlight .nb, .highlight .o{ + .highlight .nb, .highlight .o, .highlight .cm{ color: #FFBE0B; } .highlight .c1{ @@ -111,9 +117,12 @@ background-color: #343434; color: #e5e6e7; } - a, a:hover, a:focus, a:active { + a, a:visited { color: #6fb2e8; } + a:hover, a:focus, a:active { + color: #db9444; + } code, .rst-content code.literal{ background-color: rgba(220,220,220,0.1); color: #db9444; @@ -165,10 +174,10 @@ .highlight .k, .highlight .nt, .highlight .no{ color: #6fb2e8; } - .highlight .s, .highlight .s1, .highlight .s2{ + .highlight .s, .highlight .s1, .highlight .s2, .highlight .na{ color: #32AD65; } - .highlight .nb, .highlight .o{ + .highlight .nb, .highlight .o, .highlight .cm{ color: #db9444; } .highlight .c1{ From 36bf35bbd1985974babf10d4c54609d9e623d44b Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 11:23:16 +0100 Subject: [PATCH 035/110] fix indentation in root-level .gitpod.yml file too --- .gitpod.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitpod.yml b/.gitpod.yml index 804eb092df..1f30f26a16 100644 --- a/.gitpod.yml +++ b/.gitpod.yml @@ -3,7 +3,7 @@ tasks: - name: install current state of nf-core/tools command: python -m pip install --upgrade -r requirements-dev.txt -e . vscode: - extensions: # based on nf-core.nf-core-extensionpack + extensions: # based on nf-core.nf-core-extensionpack - codezombiech.gitignore # Language support for .gitignore files # - cssho.vscode-svgviewer # SVG viewer - davidanson.vscode-markdownlint # Markdown/CommonMark linting and style checking for Visual Studio Code From cdd1f9d0cbddbfd3374d7fb0cd95ca50ee5c5724 Mon Sep 17 00:00:00 2001 From: Matthias Zepper Date: Thu, 17 Mar 2022 11:29:08 +0100 Subject: [PATCH 036/110] Changelog update --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4f7a5ddd8f..5b726791db 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,6 +35,7 @@ * Handle exception in `nf-core list` when a broken git repo is found ([#1273](https://github.com/nf-core/tools/issues/1273)) * Updated URL for pipeline lint test docs ([#1348](https://github.com/nf-core/tools/issues/1348)) * Updated `nf-core create` to tolerate failures and retry when fetching pipeline logos from the website ([#1369](https://github.com/nf-core/tools/issues/1369)) +* Modified the CSS overriding `sphinx_rtd_theme` default colors to fix some glitches in the API documentation ([#1294](https://github.com/nf-core/tools/issues/1294)) ### Modules From 7de3b292fc25b5870870546a01fccdf818bd0477 Mon Sep 17 00:00:00 2001 From: Julianus Pfeuffer Date: Thu, 17 Mar 2022 13:08:57 +0100 Subject: [PATCH 037/110] Apply obvious suggestions from code review Co-authored-by: Phil Ewels --- nf_core/lint/multiqc_config.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/nf_core/lint/multiqc_config.py b/nf_core/lint/multiqc_config.py index 5d41542f36..2199faae70 100644 --- a/nf_core/lint/multiqc_config.py +++ b/nf_core/lint/multiqc_config.py @@ -26,7 +26,7 @@ def multiqc_config(self): # Return a failed status if we can't find the file if not os.path.isfile(fn): - return {"failed": ["'assets/multiqc_config.yaml' not found"]} + return {"ignored": ["'assets/multiqc_config.yaml' not found"]} try: with open(fn, "r") as fh: @@ -38,7 +38,7 @@ def multiqc_config(self): try: assert "report_section_order" in mqc_yml orders = dict() - summary_plugin_name = "nf-core-" + self.pipeline_name + "-summary" + summary_plugin_name = f"nf-core-{self.pipeline_name}-summary" min_plugins = ["software_versions", summary_plugin_name] for plugin in min_plugins: assert plugin in mqc_yml["report_section_order"] @@ -53,9 +53,9 @@ def multiqc_config(self): assert orders[summary_plugin_name] == min(orders.values()) orders.pop(summary_plugin_name) assert orders["software_versions"] == min(orders.values()) - except (AssertionError, KeyError, TypeError): + except (AssertionError, KeyError, TypeError) as e: failed.append( - "'assets/multiqc_config.yaml' does not follow the ordering scheme of the minimally required plugins." + f"'assets/multiqc_config.yaml' does not meet requirements: {e}" ) else: passed.append("'assets/multiqc_config.yaml' follows the ordering scheme of the minimally required plugins.") @@ -75,7 +75,7 @@ def multiqc_config(self): ) ) except (AssertionError, KeyError, TypeError): - failed.append("'assets/multiqc_config.yaml' does not contain a matching report_comment.") + warned.append("'assets/multiqc_config.yaml' does not contain a matching report_comment.") else: passed.append("'assets/multiqc_config.yaml' contains a matching report_comment.") From 61cd9494d3a03fd45fb40d1dc6fe0b1a54599483 Mon Sep 17 00:00:00 2001 From: jpfeuffer Date: Thu, 17 Mar 2022 12:25:49 +0000 Subject: [PATCH 038/110] Address more comments --- nf_core/lint/multiqc_config.py | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-) diff --git a/nf_core/lint/multiqc_config.py b/nf_core/lint/multiqc_config.py index 2199faae70..e2f4ffd951 100644 --- a/nf_core/lint/multiqc_config.py +++ b/nf_core/lint/multiqc_config.py @@ -20,6 +20,7 @@ def multiqc_config(self): export_plots: true """ passed = [] + warned = [] failed = [] fn = os.path.join(self.wf_path, "assets", "multiqc_config.yaml") @@ -41,22 +42,24 @@ def multiqc_config(self): summary_plugin_name = f"nf-core-{self.pipeline_name}-summary" min_plugins = ["software_versions", summary_plugin_name] for plugin in min_plugins: - assert plugin in mqc_yml["report_section_order"] - assert "order" in mqc_yml["report_section_order"][plugin] + assert plugin in mqc_yml["report_section_order"], f"Section {plugin} missing in report_section_order" + assert "order" in mqc_yml["report_section_order"][plugin], f"Section {plugin} 'order' missing. Must be < 0" plugin_order = mqc_yml["report_section_order"][plugin]["order"] - assert plugin_order < 0 + assert plugin_order < 0, f"Section {plugin} 'order' must be < 0" for plugin in mqc_yml["report_section_order"]: if "order" in mqc_yml["report_section_order"][plugin]: orders[plugin] = mqc_yml["report_section_order"][plugin]["order"] - assert orders[summary_plugin_name] == min(orders.values()) + assert orders[summary_plugin_name] == min( + orders.values() + ), f"Section {summary_plugin_name} should have the lowest order" orders.pop(summary_plugin_name) - assert orders["software_versions"] == min(orders.values()) + assert orders["software_versions"] == min( + orders.values() + ), f"Section software_versions should have the second lowest order" except (AssertionError, KeyError, TypeError) as e: - failed.append( - f"'assets/multiqc_config.yaml' does not meet requirements: {e}" - ) + failed.append(f"'assets/multiqc_config.yaml' does not meet requirements: {e}") else: passed.append("'assets/multiqc_config.yaml' follows the ordering scheme of the minimally required plugins.") @@ -65,14 +68,9 @@ def multiqc_config(self): assert "report_comment" in mqc_yml assert ( mqc_yml["report_comment"].strip() - == 'This report has been generated by the nf-core/%s ' + == f'This report has been generated by the nf-core/{self.pipeline_name} ' "analysis pipeline. For information about how to interpret these results, please see the " - 'documentation.' - % ( - self.pipeline_name, - self.pipeline_name, - self.pipeline_name, - ) + 'documentation.' ) except (AssertionError, KeyError, TypeError): warned.append("'assets/multiqc_config.yaml' does not contain a matching report_comment.") @@ -88,4 +86,4 @@ def multiqc_config(self): else: passed.append("'assets/multiqc_config.yaml' contains 'export_plots: true'.") - return {"passed": passed, "failed": failed} + return {"passed": passed, "warned": warned, "failed": failed} From d9b5251b78c818395291d2c27250b56d28fb46e2 Mon Sep 17 00:00:00 2001 From: Sarah Guinchard Date: Thu, 17 Mar 2022 13:34:42 +0100 Subject: [PATCH 039/110] Added Singularity install tutorial + clarified conda usage --- nf_core/pipeline-template/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/README.md b/nf_core/pipeline-template/README.md index e0049a72e9..2239eaf90c 100644 --- a/nf_core/pipeline-template/README.md +++ b/nf_core/pipeline-template/README.md @@ -35,7 +35,7 @@ On release, automated continuous integration tests run the pipeline on a full-si 1. Install [`Nextflow`](https://www.nextflow.io/docs/latest/getstarted.html#installation) (`>=21.10.3`) -2. Install any of [`Docker`](https://docs.docker.com/engine/installation/), [`Singularity`](https://www.sylabs.io/guides/3.0/user-guide/), [`Podman`](https://podman.io/), [`Shifter`](https://nersc.gitlab.io/development/shifter/how-to-use/) or [`Charliecloud`](https://hpc.github.io/charliecloud/) for full pipeline reproducibility _(please only use [`Conda`](https://conda.io/miniconda.html) as a last resort; see [docs](https://nf-co.re/usage/configuration#basic-configuration-profiles))_ +2. Install any of [`Docker`](https://docs.docker.com/engine/installation/), [`Singularity`](https://www.sylabs.io/guides/3.0/user-guide/) (you can follow [this tutorial](https://singularity-tutorial.github.io/01-installation/)), [`Podman`](https://podman.io/), [`Shifter`](https://nersc.gitlab.io/development/shifter/how-to-use/) or [`Charliecloud`](https://hpc.github.io/charliecloud/) for full pipeline reproducibility *(you can use [`Conda`](https://conda.io/miniconda.html) to install NextFlow but please only use it as a last resort for running the pipeline; see [docs](https://nf-co.re/usage/configuration#basic-configuration-profiles))*. 3. Download the pipeline and test it on a minimal dataset with a single command: From 57d701ef9948d3142c80874310739abff61e92aa Mon Sep 17 00:00:00 2001 From: Sarah Guinchard Date: Thu, 17 Mar 2022 13:38:28 +0100 Subject: [PATCH 040/110] Updated changelog --- nf_core/pipeline-template/CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/nf_core/pipeline-template/CHANGELOG.md b/nf_core/pipeline-template/CHANGELOG.md index c9bd47f145..5fb6220313 100644 --- a/nf_core/pipeline-template/CHANGELOG.md +++ b/nf_core/pipeline-template/CHANGELOG.md @@ -10,6 +10,7 @@ Initial release of {{ name }}, created with the [nf-core](https://nf-co.re/) tem ### `Added` ### `Fixed` + - Clarified conda usage and added an installation tutorial for Singularity since the one on Syllabs' website uses an outdate version of GO Compiler ### `Dependencies` From 89886fd4f5b0e510703d34b8d66f1000138f399b Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 13:57:26 +0100 Subject: [PATCH 041/110] Update docs api for multiqc_config.md to match new rendering style syntax --- docs/api/_src/pipeline_lint_tests/multiqc_config.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/api/_src/pipeline_lint_tests/multiqc_config.md b/docs/api/_src/pipeline_lint_tests/multiqc_config.md index 1a68bc2c21..311f6b3049 100644 --- a/docs/api/_src/pipeline_lint_tests/multiqc_config.md +++ b/docs/api/_src/pipeline_lint_tests/multiqc_config.md @@ -1,4 +1,5 @@ -multiqc_config -=============== +# multiqc_config +```{eval-rst} .. automethod:: nf_core.lint.PipelineLint.multiqc_config +``` From 3c2c6189d11bf42b8edfdf87cd243427cf541eb7 Mon Sep 17 00:00:00 2001 From: Sarah Guinchard Date: Thu, 17 Mar 2022 14:15:27 +0100 Subject: [PATCH 042/110] compliying to linting --- nf_core/pipeline-template/CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/CHANGELOG.md b/nf_core/pipeline-template/CHANGELOG.md index 5fb6220313..c62fdcbe82 100644 --- a/nf_core/pipeline-template/CHANGELOG.md +++ b/nf_core/pipeline-template/CHANGELOG.md @@ -10,7 +10,7 @@ Initial release of {{ name }}, created with the [nf-core](https://nf-co.re/) tem ### `Added` ### `Fixed` - - Clarified conda usage and added an installation tutorial for Singularity since the one on Syllabs' website uses an outdate version of GO Compiler +Clarified conda usage and added an installation tutorial for Singularity since the one on Syllabs' website uses an outdate version of GO Compiler ### `Dependencies` From c2763870db94297657355acdcca6c2e15fdd7a75 Mon Sep 17 00:00:00 2001 From: Sarah Guinchard Date: Thu, 17 Mar 2022 14:20:52 +0100 Subject: [PATCH 043/110] Adjust to linting --- nf_core/pipeline-template/CHANGELOG.md | 1 + nf_core/pipeline-template/README.md | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/nf_core/pipeline-template/CHANGELOG.md b/nf_core/pipeline-template/CHANGELOG.md index c62fdcbe82..bf576d85fb 100644 --- a/nf_core/pipeline-template/CHANGELOG.md +++ b/nf_core/pipeline-template/CHANGELOG.md @@ -10,6 +10,7 @@ Initial release of {{ name }}, created with the [nf-core](https://nf-co.re/) tem ### `Added` ### `Fixed` + Clarified conda usage and added an installation tutorial for Singularity since the one on Syllabs' website uses an outdate version of GO Compiler ### `Dependencies` diff --git a/nf_core/pipeline-template/README.md b/nf_core/pipeline-template/README.md index 2239eaf90c..45731f7f2c 100644 --- a/nf_core/pipeline-template/README.md +++ b/nf_core/pipeline-template/README.md @@ -35,7 +35,7 @@ On release, automated continuous integration tests run the pipeline on a full-si 1. Install [`Nextflow`](https://www.nextflow.io/docs/latest/getstarted.html#installation) (`>=21.10.3`) -2. Install any of [`Docker`](https://docs.docker.com/engine/installation/), [`Singularity`](https://www.sylabs.io/guides/3.0/user-guide/) (you can follow [this tutorial](https://singularity-tutorial.github.io/01-installation/)), [`Podman`](https://podman.io/), [`Shifter`](https://nersc.gitlab.io/development/shifter/how-to-use/) or [`Charliecloud`](https://hpc.github.io/charliecloud/) for full pipeline reproducibility *(you can use [`Conda`](https://conda.io/miniconda.html) to install NextFlow but please only use it as a last resort for running the pipeline; see [docs](https://nf-co.re/usage/configuration#basic-configuration-profiles))*. +2. Install any of [`Docker`](https://docs.docker.com/engine/installation/), [`Singularity`](https://www.sylabs.io/guides/3.0/user-guide/) (you can follow [this tutorial](https://singularity-tutorial.github.io/01-installation/)), [`Podman`](https://podman.io/), [`Shifter`](https://nersc.gitlab.io/development/shifter/how-to-use/) or [`Charliecloud`](https://hpc.github.io/charliecloud/) for full pipeline reproducibility *(you can use [`Conda`](https://conda.io/miniconda.html) to install NextFlow but please only use it as a last resort for running the pipeline; see [docs](https://nf-co.re/usage/configuration#basic-configuration-profiles))*. 3. Download the pipeline and test it on a minimal dataset with a single command: @@ -90,4 +90,4 @@ You can cite the `nf-core` publication as follows: > > Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen. > -> _Nat Biotechnol._ 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x). +> *Nat Biotechnol.* 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x). From 57ca562136d882740dc2afd4901a41725838cea4 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 14:20:53 +0100 Subject: [PATCH 044/110] Docstring code block --- nf_core/lint/multiqc_config.py | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/nf_core/lint/multiqc_config.py b/nf_core/lint/multiqc_config.py index e2f4ffd951..aa9ce5d33d 100644 --- a/nf_core/lint/multiqc_config.py +++ b/nf_core/lint/multiqc_config.py @@ -7,17 +7,19 @@ def multiqc_config(self): """Make sure basic multiQC plugins are installed and plots are exported Basic template: - report_comment: > - This report has been generated by the nf-core/quantms - analysis pipeline. For information about how to interpret these results, please see the - documentation. - report_section_order: - software_versions: - order: -1000 - nf-core-quantms-summary: - order: -1001 - export_plots: true + .. code-block:: yaml + report_comment: > + This report has been generated by the nf-core/quantms + analysis pipeline. For information about how to interpret these results, please see the + documentation. + report_section_order: + software_versions: + order: -1000 + nf-core-quantms-summary: + order: -1001 + + export_plots: true """ passed = [] warned = [] From f03abd3705737e299e46a1b42bd624d816e4bee7 Mon Sep 17 00:00:00 2001 From: Sarah G <99276764+G-Sarah@users.noreply.github.com> Date: Thu, 17 Mar 2022 14:36:46 +0100 Subject: [PATCH 045/110] Restore underscore emphasis Co-authored-by: Phil Ewels --- nf_core/pipeline-template/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/README.md b/nf_core/pipeline-template/README.md index 45731f7f2c..1ff811f6d9 100644 --- a/nf_core/pipeline-template/README.md +++ b/nf_core/pipeline-template/README.md @@ -90,4 +90,4 @@ You can cite the `nf-core` publication as follows: > > Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen. > -> *Nat Biotechnol.* 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x). +> _Nat Biotechnol._ 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x). From 1f199c147a96674cdfa7b8111b0697bf55e9cd77 Mon Sep 17 00:00:00 2001 From: Sarah G <99276764+G-Sarah@users.noreply.github.com> Date: Thu, 17 Mar 2022 14:37:24 +0100 Subject: [PATCH 046/110] Correction of list identation Co-authored-by: Phil Ewels --- nf_core/pipeline-template/CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/CHANGELOG.md b/nf_core/pipeline-template/CHANGELOG.md index bf576d85fb..75c4211992 100644 --- a/nf_core/pipeline-template/CHANGELOG.md +++ b/nf_core/pipeline-template/CHANGELOG.md @@ -11,7 +11,7 @@ Initial release of {{ name }}, created with the [nf-core](https://nf-co.re/) tem ### `Fixed` -Clarified conda usage and added an installation tutorial for Singularity since the one on Syllabs' website uses an outdate version of GO Compiler +- Clarified conda usage and added an installation tutorial for Singularity since the one on Syllabs' website uses an outdate version of GO Compiler ### `Dependencies` From c69dd57bae8575e50effee897e2703493148d9b8 Mon Sep 17 00:00:00 2001 From: Sarah G <99276764+G-Sarah@users.noreply.github.com> Date: Thu, 17 Mar 2022 14:38:01 +0100 Subject: [PATCH 047/110] Accept correction about conda usage Co-authored-by: Phil Ewels --- nf_core/pipeline-template/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/README.md b/nf_core/pipeline-template/README.md index 1ff811f6d9..2355cff750 100644 --- a/nf_core/pipeline-template/README.md +++ b/nf_core/pipeline-template/README.md @@ -35,7 +35,7 @@ On release, automated continuous integration tests run the pipeline on a full-si 1. Install [`Nextflow`](https://www.nextflow.io/docs/latest/getstarted.html#installation) (`>=21.10.3`) -2. Install any of [`Docker`](https://docs.docker.com/engine/installation/), [`Singularity`](https://www.sylabs.io/guides/3.0/user-guide/) (you can follow [this tutorial](https://singularity-tutorial.github.io/01-installation/)), [`Podman`](https://podman.io/), [`Shifter`](https://nersc.gitlab.io/development/shifter/how-to-use/) or [`Charliecloud`](https://hpc.github.io/charliecloud/) for full pipeline reproducibility *(you can use [`Conda`](https://conda.io/miniconda.html) to install NextFlow but please only use it as a last resort for running the pipeline; see [docs](https://nf-co.re/usage/configuration#basic-configuration-profiles))*. +2. Install any of [`Docker`](https://docs.docker.com/engine/installation/), [`Singularity`](https://www.sylabs.io/guides/3.0/user-guide/) (you can follow [this tutorial](https://singularity-tutorial.github.io/01-installation/)), [`Podman`](https://podman.io/), [`Shifter`](https://nersc.gitlab.io/development/shifter/how-to-use/) or [`Charliecloud`](https://hpc.github.io/charliecloud/) for full pipeline reproducibility _(you can use [`Conda`](https://conda.io/miniconda.html) for workflow software but please only use it as a last resort for running the pipeline; see [docs](https://nf-co.re/usage/configuration#basic-configuration-profiles))_. 3. Download the pipeline and test it on a minimal dataset with a single command: From 421ba406c1e332f281e0e778936a6698bef67ffb Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 15:29:34 +0100 Subject: [PATCH 048/110] Merge gitpod docker build actions workflow into existing docker wfs --- .github/workflows/build_gitpod.yml | 45 -------------------- .github/workflows/push_dockerhub_dev.yml | 5 ++- .github/workflows/push_dockerhub_release.yml | 7 ++- 3 files changed, 10 insertions(+), 47 deletions(-) delete mode 100644 .github/workflows/build_gitpod.yml diff --git a/.github/workflows/build_gitpod.yml b/.github/workflows/build_gitpod.yml deleted file mode 100644 index 7cdc5aa4d8..0000000000 --- a/.github/workflows/build_gitpod.yml +++ /dev/null @@ -1,45 +0,0 @@ -name: nf-core gitpod build and push -# Builds the docker image for gitpod and pushes it to DockerHub - -on: - push: - branches: - - 'master' - - 'dev' - paths: - - 'nf_core/gitpod/gitpod.Dockerfile' - - '.github/workflows/build_gitpod.yml' - -jobs: - push_to_registry: - name: Push Docker image to Docker Hub - runs-on: ubuntu-latest - # Only run for the nf-core repo, for releases and merged PRs - if: ${{ github.repository == 'nf-core/tools' }} - steps: - - name: Check out the repo - uses: actions/checkout@v2 - - - name: Log in to Docker Hub - uses: docker/login-action@v1 - with: - username: ${{ secrets.DOCKERHUB_USERNAME }} - password: ${{ secrets.DOCKERHUB_PASS }} - - - name: Extract metadata (tags, labels) for Docker - id: meta - uses: docker/metadata-action@v3 - with: - images: nfcore/gitpod - tags: | - type=semver,pattern={{version}} - type=raw,value=latest - - - name: Build and push Docker image - uses: docker/build-push-action@v2 - with: - context: . - file: nf_core/gitpod/gitpod.Dockerfile - push: true - tags: ${{ steps.meta.outputs.tags }} - labels: ${{ steps.meta.outputs.labels }} diff --git a/.github/workflows/push_dockerhub_dev.yml b/.github/workflows/push_dockerhub_dev.yml index 3a0e789d5b..beab5cd536 100644 --- a/.github/workflows/push_dockerhub_dev.yml +++ b/.github/workflows/push_dockerhub_dev.yml @@ -21,9 +21,12 @@ jobs: uses: actions/checkout@v2 - name: Build new docker image - run: docker build --no-cache . -t nfcore/tools:dev + run: | + docker build --no-cache . -t nfcore/tools:dev + docker build --no-cache nf_core/gitpod/gitpod.Dockerfile -t nfcore/gitpod:dev - name: Push Docker image to DockerHub (dev) run: | echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin docker push nfcore/tools:dev + docker push nfcore/gitpod:dev diff --git a/.github/workflows/push_dockerhub_release.yml b/.github/workflows/push_dockerhub_release.yml index 2aebb34400..11d3124767 100644 --- a/.github/workflows/push_dockerhub_release.yml +++ b/.github/workflows/push_dockerhub_release.yml @@ -21,7 +21,9 @@ jobs: uses: actions/checkout@v2 - name: Build new docker image - run: docker build --no-cache . -t nfcore/tools:latest + run: | + docker build --no-cache . -t nfcore/tools:latest + docker build --no-cache nf_core/gitpod/gitpod.Dockerfile -t nfcore/gitpod:latest - name: Push Docker image to DockerHub (release) run: | @@ -29,3 +31,6 @@ jobs: docker push nfcore/tools:latest docker tag nfcore/tools:latest nfcore/tools:${{ github.event.release.tag_name }} docker push nfcore/tools:${{ github.event.release.tag_name }} + docker push nfcore/gitpod:latest + docker tag nfcore/gitpod:latest nfcore/tools:${{ github.event.release.tag_name }} + docker push nfcore/gitpod:${{ github.event.release.tag_name }} From 66b1409f676b48d443b221c6e6e76f6884800ebf Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 16:17:31 +0100 Subject: [PATCH 049/110] Add Prettier, update actions linting workflows --- .github/workflows/create-lint-wf.yml | 28 ++++++++++--- .../{python-lint.yml => lint-code.yml} | 25 +++++++---- .github/workflows/markdown-lint.yml | 42 ------------------- .github/workflows/yamllint.yml | 39 ----------------- .prettierrc.yaml | 1 + 5 files changed, 40 insertions(+), 95 deletions(-) rename .github/workflows/{python-lint.yml => lint-code.yml} (76%) delete mode 100644 .github/workflows/markdown-lint.yml delete mode 100644 .github/workflows/yamllint.yml create mode 100644 .prettierrc.yaml diff --git a/.github/workflows/create-lint-wf.yml b/.github/workflows/create-lint-wf.yml index d714477651..dda50ae27d 100644 --- a/.github/workflows/create-lint-wf.yml +++ b/.github/workflows/create-lint-wf.yml @@ -1,4 +1,4 @@ -name: Create a pipeline and lint it +name: Create a pipeline and run `nf-core lint`` on: [push, pull_request] env: @@ -15,15 +15,17 @@ jobs: # Nextflow versions include: # Test pipeline minimum Nextflow version - - NXF_VER: '21.10.3' - NXF_EDGE: '' + - NXF_VER: "21.10.3" + NXF_EDGE: "" # Test latest edge release of Nextflow - - NXF_VER: '' - NXF_EDGE: '1' + - NXF_VER: "" + NXF_EDGE: "1" steps: + # Get the repo code - uses: actions/checkout@v2 name: Check out source-code repository + # Set up nf-core/tools - name: Set up Python 3.8 uses: actions/setup-python@v1 with: @@ -34,6 +36,7 @@ jobs: python -m pip install --upgrade pip pip install . + # Set up Nextflow - name: Install Nextflow env: NXF_VER: ${{ matrix.NXF_VER }} @@ -44,12 +47,27 @@ jobs: wget -qO- get.nextflow.io | bash sudo mv nextflow /usr/local/bin/ + # Install the Prettier linting stuff + - uses: actions/setup-node@v2 + + - name: Install Prettier + run: npm install -g prettier + + # Built a pipeline from the template - name: nf-core create run: nf-core --log-file log.txt create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface" + # Run all the linters - name: nf-core lint run: nf-core --log-file log.txt lint --dir nf-core-testpipeline --fail-ignored + - name: Run Prettier --check + run: prettier --check ${GITHUB_WORKSPACE} + + - name: Check code lints with Black + uses: psf/black@stable + + # Run the other nf-core commands - name: nf-core list run: nf-core --log-file log.txt list diff --git a/.github/workflows/python-lint.yml b/.github/workflows/lint-code.yml similarity index 76% rename from .github/workflows/python-lint.yml rename to .github/workflows/lint-code.yml index b75327e050..3a40ba8f2b 100644 --- a/.github/workflows/python-lint.yml +++ b/.github/workflows/lint-code.yml @@ -1,14 +1,21 @@ -name: Lint Python -on: - push: - paths: - - '**.py' - pull_request: - paths: - - '**.py' +name: Lint tools code formatting +on: [push, pull_request] jobs: - PythonLint: + Prettier: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + + - uses: actions/setup-node@v2 + + - name: Install Prettier + run: npm install -g prettier + + - name: Run Prettier --check + run: prettier --check ${GITHUB_WORKSPACE} + + PythonBlack: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 diff --git a/.github/workflows/markdown-lint.yml b/.github/workflows/markdown-lint.yml deleted file mode 100644 index a34f573fe8..0000000000 --- a/.github/workflows/markdown-lint.yml +++ /dev/null @@ -1,42 +0,0 @@ -name: Lint Markdown -on: [push, pull_request] - -jobs: - Markdown: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v2 - - - uses: actions/setup-node@v2 - - - name: Install markdownlint - run: npm install -g markdownlint-cli - - - name: Run Markdownlint - run: markdownlint . - - # If the above check failed, post a comment on the PR explaining the failure - - name: Post PR comment - if: failure() - uses: mshick/add-pr-comment@v1 - with: - message: | - ## Markdown linting is failing - - To keep the code consistent with lots of contributors, we run automated code consistency checks. - To fix this CI test, please run: - - * Install `markdownlint-cli` - * On Mac: `brew install markdownlint-cli` - * Everything else: [Install `npm`](https://www.npmjs.com/get-npm) then [install `markdownlint-cli`](https://www.npmjs.com/package/markdownlint-cli) (`npm install -g markdownlint-cli`) - * Fix the markdown errors - * Automatically: `markdownlint . --fix` - * Manually resolve anything left from `markdownlint .` - - Once you push these changes the test should pass, and you can hide this comment :+1: - - We highly recommend setting up markdownlint in your code editor so that this formatting is done automatically on save. Ask about it on Slack for help! - - Thanks again for your contribution! - repo-token: ${{ secrets.GITHUB_TOKEN }} - allow-repeats: false diff --git a/.github/workflows/yamllint.yml b/.github/workflows/yamllint.yml deleted file mode 100644 index f0155a3c60..0000000000 --- a/.github/workflows/yamllint.yml +++ /dev/null @@ -1,39 +0,0 @@ -name: Lint YAML -on: [push, pull_request] -jobs: - YAML: - runs-on: ubuntu-latest - steps: - - name: Checkout - uses: actions/checkout@master - - name: 'Yamllint' - uses: karancode/yamllint-github-action@master - with: - yamllint_file_or_dir: '.' - yamllint_config_filepath: '.yamllint.yml' - - # If the above check failed, post a comment on the PR explaining the failure - - name: Post PR comment - if: failure() - uses: mshick/add-pr-comment@v1 - with: - message: | - ## YAML linting is failing - - To keep the code consistent with lots of contributors, we run automated code consistency checks. - To fix this CI test, please run: - - * Install `yamllint` - * Install `yamllint` following [this](https://yamllint.readthedocs.io/en/stable/quickstart.html#installing-yamllint) - instructions or alternative install it in your [conda environment](https://anaconda.org/conda-forge/yamllint) - * Fix the markdown errors - * Run the test locally: `yamllint $(find . -type f -name "*.yml" -o -name "*.yaml") -c ./.yamllint.yml` - * Fix any reported errors in your YAML files - - Once you push these changes the test should pass, and you can hide this comment :+1: - - We highly recommend setting up yaml-lint in your code editor so that this formatting is done automatically on save. Ask about it on Slack for help! - - Thanks again for your contribution! - repo-token: ${{ secrets.GITHUB_TOKEN }} - allow-repeats: false diff --git a/.prettierrc.yaml b/.prettierrc.yaml new file mode 100644 index 0000000000..c81f9a7660 --- /dev/null +++ b/.prettierrc.yaml @@ -0,0 +1 @@ +printWidth: 120 From 2584cbdb3eae39845670980bbf10abf385f403c8 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 16:24:49 +0100 Subject: [PATCH 050/110] Ignore built API docs --- .prettierignore | 1 + 1 file changed, 1 insertion(+) create mode 100644 .prettierignore diff --git a/.prettierignore b/.prettierignore new file mode 100644 index 0000000000..776f7e8b7b --- /dev/null +++ b/.prettierignore @@ -0,0 +1 @@ +docs/api/_build From 0d0a7e4febe6b761edfac01d5e5fc1cbdc62c1b5 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 16:28:17 +0100 Subject: [PATCH 051/110] Ignore the templates because of jinja2 tags --- .prettierignore | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.prettierignore b/.prettierignore index 776f7e8b7b..26b9d5cd73 100644 --- a/.prettierignore +++ b/.prettierignore @@ -1 +1,6 @@ docs/api/_build + +# Skip the templates as the Jinja2 tags confuse prettier +# Don't worry - we run prettier on the compiled pipeline in create-lint-wf.yml +nf_core/pipeline-template/ +nf_core/module-template/ From 7dbe68c82b042a804272eba34e24077397f1f3a7 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 16:36:14 +0100 Subject: [PATCH 052/110] Actions CI: add a module to the empty template pipeline --- .github/workflows/create-lint-wf.yml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.github/workflows/create-lint-wf.yml b/.github/workflows/create-lint-wf.yml index dda50ae27d..28347d1b8a 100644 --- a/.github/workflows/create-lint-wf.yml +++ b/.github/workflows/create-lint-wf.yml @@ -53,10 +53,14 @@ jobs: - name: Install Prettier run: npm install -g prettier - # Built a pipeline from the template + # Build a pipeline from the template - name: nf-core create run: nf-core --log-file log.txt create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface" + # Build a module from the template + - name: nf-core modules create + run: nf-core --log-file log.txt modules create bpipe --author @nf-core-bot --label process_low --meta + # Run all the linters - name: nf-core lint run: nf-core --log-file log.txt lint --dir nf-core-testpipeline --fail-ignored From 0c0aac485a287c937de6123e8b6132287e8ff150 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 17:05:58 +0100 Subject: [PATCH 053/110] Remove config files for markdownlint and yamllint --- .markdownlint.yml | 18 ------------------ .yamllint.yml | 7 ------- 2 files changed, 25 deletions(-) delete mode 100644 .markdownlint.yml delete mode 100644 .yamllint.yml diff --git a/.markdownlint.yml b/.markdownlint.yml deleted file mode 100644 index 8a3490bcbd..0000000000 --- a/.markdownlint.yml +++ /dev/null @@ -1,18 +0,0 @@ -# Markdownlint configuration file -default: true -line-length: false -ul-indent: - indent: 4 -no-duplicate-header: - siblings_only: true -no-inline-html: - allowed_elements: - - img - - p - - kbd - - details - - summary -# tools only - the {{ jinja variables }} break URLs and cause this to error -no-bare-urls: false -# tools only - suppresses error messages for usage of $ in main README -commands-show-output: false diff --git a/.yamllint.yml b/.yamllint.yml deleted file mode 100644 index 05d5352339..0000000000 --- a/.yamllint.yml +++ /dev/null @@ -1,7 +0,0 @@ -extends: default -ignore: | - nf_core/*-template/ -rules: - document-start: disable - line-length: disable - truthy: disable From 5f80b6f15b669c0b8622c1aa5832d18336eb96d2 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 17:06:13 +0100 Subject: [PATCH 054/110] Run prettier --- .github/CONTRIBUTING.md | 2 +- .github/ISSUE_TEMPLATE/bug_report.yml | 1 - .github/PULL_REQUEST_TEMPLATE.md | 8 +- .github/RELEASE_CHECKLIST.md | 6 +- .github/workflows/build_gitpod.yml | 8 +- .github/workflows/create-test-wf.yml | 8 +- .github/workflows/pytest.yml | 2 +- .github/workflows/tools-api-docs-dev.yml | 2 +- .github/workflows/tools-api-docs-release.yml | 2 +- .gitpod.yml | 20 +- CHANGELOG.md | 1219 +++++++++--------- README.md | 143 +- docs/api/_src/_static/css/custom.css | 621 ++++++--- docs/api/_src/_templates/layout.html | 8 +- 14 files changed, 1137 insertions(+), 913 deletions(-) diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 999cebd8a6..b3f88970df 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -11,7 +11,7 @@ If you'd like to write some code for nf-core/tools, the standard workflow is as follows: 1. Check that there isn't [already an issue](https://github.com/nf-core/tools/issues) about your idea to avoid duplicating work. - * If there isn't one already, please create one so that others know you're working on this + - If there isn't one already, please create one so that others know you're working on this 2. Fork the [nf-core/tools repository](https://github.com/nf-core/tools) to your GitHub account 3. Make the necessary changes / additions within your forked repository 4. Submit a Pull Request against the `dev` branch and wait for the code to be reviewed and merged. diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 431cce198a..72b71ddd68 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -2,7 +2,6 @@ name: Bug report description: Report something that is broken or incorrect labels: bug body: - - type: markdown attributes: value: | diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 8f7661bd76..71411be1b9 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -11,7 +11,7 @@ Learn more about contributing: https://github.com/nf-core/tools/tree/master/.git ## PR checklist - - [ ] This comment contains a description of changes (with reason) - - [ ] `CHANGELOG.md` is updated - - [ ] If you've fixed a bug or added code that should be tested, add tests! - - [ ] Documentation in `docs` is updated +- [ ] This comment contains a description of changes (with reason) +- [ ] `CHANGELOG.md` is updated +- [ ] If you've fixed a bug or added code that should be tested, add tests! +- [ ] Documentation in `docs` is updated diff --git a/.github/RELEASE_CHECKLIST.md b/.github/RELEASE_CHECKLIST.md index e5b670c561..e086d286aa 100644 --- a/.github/RELEASE_CHECKLIST.md +++ b/.github/RELEASE_CHECKLIST.md @@ -1,9 +1,9 @@ ## Before release 1. Check issue milestones to see outstanding issues to resolve if possible or transfer to the milestones for the next release e.g. [`v1.9`](https://github.com/nf-core/tools/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.9) -2. Most importantly, pick an undeniably outstanding [name](http://www.codenamegenerator.com/) for the release where *Prefix* = *Metal* and *Dictionary* = *Animal*. +2. Most importantly, pick an undeniably outstanding [name](http://www.codenamegenerator.com/) for the release where _Prefix_ = _Metal_ and _Dictionary_ = _Animal_. 3. Check whether the GitHub Actions workflow scripts need updating of the Nextflow versions -4. Create a PR to `dev` to bump the version in `CHANGELOG.md` and `setup.py`. +4. Create a PR to `dev` to bump the version in `CHANGELOG.md` and `setup.py`. 5. Make sure all CI tests are passing! 6. Create a PR from `dev` to `master` 7. Make sure all CI tests are passing again (additional tests are run on PRs to `master`) @@ -18,4 +18,4 @@ 1. Check the automated template synchronisation has been triggered properly. This should automatically open PRs directly to individual pipeline repos with the appropriate changes to update the pipeline template. 2. Check that the automatic `PyPi` deployment has worked: [pypi.org/project/nf-core](https://pypi.org/project/nf-core/) 3. Check `BioConda` has an automated PR to bump the version, and merge. eg. [bioconda/bioconda-recipes #20065](https://github.com/bioconda/bioconda-recipes/pull/20065) -4. Create a tools PR to `dev` to bump back to the next development version in `CHANGELOG.md` and `setup.py` +4. Create a tools PR to `dev` to bump back to the next development version in `CHANGELOG.md` and `setup.py` diff --git a/.github/workflows/build_gitpod.yml b/.github/workflows/build_gitpod.yml index 7cdc5aa4d8..a05b2cbc0c 100644 --- a/.github/workflows/build_gitpod.yml +++ b/.github/workflows/build_gitpod.yml @@ -4,11 +4,11 @@ name: nf-core gitpod build and push on: push: branches: - - 'master' - - 'dev' + - "master" + - "dev" paths: - - 'nf_core/gitpod/gitpod.Dockerfile' - - '.github/workflows/build_gitpod.yml' + - "nf_core/gitpod/gitpod.Dockerfile" + - ".github/workflows/build_gitpod.yml" jobs: push_to_registry: diff --git a/.github/workflows/create-test-wf.yml b/.github/workflows/create-test-wf.yml index 90013bc439..a16c8a6409 100644 --- a/.github/workflows/create-test-wf.yml +++ b/.github/workflows/create-test-wf.yml @@ -15,11 +15,11 @@ jobs: # Nextflow versions include: # Test pipeline minimum Nextflow version - - NXF_VER: '21.10.3' - NXF_EDGE: '' + - NXF_VER: "21.10.3" + NXF_EDGE: "" # Test latest edge release of Nextflow - - NXF_VER: '' - NXF_EDGE: '1' + - NXF_VER: "" + NXF_EDGE: "1" steps: - uses: actions/checkout@v2 name: Check out source-code repository diff --git a/.github/workflows/pytest.yml b/.github/workflows/pytest.yml index 4537d7e5cc..f634f29add 100644 --- a/.github/workflows/pytest.yml +++ b/.github/workflows/pytest.yml @@ -13,7 +13,7 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: ['3.6', '3.7', '3.8', '3.9', '3.10'] + python-version: ["3.6", "3.7", "3.8", "3.9", "3.10"] steps: - uses: actions/checkout@v2 diff --git a/.github/workflows/tools-api-docs-dev.yml b/.github/workflows/tools-api-docs-dev.yml index 34feec23f1..c31c94a721 100644 --- a/.github/workflows/tools-api-docs-dev.yml +++ b/.github/workflows/tools-api-docs-dev.yml @@ -33,7 +33,7 @@ jobs: server: ${{ secrets.ftp_server }} username: ${{ secrets.ftp_username}} password: ${{ secrets.ftp_password }} - local-dir: './docs/api/_build/html/' + local-dir: "./docs/api/_build/html/" server-dir: ${{ secrets.ftp_server_dir }}/dev/ protocol: ${{ secrets.ftp_protocol }} port: ${{ secrets.ftp_port }} diff --git a/.github/workflows/tools-api-docs-release.yml b/.github/workflows/tools-api-docs-release.yml index c420824722..54c138ed68 100644 --- a/.github/workflows/tools-api-docs-release.yml +++ b/.github/workflows/tools-api-docs-release.yml @@ -37,7 +37,7 @@ jobs: server: ${{ secrets.ftp_server }} username: ${{ secrets.ftp_username}} password: ${{ secrets.ftp_password }} - local-dir: './docs/api/_build/html/' + local-dir: "./docs/api/_build/html/" server-dir: ${{ secrets.ftp_server_dir }}/${{ matrix.dir }}/ protocol: ${{ secrets.ftp_protocol }} port: ${{ secrets.ftp_port }} diff --git a/.gitpod.yml b/.gitpod.yml index 1f30f26a16..c8d7fce6e3 100644 --- a/.gitpod.yml +++ b/.gitpod.yml @@ -3,15 +3,15 @@ tasks: - name: install current state of nf-core/tools command: python -m pip install --upgrade -r requirements-dev.txt -e . vscode: - extensions: # based on nf-core.nf-core-extensionpack - - codezombiech.gitignore # Language support for .gitignore files + extensions: # based on nf-core.nf-core-extensionpack + - codezombiech.gitignore # Language support for .gitignore files # - cssho.vscode-svgviewer # SVG viewer - - davidanson.vscode-markdownlint # Markdown/CommonMark linting and style checking for Visual Studio Code - - eamodio.gitlens # Quickly glimpse into whom, why, and when a line or code block was changed - - EditorConfig.EditorConfig # override user/workspace settings with settings found in .editorconfig files - - Gruntfuggly.todo-tree # Display TODO and FIXME in a tree view in the activity bar - - mechatroner.rainbow-csv # Highlight columns in csv files in different colors + - davidanson.vscode-markdownlint # Markdown/CommonMark linting and style checking for Visual Studio Code + - eamodio.gitlens # Quickly glimpse into whom, why, and when a line or code block was changed + - EditorConfig.EditorConfig # override user/workspace settings with settings found in .editorconfig files + - Gruntfuggly.todo-tree # Display TODO and FIXME in a tree view in the activity bar + - mechatroner.rainbow-csv # Highlight columns in csv files in different colors # - nextflow.nextflow # Nextflow syntax highlighting - - oderwat.indent-rainbow # Highlight indentation level - - streetsidesoftware.code-spell-checker # Spelling checker for source code - - DavidAnson.vscode-markdownlint # Linter for markdown files + - oderwat.indent-rainbow # Highlight indentation level + - streetsidesoftware.code-spell-checker # Spelling checker for source code + - DavidAnson.vscode-markdownlint # Linter for markdown files diff --git a/CHANGELOG.md b/CHANGELOG.md index 7bb82760bd..85c41a48d5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,9 +6,9 @@ ### General -* Convert nf-core/tools API / lint test documentation to MyST ([#1245](https://github.com/nf-core/tools/pull/1245)) -* Build documentation for the `nf-core modules lint` tests ([#1250](https://github.com/nf-core/tools/pull/1250)) -* Install tools inside GitPod Docker using the repo itself and not from Conda. +- Convert nf-core/tools API / lint test documentation to MyST ([#1245](https://github.com/nf-core/tools/pull/1245)) +- Build documentation for the `nf-core modules lint` tests ([#1250](https://github.com/nf-core/tools/pull/1250)) +- Install tools inside GitPod Docker using the repo itself and not from Conda. ### Modules @@ -16,152 +16,152 @@ ### Template -* Removed mention of `--singularity_pull_docker_container` in pipeline `README.md` -* Replaced equals with ~ in nf-core headers, to stop false positive unresolved conflict errors when committing with VSCode. -* Add retry strategy for AWS megatests after releasing [nf-core/tower-action v2.2](https://github.com/nf-core/tower-action/releases/tag/v2.2) -* Added `.nf-core.yml` file with `repository_type: pipeline` for modules commands -* Update igenomes path to the `BWAIndex` to fetch the whole `version0.6.0` folder instead of only the `genome.fa` file -* Remove pinned Node version in the GitHub Actions workflows, to fix errors with `markdownlint` -* Bumped `nf-core/tower-action` to `v3` and removed `pipeline` and `revision` from the AWS workflows, which were not needed -* Add yamllint GitHub Action. -* Add `.yamllint.yml` to avoid line length and document start errors ([#1407](https://github.com/nf-core/tools/issues/1407)) -* Add `--publish_dir_mode` back into the pipeline template ([nf-core/rnaseq#752](https://github.com/nf-core/rnaseq/issues/752#issuecomment-1039451607)) -* Add optional loading of of pipeline-specific institutional configs to `nextflow.config` -* Make `--outdir` a mandatory parameter ([nf-core/tools#1415](https://github.com/nf-core/tools/issues/1415)) +- Removed mention of `--singularity_pull_docker_container` in pipeline `README.md` +- Replaced equals with ~ in nf-core headers, to stop false positive unresolved conflict errors when committing with VSCode. +- Add retry strategy for AWS megatests after releasing [nf-core/tower-action v2.2](https://github.com/nf-core/tower-action/releases/tag/v2.2) +- Added `.nf-core.yml` file with `repository_type: pipeline` for modules commands +- Update igenomes path to the `BWAIndex` to fetch the whole `version0.6.0` folder instead of only the `genome.fa` file +- Remove pinned Node version in the GitHub Actions workflows, to fix errors with `markdownlint` +- Bumped `nf-core/tower-action` to `v3` and removed `pipeline` and `revision` from the AWS workflows, which were not needed +- Add yamllint GitHub Action. +- Add `.yamllint.yml` to avoid line length and document start errors ([#1407](https://github.com/nf-core/tools/issues/1407)) +- Add `--publish_dir_mode` back into the pipeline template ([nf-core/rnaseq#752](https://github.com/nf-core/rnaseq/issues/752#issuecomment-1039451607)) +- Add optional loading of of pipeline-specific institutional configs to `nextflow.config` +- Make `--outdir` a mandatory parameter ([nf-core/tools#1415](https://github.com/nf-core/tools/issues/1415)) ### General -* Updated `nf-core download` to work with latest DSL2 syntax for containers ([#1379](https://github.com/nf-core/tools/issues/1379)) -* Made `nf-core modules create` detect repository type with explicit `.nf-core.yml` instead of random readme stuff ([#1391](https://github.com/nf-core/tools/pull/1391)) -* Added a Gitpod environment and Dockerfile ([#1384](https://github.com/nf-core/tools/pull/1384)) - * Adds conda, Nextflow, nf-core, pytest-workflow, mamba, and pip to base Gitpod Docker image. - * Adds GH action to build and push Gitpod Docker image. - * Adds Gitpod environment to template. - * Adds Gitpod environment to tools with auto build of nf-core tool. -* Shiny new command-line help formatting ([#1403](https://github.com/nf-core/tools/pull/1403)) -* Call the command line help with `-h` as well as `--help` (was formerly just the latter) ([#1404](https://github.com/nf-core/tools/pull/1404)) -* Add `.yamllint.yml` config file to avoid line length and document start errors in the tools repo itself. -* Switch to `yamllint-github-action`to be able to configure yaml lint exceptions ([#1404](https://github.com/nf-core/tools/issues/1413)) -* Prevent module linting KeyError edge case ([#1321](https://github.com/nf-core/tools/issues/1321)) -* Bump-versions: Don't trim the trailing newline on files, causes editorconfig linting to fail ([#1265](https://github.com/nf-core/tools/issues/1265)) -* Handle exception in `nf-core list` when a broken git repo is found ([#1273](https://github.com/nf-core/tools/issues/1273)) -* Updated URL for pipeline lint test docs ([#1348](https://github.com/nf-core/tools/issues/1348)) -* Updated `nf-core create` to tolerate failures and retry when fetching pipeline logos from the website ([#1369](https://github.com/nf-core/tools/issues/1369)) -* Modified the CSS overriding `sphinx_rtd_theme` default colors to fix some glitches in the API documentation ([#1294](https://github.com/nf-core/tools/issues/1294)) +- Updated `nf-core download` to work with latest DSL2 syntax for containers ([#1379](https://github.com/nf-core/tools/issues/1379)) +- Made `nf-core modules create` detect repository type with explicit `.nf-core.yml` instead of random readme stuff ([#1391](https://github.com/nf-core/tools/pull/1391)) +- Added a Gitpod environment and Dockerfile ([#1384](https://github.com/nf-core/tools/pull/1384)) + - Adds conda, Nextflow, nf-core, pytest-workflow, mamba, and pip to base Gitpod Docker image. + - Adds GH action to build and push Gitpod Docker image. + - Adds Gitpod environment to template. + - Adds Gitpod environment to tools with auto build of nf-core tool. +- Shiny new command-line help formatting ([#1403](https://github.com/nf-core/tools/pull/1403)) +- Call the command line help with `-h` as well as `--help` (was formerly just the latter) ([#1404](https://github.com/nf-core/tools/pull/1404)) +- Add `.yamllint.yml` config file to avoid line length and document start errors in the tools repo itself. +- Switch to `yamllint-github-action`to be able to configure yaml lint exceptions ([#1404](https://github.com/nf-core/tools/issues/1413)) +- Prevent module linting KeyError edge case ([#1321](https://github.com/nf-core/tools/issues/1321)) +- Bump-versions: Don't trim the trailing newline on files, causes editorconfig linting to fail ([#1265](https://github.com/nf-core/tools/issues/1265)) +- Handle exception in `nf-core list` when a broken git repo is found ([#1273](https://github.com/nf-core/tools/issues/1273)) +- Updated URL for pipeline lint test docs ([#1348](https://github.com/nf-core/tools/issues/1348)) +- Updated `nf-core create` to tolerate failures and retry when fetching pipeline logos from the website ([#1369](https://github.com/nf-core/tools/issues/1369)) +- Modified the CSS overriding `sphinx_rtd_theme` default colors to fix some glitches in the API documentation ([#1294](https://github.com/nf-core/tools/issues/1294)) ### Modules -* New command `nf-core modules info` that prints nice documentation about a module to the terminal :sparkles: ([#1427](https://github.com/nf-core/tools/issues/1427)) -* Linting a pipeline now fails instead of warning if a local copy of a module does not match the remote ([#1313](https://github.com/nf-core/tools/issues/1313)) -* Fixed linting bugs where warning was incorrectly generated for: - * `Module does not emit software version` - * `Container versions do not match` - * `input:` / `output:` not being specified in module - * Allow for containers from other biocontainers resource as defined [here](https://github.com/nf-core/modules/blob/cde237e7cec07798e5754b72aeca44efe89fc6db/modules/cat/fastq/main.nf#L7-L8) -* Fixed traceback when using `stageAs` syntax as defined [here](https://github.com/nf-core/modules/blob/cde237e7cec07798e5754b72aeca44efe89fc6db/modules/cat/fastq/main.nf#L11) -* Added `nf-core schema docs` command to output pipline parameter documentation in Markdown format for inclusion in GitHub and other documentation systems ([#741](https://github.com/nf-core/tools/issues/741)) -* Allow conditional process execution from the configuration file ([#1393](https://github.com/nf-core/tools/pull/1393)) -* Add linting for when condition([#1397](https://github.com/nf-core/tools/pull/1397)) -* Added modules ignored table to `nf-core modules bump-versions`. ([#1234](https://github.com/nf-core/tools/issues/1234)) -* Added `--conda-package-version` flag for specifying version of conda package in `nf-core modules create`. ([#1238](https://github.com/nf-core/tools/issues/1238)) -* Add option of writing diffs to file in `nf-core modules update` using either interactive prompts or the new `--diff-file` flag. -* Fixed edge case where module names that were substrings of other modules caused both to be installed ([#1380](https://github.com/nf-core/tools/issues/1380)) -* Tweak handling of empty files when generating the test YAML ([#1376](https://github.com/nf-core/tools/issues/1376)) - * Fail linting if a md5sum for an empty file is found (instead of a warning) - * Don't skip the md5 when generating a test file if an empty file is found (so that linting fails and can be manually checked) -* Linting checks test files for `TODO` statements as well as the main module code ([#1271](https://github.com/nf-core/tools/issues/1271)) -* Handle error if `manifest` isn't set in `nextflow.config` ([#1418](https://github.com/nf-core/tools/issues/1418)) +- New command `nf-core modules info` that prints nice documentation about a module to the terminal :sparkles: ([#1427](https://github.com/nf-core/tools/issues/1427)) +- Linting a pipeline now fails instead of warning if a local copy of a module does not match the remote ([#1313](https://github.com/nf-core/tools/issues/1313)) +- Fixed linting bugs where warning was incorrectly generated for: + - `Module does not emit software version` + - `Container versions do not match` + - `input:` / `output:` not being specified in module + - Allow for containers from other biocontainers resource as defined [here](https://github.com/nf-core/modules/blob/cde237e7cec07798e5754b72aeca44efe89fc6db/modules/cat/fastq/main.nf#L7-L8) +- Fixed traceback when using `stageAs` syntax as defined [here](https://github.com/nf-core/modules/blob/cde237e7cec07798e5754b72aeca44efe89fc6db/modules/cat/fastq/main.nf#L11) +- Added `nf-core schema docs` command to output pipline parameter documentation in Markdown format for inclusion in GitHub and other documentation systems ([#741](https://github.com/nf-core/tools/issues/741)) +- Allow conditional process execution from the configuration file ([#1393](https://github.com/nf-core/tools/pull/1393)) +- Add linting for when condition([#1397](https://github.com/nf-core/tools/pull/1397)) +- Added modules ignored table to `nf-core modules bump-versions`. ([#1234](https://github.com/nf-core/tools/issues/1234)) +- Added `--conda-package-version` flag for specifying version of conda package in `nf-core modules create`. ([#1238](https://github.com/nf-core/tools/issues/1238)) +- Add option of writing diffs to file in `nf-core modules update` using either interactive prompts or the new `--diff-file` flag. +- Fixed edge case where module names that were substrings of other modules caused both to be installed ([#1380](https://github.com/nf-core/tools/issues/1380)) +- Tweak handling of empty files when generating the test YAML ([#1376](https://github.com/nf-core/tools/issues/1376)) + - Fail linting if a md5sum for an empty file is found (instead of a warning) + - Don't skip the md5 when generating a test file if an empty file is found (so that linting fails and can be manually checked) +- Linting checks test files for `TODO` statements as well as the main module code ([#1271](https://github.com/nf-core/tools/issues/1271)) +- Handle error if `manifest` isn't set in `nextflow.config` ([#1418](https://github.com/nf-core/tools/issues/1418)) ## [v2.2 - Lead Liger](https://github.com/nf-core/tools/releases/tag/2.2) - [2021-12-14] ### Template -* Update repo logos to utilize [GitHub's `#gh-light/dark-mode-only`](https://docs.github.com/en/github/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#specifying-the-theme-an-image-is-shown-to), to switch between logos optimized for light or dark themes. The old repo logos have to be removed (in `docs/images` and `assets/`). -* Deal with authentication with private repositories -* Bump minimun Nextflow version to 21.10.3 -* Convert pipeline template to updated Nextflow DSL2 syntax -* Solve circular import when importing `nf_core.modules.lint` -* Disable cache in `nf_core.utils.fetch_wf_config` while performing `test_wf_use_local_configs`. -* Modify software version channel handling to support multiple software version emissions (e.g. from mulled containers), and multiple software versions. -* Update `dumpsoftwareversion` module to correctly report versions with trailing zeros. -* Remove `params.hostnames` from the pipeline template ([#1304](https://github.com/nf-core/tools/issues/1304)) -* Update `.gitattributes` to mark installed modules and subworkflows as `linguist-generated` ([#1311](https://github.com/nf-core/tools/issues/1311)) -* Adding support for [Julia](https://julialang.org) package environments to `nextflow.config`([#1317](https://github.com/nf-core/tools/pull/1317)) -* New YAML issue templates for pipeline bug reports and feature requests, with a much richer interface ([#1165](https://github.com/nf-core/tools/pull/1165)) -* Update AWS test GitHub Actions to use v2 of [nf-core/tower-action](https://github.com/nf-core/tower-action) -* Post linting comment even when `linting.yml` fails -* Update `CONTRIBUTION.md` bullets to remove points related to `scrape_software_versions.py` -* Update AWS test to set Nextflow version to 21.10.3 +- Update repo logos to utilize [GitHub's `#gh-light/dark-mode-only`](https://docs.github.com/en/github/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#specifying-the-theme-an-image-is-shown-to), to switch between logos optimized for light or dark themes. The old repo logos have to be removed (in `docs/images` and `assets/`). +- Deal with authentication with private repositories +- Bump minimun Nextflow version to 21.10.3 +- Convert pipeline template to updated Nextflow DSL2 syntax +- Solve circular import when importing `nf_core.modules.lint` +- Disable cache in `nf_core.utils.fetch_wf_config` while performing `test_wf_use_local_configs`. +- Modify software version channel handling to support multiple software version emissions (e.g. from mulled containers), and multiple software versions. +- Update `dumpsoftwareversion` module to correctly report versions with trailing zeros. +- Remove `params.hostnames` from the pipeline template ([#1304](https://github.com/nf-core/tools/issues/1304)) +- Update `.gitattributes` to mark installed modules and subworkflows as `linguist-generated` ([#1311](https://github.com/nf-core/tools/issues/1311)) +- Adding support for [Julia](https://julialang.org) package environments to `nextflow.config`([#1317](https://github.com/nf-core/tools/pull/1317)) +- New YAML issue templates for pipeline bug reports and feature requests, with a much richer interface ([#1165](https://github.com/nf-core/tools/pull/1165)) +- Update AWS test GitHub Actions to use v2 of [nf-core/tower-action](https://github.com/nf-core/tower-action) +- Post linting comment even when `linting.yml` fails +- Update `CONTRIBUTION.md` bullets to remove points related to `scrape_software_versions.py` +- Update AWS test to set Nextflow version to 21.10.3 ### General -* Made lint check for parameters defaults stricter [[#992](https://github.com/nf-core/tools/issues/992)] - * Default values in `nextflow.config` must match the defaults given in the schema (anything with `{` in, or in `main.nf` is ignored) - * Defaults in `nextflow.config` must now match the variable _type_ specified in the schema - * If you want the parameter to not have a default value, use `null` - * Strings set to `false` or an empty string in `nextflow.config` will now fail linting -* Bump minimun Nextflow version to 21.10.3 -* Changed `questionary` `ask()` to `unsafe_ask()` to not catch `KeyboardInterupts` ([#1237](https://github.com/nf-core/tools/issues/1237)) -* Fixed bug in `nf-core launch` due to revisions specified with `-r` not being added to nextflow command. ([#1246](https://github.com/nf-core/tools/issues/1246)) -* Update regex in `readme` test of `nf-core lint` to agree with the pipeline template ([#1260](https://github.com/nf-core/tools/issues/1260)) -* Update 'fix' message in `nf-core lint` to conform to the current command line options. ([#1259](https://github.com/nf-core/tools/issues/1259)) -* Fixed bug in `nf-core list` when `NXF_HOME` is set -* Run CI test used to create and lint/run the pipeline template with minimum and latest edge release of NF ([#1304](https://github.com/nf-core/tools/issues/1304)) -* New YAML issue templates for tools bug reports and feature requests, with a much richer interface ([#1165](https://github.com/nf-core/tools/pull/1165)) -* Handle synax errors in Nextflow config nicely when running `nf-core schema build` ([#1267](https://github.com/nf-core/tools/pull/1267)) -* Erase temporary files and folders while performing Python tests (pytest) -* Remove base `Dockerfile` used for DSL1 pipeline container builds -* Run tests with Python 3.10 -* [#1363](https://github.com/nf-core/tools/pull/1363) Fix tools CI workflow nextflow versions. +- Made lint check for parameters defaults stricter [[#992](https://github.com/nf-core/tools/issues/992)] + - Default values in `nextflow.config` must match the defaults given in the schema (anything with `{` in, or in `main.nf` is ignored) + - Defaults in `nextflow.config` must now match the variable _type_ specified in the schema + - If you want the parameter to not have a default value, use `null` + - Strings set to `false` or an empty string in `nextflow.config` will now fail linting +- Bump minimun Nextflow version to 21.10.3 +- Changed `questionary` `ask()` to `unsafe_ask()` to not catch `KeyboardInterupts` ([#1237](https://github.com/nf-core/tools/issues/1237)) +- Fixed bug in `nf-core launch` due to revisions specified with `-r` not being added to nextflow command. ([#1246](https://github.com/nf-core/tools/issues/1246)) +- Update regex in `readme` test of `nf-core lint` to agree with the pipeline template ([#1260](https://github.com/nf-core/tools/issues/1260)) +- Update 'fix' message in `nf-core lint` to conform to the current command line options. ([#1259](https://github.com/nf-core/tools/issues/1259)) +- Fixed bug in `nf-core list` when `NXF_HOME` is set +- Run CI test used to create and lint/run the pipeline template with minimum and latest edge release of NF ([#1304](https://github.com/nf-core/tools/issues/1304)) +- New YAML issue templates for tools bug reports and feature requests, with a much richer interface ([#1165](https://github.com/nf-core/tools/pull/1165)) +- Handle synax errors in Nextflow config nicely when running `nf-core schema build` ([#1267](https://github.com/nf-core/tools/pull/1267)) +- Erase temporary files and folders while performing Python tests (pytest) +- Remove base `Dockerfile` used for DSL1 pipeline container builds +- Run tests with Python 3.10 +- [#1363](https://github.com/nf-core/tools/pull/1363) Fix tools CI workflow nextflow versions. ### Modules -* Fixed typo in `module_utils.py`. -* Fixed failing lint test when process section was missing from module. Also added the local failing tests to the warned section of the output table. ([#1235](https://github.com/nf-core/tools/issues/1235)) -* Added `--diff` flag to `nf-core modules update` which shows the diff between the installed files and the versions -* Update `nf-core modules create` help texts which were not changed with the introduction of the `--dir` flag -* Check if README is from modules repo -* Update module template to DSL2 v2.0 (remove `functions.nf` from modules template and updating `main.nf` ([#1289](https://github.com/nf-core/tools/pull/)) -* Substitute get process/module name custom functions in module `main.nf` using template replacement ([#1284](https://github.com/nf-core/tools/issues/1284)) -* Check test YML file for md5sums corresponding to empty files ([#1302](https://github.com/nf-core/tools/issues/1302)) -* Exit with an error if empty files are found when generating the test YAML file ([#1302](https://github.com/nf-core/tools/issues/1302)) +- Fixed typo in `module_utils.py`. +- Fixed failing lint test when process section was missing from module. Also added the local failing tests to the warned section of the output table. ([#1235](https://github.com/nf-core/tools/issues/1235)) +- Added `--diff` flag to `nf-core modules update` which shows the diff between the installed files and the versions +- Update `nf-core modules create` help texts which were not changed with the introduction of the `--dir` flag +- Check if README is from modules repo +- Update module template to DSL2 v2.0 (remove `functions.nf` from modules template and updating `main.nf` ([#1289](https://github.com/nf-core/tools/pull/)) +- Substitute get process/module name custom functions in module `main.nf` using template replacement ([#1284](https://github.com/nf-core/tools/issues/1284)) +- Check test YML file for md5sums corresponding to empty files ([#1302](https://github.com/nf-core/tools/issues/1302)) +- Exit with an error if empty files are found when generating the test YAML file ([#1302](https://github.com/nf-core/tools/issues/1302)) ## [v2.1 - Zinc Zebra](https://github.com/nf-core/tools/releases/tag/2.1) - [2021-07-27] ### Template -* Correct regex pattern for file names in `nextflow_schema.json` -* Remove `.` from nf-core/tools command examples -* Update Nextflow installation link in pipeline template ([#1201](https://github.com/nf-core/tools/issues/1201)) -* Command `hostname` is not portable [[#1212](https://github.com/nf-core/tools/pull/1212)] -* Changed how singularity and docker links are written in template to avoid duplicate links +- Correct regex pattern for file names in `nextflow_schema.json` +- Remove `.` from nf-core/tools command examples +- Update Nextflow installation link in pipeline template ([#1201](https://github.com/nf-core/tools/issues/1201)) +- Command `hostname` is not portable [[#1212](https://github.com/nf-core/tools/pull/1212)] +- Changed how singularity and docker links are written in template to avoid duplicate links ### General -* Changed names of some flags with `-r` as short options to make the flags more consistent between commands. +- Changed names of some flags with `-r` as short options to make the flags more consistent between commands. ### Modules -* Added consistency checks between installed modules and `modules.json` ([#1199](https://github.com/nf-core/tools/issues/1199)) -* Added support excluding or specifying version of modules in `.nf-core.yml` when updating with `nf-core modules install --all` ([#1204](https://github.com/nf-core/tools/issues/1204)) -* Created `nf-core modules update` and removed updating options from `nf-core modules install` -* Added missing function call to `nf-core lint` ([#1198](https://github.com/nf-core/tools/issues/1198)) -* Fix `nf-core lint` not filtering modules test when run with `--key` ([#1203](https://github.com/nf-core/tools/issues/1203)) -* Fixed `nf-core modules install` not working when installing from branch with `-b` ([#1218](https://github.com/nf-core/tools/issues/1218)) -* Added prompt to choose between updating all modules or named module in `nf-core modules update` -* Check if modules is installed before trying to update in `nf-core modules update` -* Verify that a commit SHA provided with `--sha` exists for `install/update` commands -* Add new-line to `main.nf` after `bump-versions` command to make ECLint happy +- Added consistency checks between installed modules and `modules.json` ([#1199](https://github.com/nf-core/tools/issues/1199)) +- Added support excluding or specifying version of modules in `.nf-core.yml` when updating with `nf-core modules install --all` ([#1204](https://github.com/nf-core/tools/issues/1204)) +- Created `nf-core modules update` and removed updating options from `nf-core modules install` +- Added missing function call to `nf-core lint` ([#1198](https://github.com/nf-core/tools/issues/1198)) +- Fix `nf-core lint` not filtering modules test when run with `--key` ([#1203](https://github.com/nf-core/tools/issues/1203)) +- Fixed `nf-core modules install` not working when installing from branch with `-b` ([#1218](https://github.com/nf-core/tools/issues/1218)) +- Added prompt to choose between updating all modules or named module in `nf-core modules update` +- Check if modules is installed before trying to update in `nf-core modules update` +- Verify that a commit SHA provided with `--sha` exists for `install/update` commands +- Add new-line to `main.nf` after `bump-versions` command to make ECLint happy ## [v2.0.1 - Palladium Platypus Junior](https://github.com/nf-core/tools/releases/tag/2.0.1) - [2021-07-13] ### Template -* Critical tweak to add `--dir` declaration to `nf-core lint` GitHub Actions `linting.yml` workflow +- Critical tweak to add `--dir` declaration to `nf-core lint` GitHub Actions `linting.yml` workflow ### General -* Add `--dir` declaration to `nf-core sync` GitHub Actions `sync.yml` workflow +- Add `--dir` declaration to `nf-core sync` GitHub Actions `sync.yml` workflow ## [v2.0 - Palladium Platypus](https://github.com/nf-core/tools/releases/tag/2.0) - [2021-07-13] @@ -169,301 +169,301 @@ This marks the first Nextflow DSL2-centric release of `tools` which means that some commands won't work in full with DSL1 pipelines anymore. Please use a `v1.x` version of `tools` for such pipelines or better yet join us to improve our DSL2 efforts! Here are the most important changes: -* The pipeline template has been completely re-written in DSL2 -* A module template has been added to auto-create best-practice DSL2 modules to speed up development -* A whole suite of commands have been added to streamline the creation, installation, removal, linting and version bumping of DSL2 modules either installed within pipelines or the nf-core/modules repo +- The pipeline template has been completely re-written in DSL2 +- A module template has been added to auto-create best-practice DSL2 modules to speed up development +- A whole suite of commands have been added to streamline the creation, installation, removal, linting and version bumping of DSL2 modules either installed within pipelines or the nf-core/modules repo ### Template -* Move TODO item of `contains:` map in a YAML string [[#1082](https://github.com/nf-core/tools/issues/1082)] -* Trigger AWS tests via Tower API [[#1160](https://github.com/nf-core/tools/pull/1160)] +- Move TODO item of `contains:` map in a YAML string [[#1082](https://github.com/nf-core/tools/issues/1082)] +- Trigger AWS tests via Tower API [[#1160](https://github.com/nf-core/tools/pull/1160)] ### General -* Fixed a bug in the Docker image build for tools that failed due to an extra hyphen. [[#1069](https://github.com/nf-core/tools/pull/1069)] -* Regular release sync fix - this time it was to do with JSON serialisation [[#1072](https://github.com/nf-core/tools/pull/1072)] -* Fixed bug in schema validation that ignores upper/lower-case typos in parameters [[#1087](https://github.com/nf-core/tools/issues/1087)] -* Bugfix: Download should use path relative to workflow for configs -* Remove lint checks for files related to conda and docker as not needed anymore for DSL2 -* Removed `params_used` lint check because of incompatibility with DSL2 -* Added`modules bump-versions` command to `README.md` -* Update docs for v2.0 release +- Fixed a bug in the Docker image build for tools that failed due to an extra hyphen. [[#1069](https://github.com/nf-core/tools/pull/1069)] +- Regular release sync fix - this time it was to do with JSON serialisation [[#1072](https://github.com/nf-core/tools/pull/1072)] +- Fixed bug in schema validation that ignores upper/lower-case typos in parameters [[#1087](https://github.com/nf-core/tools/issues/1087)] +- Bugfix: Download should use path relative to workflow for configs +- Remove lint checks for files related to conda and docker as not needed anymore for DSL2 +- Removed `params_used` lint check because of incompatibility with DSL2 +- Added`modules bump-versions` command to `README.md` +- Update docs for v2.0 release ### Modules -* Update comment style of modules `functions.nf` template file [[#1076](https://github.com/nf-core/tools/issues/1076)] -* Changed working directory to temporary directory for `nf-core modules create-test-yml` [[#908](https://github.com/nf-core/tools/issues/908)] -* Use Biocontainers API instead of quayi.io API for `nf-core modules create` [[#875](https://github.com/nf-core/tools/issues/875)] -* Update `nf-core modules install` to handle different versions of modules [#1116](https://github.com/nf-core/tools/pull/1116) -* Added `nf-core modules bump-versions` command to update all versions in the `nf-core/modules` repository [[#1123](https://github.com/nf-core/tools/issues/1123)] -* Updated `nf-core modules lint` to check whether a `git_sha` exists in the `modules.json` file or whether a new version is available [[#1114](https://github.com/nf-core/tools/issues/1114)] -* Refactored `nf-core modules` command into one file per command [#1124](https://github.com/nf-core/tools/pull/1124) -* Updated `nf-core modules remove` to also remove entry in `modules.json` file ([#1115](https://github.com/nf-core/tools/issues/1115)) -* Bugfix: Interactive prompt for `nf-core modules install` was receiving too few arguments -* Added progress bar to creation of 'modules.json' -* Updated `nf-core modules list` to show versions of local modules -* Improved exit behavior by replacing `sys.exit` with exceptions -* Updated `nf-core modules remove` to remove module entry in `modules.json` if module directory is missing -* Create extra tempdir as work directory for `nf-core modules create-test-yml` to avoid adding the temporary files to the `test.yml` -* Refactored passing of command line arguments to `nf-core` commands and subcommands ([#1139](https://github.com/nf-core/tools/issues/1139), [#1140](https://github.com/nf-core/tools/issues/1140)) -* Check for `modules.json` for entries of modules that are not actually installed in the pipeline [[#1141](https://github.com/nf-core/tools/issues/1141)] -* Added `` argument to `nf-core modules list` for filtering the listed modules. ([#1139](https://github.com/nf-core/tools/issues/1139) -* Added support for a `bump-versions` configuration file [[#1142](https://github.com/nf-core/tools/issues/1142)] -* Fixed `nf-core modules create-test-yml` so it doesn't break when the output directory is supplied [[#1148](https://github.com/nf-core/tools/issues/1148)] -* Updated `nf-core modules lint` to work with new directory structure [[#1159](https://github.com/nf-core/tools/issues/1159)] -* Updated `nf-core modules install` and `modules.json` to work with new directory structure ([#1159](https://github.com/nf-core/tools/issues/1159)) -* Updated `nf-core modules remove` to work with new directory structure [[#1159](https://github.com/nf-core/tools/issues/1159)] -* Restructured code and removed old table style in `nf-core modules list` -* Fixed bug causing `modules.json` creation to loop indefinitly -* Added `--all` flag to `nf-core modules install` -* Added `remote` and `local` subcommands to `nf-core modules list` -* Fix bug due to restructuring in modules template -* Added checks for verifying that the remote repository is well formed -* Added checks to `ModulesCommand` for verifying validity of remote repositories -* Misc. changes to `modules install`: check that module exist in remote, `--all` is has `--latest` by default. +- Update comment style of modules `functions.nf` template file [[#1076](https://github.com/nf-core/tools/issues/1076)] +- Changed working directory to temporary directory for `nf-core modules create-test-yml` [[#908](https://github.com/nf-core/tools/issues/908)] +- Use Biocontainers API instead of quayi.io API for `nf-core modules create` [[#875](https://github.com/nf-core/tools/issues/875)] +- Update `nf-core modules install` to handle different versions of modules [#1116](https://github.com/nf-core/tools/pull/1116) +- Added `nf-core modules bump-versions` command to update all versions in the `nf-core/modules` repository [[#1123](https://github.com/nf-core/tools/issues/1123)] +- Updated `nf-core modules lint` to check whether a `git_sha` exists in the `modules.json` file or whether a new version is available [[#1114](https://github.com/nf-core/tools/issues/1114)] +- Refactored `nf-core modules` command into one file per command [#1124](https://github.com/nf-core/tools/pull/1124) +- Updated `nf-core modules remove` to also remove entry in `modules.json` file ([#1115](https://github.com/nf-core/tools/issues/1115)) +- Bugfix: Interactive prompt for `nf-core modules install` was receiving too few arguments +- Added progress bar to creation of 'modules.json' +- Updated `nf-core modules list` to show versions of local modules +- Improved exit behavior by replacing `sys.exit` with exceptions +- Updated `nf-core modules remove` to remove module entry in `modules.json` if module directory is missing +- Create extra tempdir as work directory for `nf-core modules create-test-yml` to avoid adding the temporary files to the `test.yml` +- Refactored passing of command line arguments to `nf-core` commands and subcommands ([#1139](https://github.com/nf-core/tools/issues/1139), [#1140](https://github.com/nf-core/tools/issues/1140)) +- Check for `modules.json` for entries of modules that are not actually installed in the pipeline [[#1141](https://github.com/nf-core/tools/issues/1141)] +- Added `` argument to `nf-core modules list` for filtering the listed modules. ([#1139](https://github.com/nf-core/tools/issues/1139) +- Added support for a `bump-versions` configuration file [[#1142](https://github.com/nf-core/tools/issues/1142)] +- Fixed `nf-core modules create-test-yml` so it doesn't break when the output directory is supplied [[#1148](https://github.com/nf-core/tools/issues/1148)] +- Updated `nf-core modules lint` to work with new directory structure [[#1159](https://github.com/nf-core/tools/issues/1159)] +- Updated `nf-core modules install` and `modules.json` to work with new directory structure ([#1159](https://github.com/nf-core/tools/issues/1159)) +- Updated `nf-core modules remove` to work with new directory structure [[#1159](https://github.com/nf-core/tools/issues/1159)] +- Restructured code and removed old table style in `nf-core modules list` +- Fixed bug causing `modules.json` creation to loop indefinitly +- Added `--all` flag to `nf-core modules install` +- Added `remote` and `local` subcommands to `nf-core modules list` +- Fix bug due to restructuring in modules template +- Added checks for verifying that the remote repository is well formed +- Added checks to `ModulesCommand` for verifying validity of remote repositories +- Misc. changes to `modules install`: check that module exist in remote, `--all` is has `--latest` by default. #### Sync -* Don't set the default value to `"null"` when a parameter is initialised as `null` in the config [[#1074](https://github.com/nf-core/tools/pull/1074)] +- Don't set the default value to `"null"` when a parameter is initialised as `null` in the config [[#1074](https://github.com/nf-core/tools/pull/1074)] #### Tests -* Added a test for the `version_consistency` lint check -* Refactored modules tests into separate files, and removed direct comparisons with number of tests in `lint` tests ([#1158](https://github.com/nf-core/tools/issues/1158)) +- Added a test for the `version_consistency` lint check +- Refactored modules tests into separate files, and removed direct comparisons with number of tests in `lint` tests ([#1158](https://github.com/nf-core/tools/issues/1158)) ## [v1.14 - Brass Chicken :chicken:](https://github.com/nf-core/tools/releases/tag/1.14) - [2021-05-11] ### Template -* Add the implicit workflow declaration to `main.nf` DSL2 template [[#1056](https://github.com/nf-core/tools/issues/1056)] -* Fixed an issue regarding explicit disabling of unused container engines [[#972](https://github.com/nf-core/tools/pull/972)] -* Removed trailing slash from `params.igenomes_base` to yield valid s3 paths (previous paths work with Nextflow but not aws cli) -* Added a timestamp to the trace + timetime + report + dag filenames to fix overwrite issue on AWS -* Rewrite the `params_summary_log()` function to properly ignore unset params and have nicer formatting [[#971](https://github.com/nf-core/tools/issues/971)] -* Fix overly strict `--max_time` formatting regex in template schema [[#973](https://github.com/nf-core/tools/issues/973)] -* Convert `d` to `day` in the `cleanParameters` function to make Duration objects like `2d` pass the validation [[#858](https://github.com/nf-core/tools/issues/858)] -* Added nextflow version to quick start section and adjusted `nf-core bump-version` [[#1032](https://github.com/nf-core/tools/issues/1032)] -* Use latest stable Nextflow version `21.04.0` for CI tests instead of the `-edge` release +- Add the implicit workflow declaration to `main.nf` DSL2 template [[#1056](https://github.com/nf-core/tools/issues/1056)] +- Fixed an issue regarding explicit disabling of unused container engines [[#972](https://github.com/nf-core/tools/pull/972)] +- Removed trailing slash from `params.igenomes_base` to yield valid s3 paths (previous paths work with Nextflow but not aws cli) +- Added a timestamp to the trace + timetime + report + dag filenames to fix overwrite issue on AWS +- Rewrite the `params_summary_log()` function to properly ignore unset params and have nicer formatting [[#971](https://github.com/nf-core/tools/issues/971)] +- Fix overly strict `--max_time` formatting regex in template schema [[#973](https://github.com/nf-core/tools/issues/973)] +- Convert `d` to `day` in the `cleanParameters` function to make Duration objects like `2d` pass the validation [[#858](https://github.com/nf-core/tools/issues/858)] +- Added nextflow version to quick start section and adjusted `nf-core bump-version` [[#1032](https://github.com/nf-core/tools/issues/1032)] +- Use latest stable Nextflow version `21.04.0` for CI tests instead of the `-edge` release ### Download -* Fix bug in `nf-core download` where image names were getting a hyphen in `nf-core` which was breaking things. -* Extensive new interactive prompts for all command line flags [[#1027](https://github.com/nf-core/tools/issues/1027)] - * It is now recommended to run `nf-core download` without any cli options and follow prompts (though flags can be used to run non-interactively if you wish) -* New helper code to set `$NXF_SINGULARITY_CACHEDIR` and add to `.bashrc` if desired [[#1027](https://github.com/nf-core/tools/issues/1027)] +- Fix bug in `nf-core download` where image names were getting a hyphen in `nf-core` which was breaking things. +- Extensive new interactive prompts for all command line flags [[#1027](https://github.com/nf-core/tools/issues/1027)] + - It is now recommended to run `nf-core download` without any cli options and follow prompts (though flags can be used to run non-interactively if you wish) +- New helper code to set `$NXF_SINGULARITY_CACHEDIR` and add to `.bashrc` if desired [[#1027](https://github.com/nf-core/tools/issues/1027)] ### Launch -* Strip values from `nf-core launch` web response which are `False` and have no default in the schema [[#976](https://github.com/nf-core/tools/issues/976)] -* Improve API caching code when polling the website, fixes noisy log message when waiting for a response [[#1029](https://github.com/nf-core/tools/issues/1029)] -* New interactive prompts for pipeline name [[#1027](https://github.com/nf-core/tools/issues/1027)] +- Strip values from `nf-core launch` web response which are `False` and have no default in the schema [[#976](https://github.com/nf-core/tools/issues/976)] +- Improve API caching code when polling the website, fixes noisy log message when waiting for a response [[#1029](https://github.com/nf-core/tools/issues/1029)] +- New interactive prompts for pipeline name [[#1027](https://github.com/nf-core/tools/issues/1027)] ### Modules -* Added `tool_name_underscore` to the module template to allow TOOL_SUBTOOL in `main.nf` [[#1011](https://github.com/nf-core/tools/issues/1011)] -* Added `--conda-name` flag to `nf-core modules create` command to allow sidestepping questionary [[#988](https://github.com/nf-core/tools/issues/988)] -* Extended `nf-core modules lint` functionality to check tags in `test.yml` and to look for a entry in the `pytest_software.yml` file -* Update `modules` commands to use new test tag format `tool/subtool` -* New modules lint test comparing the `functions.nf` file to the template version -* Modules installed from alternative sources are put in folders based on the name of the source repository +- Added `tool_name_underscore` to the module template to allow TOOL_SUBTOOL in `main.nf` [[#1011](https://github.com/nf-core/tools/issues/1011)] +- Added `--conda-name` flag to `nf-core modules create` command to allow sidestepping questionary [[#988](https://github.com/nf-core/tools/issues/988)] +- Extended `nf-core modules lint` functionality to check tags in `test.yml` and to look for a entry in the `pytest_software.yml` file +- Update `modules` commands to use new test tag format `tool/subtool` +- New modules lint test comparing the `functions.nf` file to the template version +- Modules installed from alternative sources are put in folders based on the name of the source repository ### Linting -* Fix bug in nf-core lint config skipping for the `nextflow_config` test [[#1019](https://github.com/nf-core/tools/issues/1019)] -* New `-k`/`--key` cli option for `nf-core lint` to allow you to run only named lint tests, for faster local debugging -* Merge markers lint test - ignore binary files, allow config to ignore specific files [[#1040](https://github.com/nf-core/tools/pull/1040)] -* New lint test to check if all defined pipeline parameters are mentioned in `main.nf` [[#1038](https://github.com/nf-core/tools/issues/1038)] -* Added fix to remove warnings about params that get converted from camelCase to camel-case [[#1035](https://github.com/nf-core/tools/issues/1035)] -* Added pipeline schema lint checks for missing parameter description and parameters outside of groups [[#1017](https://github.com/nf-core/tools/issues/1017)] +- Fix bug in nf-core lint config skipping for the `nextflow_config` test [[#1019](https://github.com/nf-core/tools/issues/1019)] +- New `-k`/`--key` cli option for `nf-core lint` to allow you to run only named lint tests, for faster local debugging +- Merge markers lint test - ignore binary files, allow config to ignore specific files [[#1040](https://github.com/nf-core/tools/pull/1040)] +- New lint test to check if all defined pipeline parameters are mentioned in `main.nf` [[#1038](https://github.com/nf-core/tools/issues/1038)] +- Added fix to remove warnings about params that get converted from camelCase to camel-case [[#1035](https://github.com/nf-core/tools/issues/1035)] +- Added pipeline schema lint checks for missing parameter description and parameters outside of groups [[#1017](https://github.com/nf-core/tools/issues/1017)] ### General -* Try to fix the fix for the automated sync when we submit too many PRs at once [[#970](https://github.com/nf-core/tools/issues/970)] -* Rewrite how the tools documentation is deployed to the website, to allow multiple versions -* Created new Docker image for the tools cli package - see installation docs for details [[#917](https://github.com/nf-core/tools/issues/917)] -* Ignore permission errors for setting up requests cache directories to allow starting with an invalid or read-only `HOME` directory +- Try to fix the fix for the automated sync when we submit too many PRs at once [[#970](https://github.com/nf-core/tools/issues/970)] +- Rewrite how the tools documentation is deployed to the website, to allow multiple versions +- Created new Docker image for the tools cli package - see installation docs for details [[#917](https://github.com/nf-core/tools/issues/917)] +- Ignore permission errors for setting up requests cache directories to allow starting with an invalid or read-only `HOME` directory ## [v1.13.3 - Copper Crocodile Resurrection :crocodile:](https://github.com/nf-core/tools/releases/tag/1.13.2) - [2021-03-24] -* Running tests twice with `nf-core modules create-test-yml` to catch unreproducible md5 sums [[#890](https://github.com/nf-core/tools/issues/890)] -* Fix sync error again where the Nextflow edge release needs to be used for some pipelines -* Fix bug with `nf-core lint --release` (`NameError: name 'os' is not defined`) -* Added linebreak to linting comment so that markdown header renders on PR comment properly -* `nf-core modules create` command - if no bioconda package is found, prompt user for a different bioconda package name -* Updated module template `main.nf` with new test data paths +- Running tests twice with `nf-core modules create-test-yml` to catch unreproducible md5 sums [[#890](https://github.com/nf-core/tools/issues/890)] +- Fix sync error again where the Nextflow edge release needs to be used for some pipelines +- Fix bug with `nf-core lint --release` (`NameError: name 'os' is not defined`) +- Added linebreak to linting comment so that markdown header renders on PR comment properly +- `nf-core modules create` command - if no bioconda package is found, prompt user for a different bioconda package name +- Updated module template `main.nf` with new test data paths ## [v1.13.2 - Copper Crocodile CPR :crocodile: :face_with_head_bandage:](https://github.com/nf-core/tools/releases/tag/1.13.2) - [2021-03-23] -* Make module template pass the EC linter [[#953](https://github.com/nf-core/tools/pull/953)] -* Added better logging message if a user doesn't specificy the directory correctly with `nf-core modules` commands [[#942](https://github.com/nf-core/tools/pull/942)] -* Fixed parameter validation bug caused by JSONObject [[#937](https://github.com/nf-core/tools/issues/937)] -* Fixed template creation error regarding file permissions [[#932](https://github.com/nf-core/tools/issues/932)] -* Split the `create-lint-wf` tests up into separate steps in GitHub Actions to make the CI results easier to read -* Added automated PR comments to the Markdown, YAML and Python lint CI tests to explain failures (tools and pipeline template) -* Make `nf-core lint` summary table borders coloured according to overall pass / fail status -* Attempted a fix for the automated sync when we submit too many PRs at once [[#911](https://github.com/nf-core/tools/issues/911)] +- Make module template pass the EC linter [[#953](https://github.com/nf-core/tools/pull/953)] +- Added better logging message if a user doesn't specificy the directory correctly with `nf-core modules` commands [[#942](https://github.com/nf-core/tools/pull/942)] +- Fixed parameter validation bug caused by JSONObject [[#937](https://github.com/nf-core/tools/issues/937)] +- Fixed template creation error regarding file permissions [[#932](https://github.com/nf-core/tools/issues/932)] +- Split the `create-lint-wf` tests up into separate steps in GitHub Actions to make the CI results easier to read +- Added automated PR comments to the Markdown, YAML and Python lint CI tests to explain failures (tools and pipeline template) +- Make `nf-core lint` summary table borders coloured according to overall pass / fail status +- Attempted a fix for the automated sync when we submit too many PRs at once [[#911](https://github.com/nf-core/tools/issues/911)] ## [v1.13.1 - Copper Crocodile Patch :crocodile: :pirate_flag:](https://github.com/nf-core/tools/releases/tag/1.13.1) - [2021-03-19] -* Fixed bug in pipeline linting markdown output that gets posted to PR comments [[#914]](https://github.com/nf-core/tools/issues/914) -* Made text for the PR branch CI check less verbose with a TLDR in bold at the top -* A number of minor tweaks to the new `nf-core modules lint` code +- Fixed bug in pipeline linting markdown output that gets posted to PR comments [[#914]](https://github.com/nf-core/tools/issues/914) +- Made text for the PR branch CI check less verbose with a TLDR in bold at the top +- A number of minor tweaks to the new `nf-core modules lint` code ## [v1.13 - Copper Crocodile](https://github.com/nf-core/tools/releases/tag/1.13) - [2021-03-18] ### Template -* **Major new feature** - Validation of pipeline parameters [[#426]](https://github.com/nf-core/tools/issues/426) - * The addition runs as soon as the pipeline launches and checks the pipeline input parameters two main things: - * No parameters are supplied that share a name with core Nextflow options (eg. `--resume` instead of `-resume`) - * Supplied parameters validate against the pipeline JSON schema (eg. correct variable types, required values) - * If either parameter validation fails or the pipeline has errors, a warning is given about any unexpected parameters found which are not described in the pipeline schema. - * This behaviour can be disabled by using `--validate_params false` -* Added profiles to support the [Charliecloud](https://hpc.github.io/charliecloud/) and [Shifter](https://nersc.gitlab.io/development/shifter/how-to-use/) container engines [[#824](https://github.com/nf-core/tools/issues/824)] - * Note that Charliecloud requires Nextflow version `v21.03.0-edge` or later. -* Profiles for container engines now explicitly _disable_ all other engines [[#867](https://github.com/nf-core/tools/issues/867)] -* Fixed typo in nf-core-lint CI that prevented the markdown summary from being automatically posted on PRs as a comment. -* Changed default for `--input` from `data/*{1,2}.fastq.gz` to `null`, as this is now validated by the schema as a required value. -* Removed support for `--name` parameter for custom run names. - * The same functionality for MultiQC still exists with the core Nextflow `-name` option. -* Added to template docs about how to identify process name for resource customisation -* The parameters `--max_memory` and `--max_time` are now validated against a regular expression [[#793](https://github.com/nf-core/tools/issues/793)] - * Must be written in the format `123.GB` / `456.h` with any of the prefixes listed in the [Nextflow docs](https://www.nextflow.io/docs/latest/process.html#memory) - * Bare numbers no longer allowed, avoiding people from trying to specify GB and actually specifying bytes. -* Switched from cookiecutter to Jinja2 [[#880]](https://github.com/nf-core/tools/pull/880) -* Finally dropped the wonderful [cookiecutter](https://github.com/cookiecutter/cookiecutter) library that was behind the first pipeline template that led to nf-core [[#880](https://github.com/nf-core/tools/pull/880)] - * Now rendering templates directly using [Jinja](https://jinja.palletsprojects.com/), which is what cookiecutter was doing anyway +- **Major new feature** - Validation of pipeline parameters [[#426]](https://github.com/nf-core/tools/issues/426) + - The addition runs as soon as the pipeline launches and checks the pipeline input parameters two main things: + - No parameters are supplied that share a name with core Nextflow options (eg. `--resume` instead of `-resume`) + - Supplied parameters validate against the pipeline JSON schema (eg. correct variable types, required values) + - If either parameter validation fails or the pipeline has errors, a warning is given about any unexpected parameters found which are not described in the pipeline schema. + - This behaviour can be disabled by using `--validate_params false` +- Added profiles to support the [Charliecloud](https://hpc.github.io/charliecloud/) and [Shifter](https://nersc.gitlab.io/development/shifter/how-to-use/) container engines [[#824](https://github.com/nf-core/tools/issues/824)] + - Note that Charliecloud requires Nextflow version `v21.03.0-edge` or later. +- Profiles for container engines now explicitly _disable_ all other engines [[#867](https://github.com/nf-core/tools/issues/867)] +- Fixed typo in nf-core-lint CI that prevented the markdown summary from being automatically posted on PRs as a comment. +- Changed default for `--input` from `data/*{1,2}.fastq.gz` to `null`, as this is now validated by the schema as a required value. +- Removed support for `--name` parameter for custom run names. + - The same functionality for MultiQC still exists with the core Nextflow `-name` option. +- Added to template docs about how to identify process name for resource customisation +- The parameters `--max_memory` and `--max_time` are now validated against a regular expression [[#793](https://github.com/nf-core/tools/issues/793)] + - Must be written in the format `123.GB` / `456.h` with any of the prefixes listed in the [Nextflow docs](https://www.nextflow.io/docs/latest/process.html#memory) + - Bare numbers no longer allowed, avoiding people from trying to specify GB and actually specifying bytes. +- Switched from cookiecutter to Jinja2 [[#880]](https://github.com/nf-core/tools/pull/880) +- Finally dropped the wonderful [cookiecutter](https://github.com/cookiecutter/cookiecutter) library that was behind the first pipeline template that led to nf-core [[#880](https://github.com/nf-core/tools/pull/880)] + - Now rendering templates directly using [Jinja](https://jinja.palletsprojects.com/), which is what cookiecutter was doing anyway ### Modules Initial addition of a number of new helper commands for working with DSL2 modules: -* `modules list` - List available modules -* `modules install` - Install a module from nf-core/modules -* `modules remove` - Remove a module from a pipeline -* `modules create` - Create a module from the template -* `modules create-test-yml` - Create the `test.yml` file for a module with md5 sums, tags, commands and names added -* `modules lint` - Check a module against nf-core guidelines +- `modules list` - List available modules +- `modules install` - Install a module from nf-core/modules +- `modules remove` - Remove a module from a pipeline +- `modules create` - Create a module from the template +- `modules create-test-yml` - Create the `test.yml` file for a module with md5 sums, tags, commands and names added +- `modules lint` - Check a module against nf-core guidelines You can read more about each of these commands in the main tools documentation (see `README.md` or ) ### Tools helper code -* Fixed some bugs in the command line interface for `nf-core launch` and improved formatting [[#829](https://github.com/nf-core/tools/pull/829)] -* New functionality for `nf-core download` to make it compatible with DSL2 pipelines [[#832](https://github.com/nf-core/tools/pull/832)] - * Singularity images in module files are now discovered and fetched - * Direct downloads of Singularity images in python allowed (much faster than running `singularity pull`) - * Downloads now work with `$NXF_SINGULARITY_CACHEDIR` so that pipelines sharing containers have efficient downloads -* Changed behaviour of `nf-core sync` command [[#787](https://github.com/nf-core/tools/issues/787)] - * Instead of opening or updating a PR from `TEMPLATE` directly to `dev`, a new branch is now created from `TEMPLATE` and a PR opened from this to `dev`. - * This is to make it easier to fix merge conflicts without accidentally bringing the entire pipeline history back into the `TEMPLATE` branch (which makes subsequent sync merges much more difficult) +- Fixed some bugs in the command line interface for `nf-core launch` and improved formatting [[#829](https://github.com/nf-core/tools/pull/829)] +- New functionality for `nf-core download` to make it compatible with DSL2 pipelines [[#832](https://github.com/nf-core/tools/pull/832)] + - Singularity images in module files are now discovered and fetched + - Direct downloads of Singularity images in python allowed (much faster than running `singularity pull`) + - Downloads now work with `$NXF_SINGULARITY_CACHEDIR` so that pipelines sharing containers have efficient downloads +- Changed behaviour of `nf-core sync` command [[#787](https://github.com/nf-core/tools/issues/787)] + - Instead of opening or updating a PR from `TEMPLATE` directly to `dev`, a new branch is now created from `TEMPLATE` and a PR opened from this to `dev`. + - This is to make it easier to fix merge conflicts without accidentally bringing the entire pipeline history back into the `TEMPLATE` branch (which makes subsequent sync merges much more difficult) ### Linting -* Major refactor and rewrite of pipieline linting code - * Much better code organisation and maintainability - * New automatically generated documentation using Sphinx - * Numerous new tests and functions, removal of some unnecessary tests -* Added lint check for merge markers [[#321]](https://github.com/nf-core/tools/issues/321) -* Added new option `--fix` to automatically correct some problems detected by linting -* Added validation of default params to `nf-core schema lint` [[#823](https://github.com/nf-core/tools/issues/823)] -* Added schema validation of GitHub action workflows to lint function [[#795](https://github.com/nf-core/tools/issues/795)] -* Fixed bug in schema title and description validation -* Added second progress bar for conda dependencies lint check, as it can be slow [[#299](https://github.com/nf-core/tools/issues/299)] -* Added new lint test to check files that should be unchanged from the pipeline. -* Added the possibility to ignore lint tests using a `nf-core-lint.yml` config file [[#809](https://github.com/nf-core/tools/pull/809)] +- Major refactor and rewrite of pipieline linting code + - Much better code organisation and maintainability + - New automatically generated documentation using Sphinx + - Numerous new tests and functions, removal of some unnecessary tests +- Added lint check for merge markers [[#321]](https://github.com/nf-core/tools/issues/321) +- Added new option `--fix` to automatically correct some problems detected by linting +- Added validation of default params to `nf-core schema lint` [[#823](https://github.com/nf-core/tools/issues/823)] +- Added schema validation of GitHub action workflows to lint function [[#795](https://github.com/nf-core/tools/issues/795)] +- Fixed bug in schema title and description validation +- Added second progress bar for conda dependencies lint check, as it can be slow [[#299](https://github.com/nf-core/tools/issues/299)] +- Added new lint test to check files that should be unchanged from the pipeline. +- Added the possibility to ignore lint tests using a `nf-core-lint.yml` config file [[#809](https://github.com/nf-core/tools/pull/809)] ## [v1.12.1 - Silver Dolphin](https://github.com/nf-core/tools/releases/tag/1.12.1) - [2020-12-03] ### Template -* Finished switch from `$baseDir` to `$projectDir` in `iGenomes.conf` and `main.nf` - * Main fix is for `smail_fields` which was a bug introduced in the previous release. Sorry about that! -* Ported a number of small content tweaks from nf-core/eager to the template [[#786](https://github.com/nf-core/tools/issues/786)] - * Better contributing documentation, more placeholders in documentation files, more relaxed markdownlint exceptions for certain HTML tags, more content for the PR and issue templates. +- Finished switch from `$baseDir` to `$projectDir` in `iGenomes.conf` and `main.nf` + - Main fix is for `smail_fields` which was a bug introduced in the previous release. Sorry about that! +- Ported a number of small content tweaks from nf-core/eager to the template [[#786](https://github.com/nf-core/tools/issues/786)] + - Better contributing documentation, more placeholders in documentation files, more relaxed markdownlint exceptions for certain HTML tags, more content for the PR and issue templates. ### Tools helper code -* Pipeline schema: make parameters of type `range` to `number`. [[#738](https://github.com/nf-core/tools/issues/738)] -* Respect `$NXF_HOME` when looking for pipelines with `nf-core list` [[#798](https://github.com/nf-core/tools/issues/798)] -* Swapped PyInquirer with questionary for command line questions in `launch.py` [[#726](https://github.com/nf-core/tools/issues/726)] - * This should fix conda installation issues that some people had been hitting - * The change also allows other improvements to the UI -* Fix linting crash when a file deleted but not yet staged in git [[#796](https://github.com/nf-core/tools/issues/796)] +- Pipeline schema: make parameters of type `range` to `number`. [[#738](https://github.com/nf-core/tools/issues/738)] +- Respect `$NXF_HOME` when looking for pipelines with `nf-core list` [[#798](https://github.com/nf-core/tools/issues/798)] +- Swapped PyInquirer with questionary for command line questions in `launch.py` [[#726](https://github.com/nf-core/tools/issues/726)] + - This should fix conda installation issues that some people had been hitting + - The change also allows other improvements to the UI +- Fix linting crash when a file deleted but not yet staged in git [[#796](https://github.com/nf-core/tools/issues/796)] ## [v1.12 - Mercury Weasel](https://github.com/nf-core/tools/releases/tag/1.12) - [2020-11-19] ### Tools helper code -* Updated `nf_core` documentation generator for building [https://nf-co.re/tools-docs/](https://nf-co.re/tools-docs/) +- Updated `nf_core` documentation generator for building [https://nf-co.re/tools-docs/](https://nf-co.re/tools-docs/) ### Template -* Make CI comments work with PRs from forks [[#765](https://github.com/nf-core/tools/issues/765)] - * Branch protection and linting results should now show on all PRs -* Updated GitHub issue templates, which had stopped working -* Refactored GitHub Actions so that the AWS full-scale tests are triggered after docker build is finished - * DockerHub push workflow split into two - one for dev, one for releases -* Updated actions to no longer use `set-env` which is now depreciating [[#739](https://github.com/nf-core/tools/issues/739)] -* Added config import for `test_full` in `nextflow.config` -* Switched depreciated `$baseDir` to `$projectDir` -* Updated minimum Nextflow version to `20.04.10` -* Make Nextflow installation less verbose in GitHub Actions [[#780](https://github.com/nf-core/tools/pull/780)] +- Make CI comments work with PRs from forks [[#765](https://github.com/nf-core/tools/issues/765)] + - Branch protection and linting results should now show on all PRs +- Updated GitHub issue templates, which had stopped working +- Refactored GitHub Actions so that the AWS full-scale tests are triggered after docker build is finished + - DockerHub push workflow split into two - one for dev, one for releases +- Updated actions to no longer use `set-env` which is now depreciating [[#739](https://github.com/nf-core/tools/issues/739)] +- Added config import for `test_full` in `nextflow.config` +- Switched depreciated `$baseDir` to `$projectDir` +- Updated minimum Nextflow version to `20.04.10` +- Make Nextflow installation less verbose in GitHub Actions [[#780](https://github.com/nf-core/tools/pull/780)] ### Linting -* Updated code to display colours in GitHub Actions log output -* Allow tests to pass with `dev` version of nf-core/tools (previous failure due to base image version) -* Lint code no longer tries to post GitHub PR comments. This is now done in a GitHub Action only. +- Updated code to display colours in GitHub Actions log output +- Allow tests to pass with `dev` version of nf-core/tools (previous failure due to base image version) +- Lint code no longer tries to post GitHub PR comments. This is now done in a GitHub Action only. ## [v1.11 - Iron Tiger](https://github.com/nf-core/tools/releases/tag/1.11) - [2020-10-27] ### Template -* Fix command error in `awstest.yml` GitHub Action workflow. -* Allow manual triggering of AWS test GitHub Action workflows. -* Remove TODO item, which was proposing the usage of additional files beside `usage.md` and `output.md` for documentation. -* Added a Podman profile, which enables Podman as container. -* Updated linting for GitHub actions AWS tests workflows. +- Fix command error in `awstest.yml` GitHub Action workflow. +- Allow manual triggering of AWS test GitHub Action workflows. +- Remove TODO item, which was proposing the usage of additional files beside `usage.md` and `output.md` for documentation. +- Added a Podman profile, which enables Podman as container. +- Updated linting for GitHub actions AWS tests workflows. ### Linting -* Made a base-level `Dockerfile` a warning instead of failure -* Added a lint failure if the old `bin/markdown_to_html.r` script is found -* Update `rich` package dependency and use new markup escaping to change `[[!]]` back to `[!]` again +- Made a base-level `Dockerfile` a warning instead of failure +- Added a lint failure if the old `bin/markdown_to_html.r` script is found +- Update `rich` package dependency and use new markup escaping to change `[[!]]` back to `[!]` again ### Other -* Pipeline sync - fetch full repo when checking out before sync -* Sync - Add GitHub actions manual trigger option +- Pipeline sync - fetch full repo when checking out before sync +- Sync - Add GitHub actions manual trigger option ## [v1.10.2 - Copper Camel _(brought back from the dead)_](https://github.com/nf-core/tools/releases/tag/1.10.2) - [2020-07-31] Second patch release to address some small errors discovered in the pipeline template. Apologies for the inconvenience. -* Fix syntax error in `/push_dockerhub.yml` GitHub Action workflow -* Change `params.readPaths` -> `params.input_paths` in `test_full.config` -* Check results when posting the lint results as a GitHub comment - * This feature is unfortunately not possible when making PRs from forks outside of the nf-core organisation for now. -* More major refactoring of the automated pipeline sync - * New GitHub Actions matrix parallelisation of sync jobs across pipelines [[#673](https://github.com/nf-core/tools/issues/673)] - * Removed the `--all` behaviour from `nf-core sync` as we no longer need it - * Sync now uses a new list of pipelines on the website which does not include archived pipelines [[#712](https://github.com/nf-core/tools/issues/712)] - * When making a PR it checks if a PR already exists - if so it updates it [[#710](https://github.com/nf-core/tools/issues/710)] - * More tests and code refactoring for more stable code. Hopefully fixes 404 error [[#711](https://github.com/nf-core/tools/issues/711)] +- Fix syntax error in `/push_dockerhub.yml` GitHub Action workflow +- Change `params.readPaths` -> `params.input_paths` in `test_full.config` +- Check results when posting the lint results as a GitHub comment + - This feature is unfortunately not possible when making PRs from forks outside of the nf-core organisation for now. +- More major refactoring of the automated pipeline sync + - New GitHub Actions matrix parallelisation of sync jobs across pipelines [[#673](https://github.com/nf-core/tools/issues/673)] + - Removed the `--all` behaviour from `nf-core sync` as we no longer need it + - Sync now uses a new list of pipelines on the website which does not include archived pipelines [[#712](https://github.com/nf-core/tools/issues/712)] + - When making a PR it checks if a PR already exists - if so it updates it [[#710](https://github.com/nf-core/tools/issues/710)] + - More tests and code refactoring for more stable code. Hopefully fixes 404 error [[#711](https://github.com/nf-core/tools/issues/711)] ## [v1.10.1 - Copper Camel _(patch)_](https://github.com/nf-core/tools/releases/tag/1.10.1) - [2020-07-30] Patch release to fix the automatic template synchronisation, which failed in the v1.10 release. -* Improved logging: `nf-core --log-file log.txt` now saves a verbose log to disk. -* nf-core/tools GitHub Actions pipeline sync now uploads verbose log as an artifact. -* Sync - fixed several minor bugs, made logging less verbose. -* Python Rich library updated to `>=4.2.1` -* Hopefully fix git config for pipeline sync so that commit comes from @nf-core-bot -* Fix sync auto-PR text indentation so that it doesn't all show as code -* Added explicit flag `--show-passed` for `nf-core lint` instead of taking logging verbosity +- Improved logging: `nf-core --log-file log.txt` now saves a verbose log to disk. +- nf-core/tools GitHub Actions pipeline sync now uploads verbose log as an artifact. +- Sync - fixed several minor bugs, made logging less verbose. +- Python Rich library updated to `>=4.2.1` +- Hopefully fix git config for pipeline sync so that commit comes from @nf-core-bot +- Fix sync auto-PR text indentation so that it doesn't all show as code +- Added explicit flag `--show-passed` for `nf-core lint` instead of taking logging verbosity ## [v1.10 - Copper Camel](https://github.com/nf-core/tools/releases/tag/1.10) - [2020-07-30] @@ -476,23 +476,23 @@ pipeline with their ID, a description, a longer help text, an optional default v The files will be used in a number of places: -* Automatic validation of supplied parameters when running pipelines - * Pipeline execution can be immediately stopped if a required `param` is missing, +- Automatic validation of supplied parameters when running pipelines + - Pipeline execution can be immediately stopped if a required `param` is missing, or does not conform to the patterns / allowed values in the schema. -* Generation of pipeline command-line help - * Running `nextflow run --help` will use the schema to generate a help text automatically -* Building online documentation on the [nf-core website](https://nf-co.re) -* Integration with 3rd party graphical user interfaces +- Generation of pipeline command-line help + - Running `nextflow run --help` will use the schema to generate a help text automatically +- Building online documentation on the [nf-core website](https://nf-co.re) +- Integration with 3rd party graphical user interfaces To support these new schema files, nf-core/tools now comes with a new set of commands: `nf-core schema`. -* Pipeline schema can be generated or updated using `nf-core schema build` - this takes the parameters from +- Pipeline schema can be generated or updated using `nf-core schema build` - this takes the parameters from the pipeline config file and prompts the developer for any mismatch between schema and pipeline. - * Once a skeleton Schema file has been built, the command makes use of a new nf-core website tool to provide + - Once a skeleton Schema file has been built, the command makes use of a new nf-core website tool to provide a user friendly graphical interface for developers to add content to their schema: [https://nf-co.re/pipeline_schema_builder](https://nf-co.re/pipeline_schema_builder) -* Pipelines will be automatically tested for valid schema that describe all pipeline parameters using the +- Pipelines will be automatically tested for valid schema that describe all pipeline parameters using the `nf-core schema lint` command (also included as part of the main `nf-core lint` command). -* Users can validate their set of pipeline inputs using the `nf-core schema validate` command. +- Users can validate their set of pipeline inputs using the `nf-core schema validate` command. In addition to the new schema commands, the `nf-core launch` command has been completely rewritten from scratch to make use of the new pipeline schema. This command can use either an interactive command-line @@ -514,399 +514,400 @@ making a pull-request. See [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md) ### Template -* Add `--publish_dir_mode` parameter [#585](https://github.com/nf-core/tools/issues/585) -* Isolate R library paths to those in container [#541](https://github.com/nf-core/tools/issues/541) -* Added new style of pipeline parameters JSON schema to pipeline template -* Add ability to attach MultiQC reports to completion emails when using `mail` -* Update `output.md` and add in 'Pipeline information' section describing standard NF and pipeline reporting. -* Build Docker image using GitHub Actions, then push to Docker Hub (instead of building on Docker Hub) -* Add Slack channel badge in pipeline README -* Allow multiple container tags in `ci.yml` if performing multiple tests in parallel -* Add AWS CI tests and full tests GitHub Actions workflows -* Update AWS CI tests and full tests secrets names -* Added `macs_gsize` for danRer10, based on [this post](https://biostar.galaxyproject.org/p/18272/) -* Add information about config files used for workflow execution (`workflow.configFiles`) to summary -* Fix `markdown_to_html.py` to work with Python 2 and 3. -* Change `params.reads` -> `params.input` -* Adding TODOs and MultiQC process in DSL2 template -* Change `params.readPaths` -> `params.input_paths` -* Added a `.github/.dockstore.yml` config file for automatic workflow registration with [dockstore.org](https://dockstore.org/) +- Add `--publish_dir_mode` parameter [#585](https://github.com/nf-core/tools/issues/585) +- Isolate R library paths to those in container [#541](https://github.com/nf-core/tools/issues/541) +- Added new style of pipeline parameters JSON schema to pipeline template +- Add ability to attach MultiQC reports to completion emails when using `mail` +- Update `output.md` and add in 'Pipeline information' section describing standard NF and pipeline reporting. +- Build Docker image using GitHub Actions, then push to Docker Hub (instead of building on Docker Hub) +- Add Slack channel badge in pipeline README +- Allow multiple container tags in `ci.yml` if performing multiple tests in parallel +- Add AWS CI tests and full tests GitHub Actions workflows +- Update AWS CI tests and full tests secrets names +- Added `macs_gsize` for danRer10, based on [this post](https://biostar.galaxyproject.org/p/18272/) +- Add information about config files used for workflow execution (`workflow.configFiles`) to summary +- Fix `markdown_to_html.py` to work with Python 2 and 3. +- Change `params.reads` -> `params.input` +- Adding TODOs and MultiQC process in DSL2 template +- Change `params.readPaths` -> `params.input_paths` +- Added a `.github/.dockstore.yml` config file for automatic workflow registration with [dockstore.org](https://dockstore.org/) ### Linting -* Refactored PR branch tests to be a little clearer. -* Linting error docs explain how to add an additional branch protecton rule to the `branch.yml` GitHub Actions workflow. -* Adapted linting docs to the new PR branch tests. -* Failure for missing the readme bioconda badge is now a warn, in case this badge is not relevant -* Added test for template `{{ cookiecutter.var }}` placeholders -* Fix failure when providing version along with build id for Conda packages -* New `--json` and `--markdown` options to print lint results to JSON / markdown files -* Linting code now automatically posts warning / failing results to GitHub PRs as a comment if it can -* Added AWS GitHub Actions workflows linting -* Fail if `params.input` isn't defined. -* Beautiful new progress bar to look at whilst linting is running and awesome new formatted output on the command line :heart_eyes: - * All made using the excellent [`rich` python library](https://github.com/willmcgugan/rich) - check it out! -* Tests looking for `TODO` strings should now ignore editor backup files. [#477](https://github.com/nf-core/tools/issues/477) +- Refactored PR branch tests to be a little clearer. +- Linting error docs explain how to add an additional branch protecton rule to the `branch.yml` GitHub Actions workflow. +- Adapted linting docs to the new PR branch tests. +- Failure for missing the readme bioconda badge is now a warn, in case this badge is not relevant +- Added test for template `{{ cookiecutter.var }}` placeholders +- Fix failure when providing version along with build id for Conda packages +- New `--json` and `--markdown` options to print lint results to JSON / markdown files +- Linting code now automatically posts warning / failing results to GitHub PRs as a comment if it can +- Added AWS GitHub Actions workflows linting +- Fail if `params.input` isn't defined. +- Beautiful new progress bar to look at whilst linting is running and awesome new formatted output on the command line :heart_eyes: + - All made using the excellent [`rich` python library](https://github.com/willmcgugan/rich) - check it out! +- Tests looking for `TODO` strings should now ignore editor backup files. [#477](https://github.com/nf-core/tools/issues/477) ### nf-core/tools Continuous Integration -* Added CI test to check for PRs against `master` in tools repo -* CI PR branch tests fixed & now automatically add a comment on the PR if failing, explaining what is wrong -* Move some of the issue and PR templates into HTML `` so that they don't show in issues / PRs +- Added CI test to check for PRs against `master` in tools repo +- CI PR branch tests fixed & now automatically add a comment on the PR if failing, explaining what is wrong +- Move some of the issue and PR templates into HTML `` so that they don't show in issues / PRs ### Other -* Describe alternative installation method via conda with `conda env create` -* nf-core/tools version number now printed underneath header artwork -* Bumped Conda version shipped with nfcore/base to 4.8.2 -* Added log message when creating new pipelines that people should talk to the community about their plans -* Fixed 'on completion' emails sent using the `mail` command not containing body text. -* Improved command-line help text for nf-core/tools -* `nf-core list` now hides archived pipelines unless `--show_archived` flag is set -* Command line tools now checks if there is a new version of nf-core/tools available - * Disable this by setting the environment variable `NFCORE_NO_VERSION_CHECK`, eg. `export NFCORE_NO_VERSION_CHECK=1` -* Better command-line output formatting of nearly all `nf-core` commands using [`rich`](https://github.com/willmcgugan/rich) +- Describe alternative installation method via conda with `conda env create` +- nf-core/tools version number now printed underneath header artwork +- Bumped Conda version shipped with nfcore/base to 4.8.2 +- Added log message when creating new pipelines that people should talk to the community about their plans +- Fixed 'on completion' emails sent using the `mail` command not containing body text. +- Improved command-line help text for nf-core/tools +- `nf-core list` now hides archived pipelines unless `--show_archived` flag is set +- Command line tools now checks if there is a new version of nf-core/tools available + - Disable this by setting the environment variable `NFCORE_NO_VERSION_CHECK`, eg. `export NFCORE_NO_VERSION_CHECK=1` +- Better command-line output formatting of nearly all `nf-core` commands using [`rich`](https://github.com/willmcgugan/rich) ## [v1.9 - Platinum Pigeon](https://github.com/nf-core/tools/releases/tag/1.9) - [2020-02-20] ### Continuous integration -* Travis CI tests are now deprecated in favor of GitHub Actions within the pipeline template. - * `nf-core bump-version` support has been removed for `.travis.yml` - * `nf-core lint` now fails if a `.travis.yml` file is found -* Ported nf-core/tools Travis CI automation to GitHub Actions. -* Fixed the build for the nf-core/tools API documentation on the website +- Travis CI tests are now deprecated in favor of GitHub Actions within the pipeline template. + - `nf-core bump-version` support has been removed for `.travis.yml` + - `nf-core lint` now fails if a `.travis.yml` file is found +- Ported nf-core/tools Travis CI automation to GitHub Actions. +- Fixed the build for the nf-core/tools API documentation on the website ### Template -* Rewrote the documentation markdown > HTML conversion in Python instead of R -* Fixed rendering of images in output documentation [#391](https://github.com/nf-core/tools/issues/391) -* Removed the requirement for R in the conda environment -* Make `params.multiqc_config` give an _additional_ MultiQC config file instead of replacing the one that ships with the pipeline -* Ignore only `tests/` and `testing/` directories in `.gitignore` to avoid ignoring `test.config` configuration file -* Rephrase docs to promote usage of containers over Conda to ensure reproducibility -* Stage the workflow summary YAML file within MultiQC work directory +- Rewrote the documentation markdown > HTML conversion in Python instead of R +- Fixed rendering of images in output documentation [#391](https://github.com/nf-core/tools/issues/391) +- Removed the requirement for R in the conda environment +- Make `params.multiqc_config` give an _additional_ MultiQC config file instead of replacing the one that ships with the pipeline +- Ignore only `tests/` and `testing/` directories in `.gitignore` to avoid ignoring `test.config` configuration file +- Rephrase docs to promote usage of containers over Conda to ensure reproducibility +- Stage the workflow summary YAML file within MultiQC work directory ### Linting -* Removed linting for CircleCI -* Allow any one of `params.reads` or `params.input` or `params.design` before warning -* Added whitespace padding to lint error URLs -* Improved documentation for lint errors -* Allow either `>=` or `!>=` in nextflow version checks (the latter exits with an error instead of just warning) [#506](https://github.com/nf-core/tools/issues/506) -* Check that `manifest.version` ends in `dev` and throw a warning if not - * If running with `--release` check the opposite and fail if not -* Tidied up error messages and syntax for linting GitHub actions branch tests -* Add YAML validator -* Don't print test results if we have a critical error +- Removed linting for CircleCI +- Allow any one of `params.reads` or `params.input` or `params.design` before warning +- Added whitespace padding to lint error URLs +- Improved documentation for lint errors +- Allow either `>=` or `!>=` in nextflow version checks (the latter exits with an error instead of just warning) [#506](https://github.com/nf-core/tools/issues/506) +- Check that `manifest.version` ends in `dev` and throw a warning if not + - If running with `--release` check the opposite and fail if not +- Tidied up error messages and syntax for linting GitHub actions branch tests +- Add YAML validator +- Don't print test results if we have a critical error ### Other -* Fix automatic synchronisation of the template after releases of nf-core/tools -* Improve documentation for installing `nf-core/tools` -* Replace preprint by the new nf-core publication in Nature Biotechnology :champagne: -* Use `stderr` instead of `stdout` for header artwork -* Tolerate unexpected output from `nextflow config` command -* Add social preview image -* Added a [release checklist](.github/RELEASE_CHECKLIST.md) for the tools repo +- Fix automatic synchronisation of the template after releases of nf-core/tools +- Improve documentation for installing `nf-core/tools` +- Replace preprint by the new nf-core publication in Nature Biotechnology :champagne: +- Use `stderr` instead of `stdout` for header artwork +- Tolerate unexpected output from `nextflow config` command +- Add social preview image +- Added a [release checklist](.github/RELEASE_CHECKLIST.md) for the tools repo ## [v1.8 - Black Sheep](https://github.com/nf-core/tools/releases/tag/1.8) - [2020-01-27] ### Continuous integration -* GitHub Actions CI workflows are now included in the template pipeline - * Please update these files to match the existing tests that you have in `.travis.yml` -* Travis CI tests will be deprecated from the next `tools` release -* Linting will generate a warning if GitHub Actions workflows do not exist and if applicable to remove Travis CI workflow file i.e. `.travis.yml`. +- GitHub Actions CI workflows are now included in the template pipeline + - Please update these files to match the existing tests that you have in `.travis.yml` +- Travis CI tests will be deprecated from the next `tools` release +- Linting will generate a warning if GitHub Actions workflows do not exist and if applicable to remove Travis CI workflow file i.e. `.travis.yml`. ### Tools helper code -* Refactored the template synchronisation code to be part of the main nf-core tool -* `nf-core bump-version` now also bumps the version string of the exported conda environment in the Dockerfile -* Updated Blacklist of synced pipelines -* Ignore pre-releases in `nf-core list` -* Updated documentation for `nf-core download` -* Fixed typo in `nf-core launch` final command -* Handle missing pipeline descriptions in `nf-core list` -* Migrate tools package CI to GitHub Actions +- Refactored the template synchronisation code to be part of the main nf-core tool +- `nf-core bump-version` now also bumps the version string of the exported conda environment in the Dockerfile +- Updated Blacklist of synced pipelines +- Ignore pre-releases in `nf-core list` +- Updated documentation for `nf-core download` +- Fixed typo in `nf-core launch` final command +- Handle missing pipeline descriptions in `nf-core list` +- Migrate tools package CI to GitHub Actions ### Linting -* Adjusted linting to enable `patch` branches from being tested -* Warn if GitHub Actions workflows do not exist, warn if `.travis.yml` and circleCI are there -* Lint for `Singularity` file and raise error if found [#458](https://github.com/nf-core/tools/issues/458) -* Added linting of GitHub Actions workflows `linting.yml`, `ci.yml` and `branch.yml` -* Warn if pipeline name contains upper case letters or non alphabetical characters [#85](https://github.com/nf-core/tools/issues/85) -* Make CI tests of lint code pass for releases +- Adjusted linting to enable `patch` branches from being tested +- Warn if GitHub Actions workflows do not exist, warn if `.travis.yml` and circleCI are there +- Lint for `Singularity` file and raise error if found [#458](https://github.com/nf-core/tools/issues/458) +- Added linting of GitHub Actions workflows `linting.yml`, `ci.yml` and `branch.yml` +- Warn if pipeline name contains upper case letters or non alphabetical characters [#85](https://github.com/nf-core/tools/issues/85) +- Make CI tests of lint code pass for releases ### Template pipeline -* Fixed incorrect paths in iGenomes config as described in issue [#418](https://github.com/nf-core/tools/issues/418) -* Fixed incorrect usage of non-existent parameter in the template [#446](https://github.com/nf-core/tools/issues/446) -* Add UCSC genomes to `igenomes.config` and add paths to all genome indices -* Change `maxMultiqcEmailFileSize` parameter to `max_multiqc_email_size` -* Export conda environment in Docker file [#349](https://github.com/nf-core/tools/issues/349) -* Change remaining parameters from `camelCase` to `snake_case` [#39](https://github.com/nf-core/hic/issues/39) - * `--singleEnd` to `--single_end` - * `--igenomesIgnore` to `--igenomes_ignore` - * Having the old camelCase versions of these will now throw an error -* Add `autoMounts=true` to default singularity profile -* Add in `markdownlint` checks that were being ignored by default -* Disable ansi logging in the travis CI tests -* Move `params`section from `base.config` to `nextflow.config` -* Use `env` scope to export `PYTHONNOUSERSITE` in `nextflow.config` to prevent conflicts with host Python environment -* Bump minimum Nextflow version to `19.10.0` - required to properly use `env` scope in `nextflow.config` -* Added support for nf-tower in the travis tests, using public mailbox nf-core@mailinator.com -* Add link to [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and [Semantic Versioning](http://semver.org/spec/v2.0.0.html) to CHANGELOG -* Adjusted `.travis.yml` checks to allow for `patch` branches to be tested -* Add Python 3.7 dependency to the `environment.yml` file -* Remove `awsbatch` profile cf [nf-core/configs#71](https://github.com/nf-core/configs/pull/71) -* Make `scrape_software_versions.py` compatible with Python3 to enable miniconda3 in [base image PR](https://github.com/nf-core/tools/pull/462) -* Add GitHub Actions workflows and respective linting -* Add `NXF_ANSI_LOG` as global environment variable to template GitHub Actions CI workflow -* Fixed global environment variable in GitHub Actions CI workflow -* Add `--awscli` parameter -* Add `README.txt` path for genomes in `igenomes.config` [nf-core/atacseq#75](https://github.com/nf-core/atacseq/issues/75) -* Fix buggy ANSI codes in pipeline summary log messages -* Add a `TODO` line in the new GitHub Actions CI test files +- Fixed incorrect paths in iGenomes config as described in issue [#418](https://github.com/nf-core/tools/issues/418) +- Fixed incorrect usage of non-existent parameter in the template [#446](https://github.com/nf-core/tools/issues/446) +- Add UCSC genomes to `igenomes.config` and add paths to all genome indices +- Change `maxMultiqcEmailFileSize` parameter to `max_multiqc_email_size` +- Export conda environment in Docker file [#349](https://github.com/nf-core/tools/issues/349) +- Change remaining parameters from `camelCase` to `snake_case` [#39](https://github.com/nf-core/hic/issues/39) + - `--singleEnd` to `--single_end` + - `--igenomesIgnore` to `--igenomes_ignore` + - Having the old camelCase versions of these will now throw an error +- Add `autoMounts=true` to default singularity profile +- Add in `markdownlint` checks that were being ignored by default +- Disable ansi logging in the travis CI tests +- Move `params`section from `base.config` to `nextflow.config` +- Use `env` scope to export `PYTHONNOUSERSITE` in `nextflow.config` to prevent conflicts with host Python environment +- Bump minimum Nextflow version to `19.10.0` - required to properly use `env` scope in `nextflow.config` +- Added support for nf-tower in the travis tests, using public mailbox nf-core@mailinator.com +- Add link to [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and [Semantic Versioning](http://semver.org/spec/v2.0.0.html) to CHANGELOG +- Adjusted `.travis.yml` checks to allow for `patch` branches to be tested +- Add Python 3.7 dependency to the `environment.yml` file +- Remove `awsbatch` profile cf [nf-core/configs#71](https://github.com/nf-core/configs/pull/71) +- Make `scrape_software_versions.py` compatible with Python3 to enable miniconda3 in [base image PR](https://github.com/nf-core/tools/pull/462) +- Add GitHub Actions workflows and respective linting +- Add `NXF_ANSI_LOG` as global environment variable to template GitHub Actions CI workflow +- Fixed global environment variable in GitHub Actions CI workflow +- Add `--awscli` parameter +- Add `README.txt` path for genomes in `igenomes.config` [nf-core/atacseq#75](https://github.com/nf-core/atacseq/issues/75) +- Fix buggy ANSI codes in pipeline summary log messages +- Add a `TODO` line in the new GitHub Actions CI test files ### Base Docker image -* Use miniconda3 instead of miniconda for a Python 3k base environment - * If you still need Python 2 for your pipeline, add `conda-forge::python=2.7.4` to the dependencies in your `environment.yml` -* Update conda version to 4.7.12 +- Use miniconda3 instead of miniconda for a Python 3k base environment + - If you still need Python 2 for your pipeline, add `conda-forge::python=2.7.4` to the dependencies in your `environment.yml` +- Update conda version to 4.7.12 ### Other -* Updated Base Dockerfile to Conda 4.7.10 -* Entirely switched from Travis-Ci.org to Travis-Ci.com for template and tools -* Improved core documentation (`-profile`) +- Updated Base Dockerfile to Conda 4.7.10 +- Entirely switched from Travis-Ci.org to Travis-Ci.com for template and tools +- Improved core documentation (`-profile`) ## [v1.7 - Titanium Kangaroo](https://github.com/nf-core/tools/releases/tag/1.7) - [2019-10-07] ### Tools helper code -* The tools `create` command now sets up a `TEMPLATE` and a `dev` branch for syncing -* Fixed issue [379](https://github.com/nf-core/tools/issues/379) -* nf-core launch now uses stable parameter schema version 0.1.0 -* Check that PR from patch or dev branch is acceptable by linting -* Made code compatible with Python 3.7 -* The `download` command now also fetches institutional configs from nf-core/configs -* When listing pipelines, a nicer message is given for the rare case of a detached `HEAD` ref in a locally pulled pipeline. [#297](https://github.com/nf-core/tools/issues/297) -* The `download` command can now compress files into a single archive. -* `nf-core create` now fetches a logo for the pipeline from the nf-core website -* The readme should now be rendered properly on PyPI. +- The tools `create` command now sets up a `TEMPLATE` and a `dev` branch for syncing +- Fixed issue [379](https://github.com/nf-core/tools/issues/379) +- nf-core launch now uses stable parameter schema version 0.1.0 +- Check that PR from patch or dev branch is acceptable by linting +- Made code compatible with Python 3.7 +- The `download` command now also fetches institutional configs from nf-core/configs +- When listing pipelines, a nicer message is given for the rare case of a detached `HEAD` ref in a locally pulled pipeline. [#297](https://github.com/nf-core/tools/issues/297) +- The `download` command can now compress files into a single archive. +- `nf-core create` now fetches a logo for the pipeline from the nf-core website +- The readme should now be rendered properly on PyPI. ### Syncing -* Can now sync a targeted pipeline via command-line -* Updated Blacklist of synced pipelines -* Removed `chipseq` from Blacklist of synced pipelines -* Fixed issue [#314](https://github.com/nf-core/tools/issues/314) +- Can now sync a targeted pipeline via command-line +- Updated Blacklist of synced pipelines +- Removed `chipseq` from Blacklist of synced pipelines +- Fixed issue [#314](https://github.com/nf-core/tools/issues/314) ### Linting -* If the container slug does not contain the nf-core organisation (for example during development on a fork), linting will raise a warning, and an error with release mode on +- If the container slug does not contain the nf-core organisation (for example during development on a fork), linting will raise a warning, and an error with release mode on ### Template pipeline -* Add new code for Travis CI to allow PRs from patch branches too -* Fix small typo in central readme of tools for future releases -* Small code polishing + typo fix in the template main.nf file -* Header ANSI codes no longer print `[2m` to console when using `-with-ansi` -* Switched to yaml.safe_load() to fix PyYAML warning that was thrown because of a possible [exploit](https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation) -* Add `nf-core` citation -* Add proper `nf-core` logo for tools -* Add `Quick Start` section to main README of template -* Fix [Docker RunOptions](https://github.com/nf-core/tools/pull/351) to get UID and GID set in the template -* `Dockerfile` now specifically uses the proper release tag of the nfcore/base image -* Use [`file`](https://github.com/nf-core/tools/pull/354) instead of `new File` +- Add new code for Travis CI to allow PRs from patch branches too +- Fix small typo in central readme of tools for future releases +- Small code polishing + typo fix in the template main.nf file +- Header ANSI codes no longer print `[2m` to console when using `-with-ansi` +- Switched to yaml.safe_load() to fix PyYAML warning that was thrown because of a possible [exploit]() +- Add `nf-core` citation +- Add proper `nf-core` logo for tools +- Add `Quick Start` section to main README of template +- Fix [Docker RunOptions](https://github.com/nf-core/tools/pull/351) to get UID and GID set in the template +- `Dockerfile` now specifically uses the proper release tag of the nfcore/base image +- Use [`file`](https://github.com/nf-core/tools/pull/354) instead of `new File` to avoid weird behavior such as making an `s3:/` directory locally when using an AWS S3 bucket as the `--outdir`. -* Fix workflow.onComplete() message when finishing pipeline -* Update URL for joining the nf-core slack to [https://nf-co.re/join/slack](https://nf-co.re/join/slack) -* Add GitHub Action for CI and Linting -* [Increased default time limit](https://github.com/nf-core/tools/issues/370) to 4h -* Add direct link to the pipeline slack channel in the contribution guidelines -* Add contributions and support heading with links to contribution guidelines and link to the pipeline slack channel in the main README -* Fix Parameters JSON due to new versionized structure -* Added conda-forge::r-markdown=1.1 and conda-forge::r-base=3.6.1 to environment -* Plain-text email template now has nf-core ASCII artwork -* Template configured to use logo fetched from website -* New option `--email_on_fail` which only sends emails if the workflow is not successful -* Add file existence check when checking software versions -* Fixed issue [#165](https://github.com/nf-core/tools/issues/165) - Use `checkIfExists` -* Consistent spacing for `if` statements -* Add sensible resource labels to `base.config` +- Fix workflow.onComplete() message when finishing pipeline +- Update URL for joining the nf-core slack to [https://nf-co.re/join/slack](https://nf-co.re/join/slack) +- Add GitHub Action for CI and Linting +- [Increased default time limit](https://github.com/nf-core/tools/issues/370) to 4h +- Add direct link to the pipeline slack channel in the contribution guidelines +- Add contributions and support heading with links to contribution guidelines and link to the pipeline slack channel in the main README +- Fix Parameters JSON due to new versionized structure +- Added conda-forge::r-markdown=1.1 and conda-forge::r-base=3.6.1 to environment +- Plain-text email template now has nf-core ASCII artwork +- Template configured to use logo fetched from website +- New option `--email_on_fail` which only sends emails if the workflow is not successful +- Add file existence check when checking software versions +- Fixed issue [#165](https://github.com/nf-core/tools/issues/165) - Use `checkIfExists` +- Consistent spacing for `if` statements +- Add sensible resource labels to `base.config` ### Other -* Bump `conda` to 4.6.14 in base nf-core Dockerfile -* Added a Code of Conduct to nf-core/tools, as only the template had this before -* TravisCI tests will now also start for PRs from `patch` branches, [to allow fixing critical issues](https://github.com/nf-core/tools/pull/392) without making a new major release +- Bump `conda` to 4.6.14 in base nf-core Dockerfile +- Added a Code of Conduct to nf-core/tools, as only the template had this before +- TravisCI tests will now also start for PRs from `patch` branches, [to allow fixing critical issues](https://github.com/nf-core/tools/pull/392) without making a new major release ## [v1.6 - Brass Walrus](https://github.com/nf-core/tools/releases/tag/1.6) - [2020-04-09] ### Syncing -* Code refactoring to make the script more readable -* No travis build failure anymore on sync errors -* More verbose logging +- Code refactoring to make the script more readable +- No travis build failure anymore on sync errors +- More verbose logging ### Template pipeline -* awsbatch `work-dir` checking moved to nextflow itself. Removed unsatisfiable check in main.nf template. -* Fixed markdown linting -* Tools CI testing now runs markdown lint on compiled template pipeline -* Migrated large portions of documentation to the [nf-core website](https://github.com/nf-core/nf-co.re/pull/93) -* Removed Gitter references in `.github/` directories for `tools/` and pipeline template. -* Changed `scrape_software_versions.py` to output `.csv` file -* Added `export_plots` parameter to multiqc config -* Corrected some typos as listed [here](https://github.com/nf-core/tools/issues/348) to Guidelines +- awsbatch `work-dir` checking moved to nextflow itself. Removed unsatisfiable check in main.nf template. +- Fixed markdown linting +- Tools CI testing now runs markdown lint on compiled template pipeline +- Migrated large portions of documentation to the [nf-core website](https://github.com/nf-core/nf-co.re/pull/93) +- Removed Gitter references in `.github/` directories for `tools/` and pipeline template. +- Changed `scrape_software_versions.py` to output `.csv` file +- Added `export_plots` parameter to multiqc config +- Corrected some typos as listed [here](https://github.com/nf-core/tools/issues/348) to Guidelines ### Tools helper code -* Drop [nf-core/rnaseq](https://github.com/nf-core/rnaseq]) from `blacklist.json` to make template sync available -* Updated main help command to sort the subcommands in a more logical order -* Updated readme to describe the new `nf-core launch` command -* Fix bugs in `nf-core download` - * The _latest_ release is now fetched by default if not specified - * Downloaded pipeline files are now properly executable. -* Fixed bugs in `nf-core list` - * Sorting now works again - * Output is partially coloured (better highlighting out of date pipelines) - * Improved documentation -* Fixed bugs in `nf-core lint` - * The order of conda channels is now correct, avoiding occasional erroneous errors that packages weren't found ([#207](https://github.com/nf-core/tools/issues/207)) - * Allow edge versions in nf-core pipelines -* Add reporting of ignored errored process - * As a solution for [#103](https://github.com/nf-core/tools/issues/103)) -* Add Bowtie2 and BWA in iGenome config file template +- Drop [nf-core/rnaseq](https://github.com/nf-core/rnaseq]) from `blacklist.json` to make template sync available +- Updated main help command to sort the subcommands in a more logical order +- Updated readme to describe the new `nf-core launch` command +- Fix bugs in `nf-core download` + - The _latest_ release is now fetched by default if not specified + - Downloaded pipeline files are now properly executable. +- Fixed bugs in `nf-core list` + - Sorting now works again + - Output is partially coloured (better highlighting out of date pipelines) + - Improved documentation +- Fixed bugs in `nf-core lint` + - The order of conda channels is now correct, avoiding occasional erroneous errors that packages weren't found ([#207](https://github.com/nf-core/tools/issues/207)) + - Allow edge versions in nf-core pipelines +- Add reporting of ignored errored process + - As a solution for [#103](https://github.com/nf-core/tools/issues/103)) +- Add Bowtie2 and BWA in iGenome config file template ## [v1.5 - Iron Shark](https://github.com/nf-core/tools/releases/tag/1.5) - [2019-03-13] ### Template pipeline -* Dropped Singularity file -* Summary now logs details of the cluster profile used if from [nf-core/configs](https://github.com/nf-core/configs) -* Dockerhub is used in favor of Singularity Hub for pulling when using the Singularity profile -* Changed default container tag from latest to dev -* Brought the logo to life -* Change the default filenames for the pipeline trace files -* Remote fetch of nf-core/configs profiles fails gracefully if offline -* Remove `params.container` and just directly define `process.container` now -* Completion email now includes MultiQC report if not too big -* `params.genome` is now checked if set, to ensure that it's a valid iGenomes key -* Together with nf-core/configs, helper function now checks hostname and suggests a valid config profile -* `awsbatch` executor requires the `tracedir` not to be set to an `s3` bucket. +- Dropped Singularity file +- Summary now logs details of the cluster profile used if from [nf-core/configs](https://github.com/nf-core/configs) +- Dockerhub is used in favor of Singularity Hub for pulling when using the Singularity profile +- Changed default container tag from latest to dev +- Brought the logo to life +- Change the default filenames for the pipeline trace files +- Remote fetch of nf-core/configs profiles fails gracefully if offline +- Remove `params.container` and just directly define `process.container` now +- Completion email now includes MultiQC report if not too big +- `params.genome` is now checked if set, to ensure that it's a valid iGenomes key +- Together with nf-core/configs, helper function now checks hostname and suggests a valid config profile +- `awsbatch` executor requires the `tracedir` not to be set to an `s3` bucket. ### Tools helper code -* New `nf-core launch` command to interactively launch nf-core pipelines from command-line - * Works with a `parameters.settings.json` file shipped with each pipeline - * Discovers additional `params` from the pipeline dynamically -* Drop Python 3.4 support -* `nf-core list` now only shows a value for _"is local latest version"_ column if there is a local copy. -* Lint markdown formatting in automated tests - * Added `markdownlint-cli` for checking Markdown syntax in pipelines and tools repo -* Syncing now reads from a `blacklist.json` in order to exclude pipelines from being synced if necessary. -* Added nf-core tools API description to assist developers with the classes and functions available. - * Docs are automatically built by Travis CI and updated on the nf-co.re website. -* Introduced test for filtering remote workflows by keyword. -* Build tools python API docs - * Use Travis job for api doc generation and publish - -* `nf-core bump-version` now stops before making changes if the linting fails -* Code test coverage - * Introduced test for filtering remote workflows by keyword -* Linting updates - * Now properly searches for conda packages in default channels - * Now correctly validates version pinning for packages from PyPI - * Updates for changes to `process.container` definition +- New `nf-core launch` command to interactively launch nf-core pipelines from command-line + - Works with a `parameters.settings.json` file shipped with each pipeline + - Discovers additional `params` from the pipeline dynamically +- Drop Python 3.4 support +- `nf-core list` now only shows a value for _"is local latest version"_ column if there is a local copy. +- Lint markdown formatting in automated tests + - Added `markdownlint-cli` for checking Markdown syntax in pipelines and tools repo +- Syncing now reads from a `blacklist.json` in order to exclude pipelines from being synced if necessary. +- Added nf-core tools API description to assist developers with the classes and functions available. + - Docs are automatically built by Travis CI and updated on the nf-co.re website. +- Introduced test for filtering remote workflows by keyword. +- Build tools python API docs + + - Use Travis job for api doc generation and publish + +- `nf-core bump-version` now stops before making changes if the linting fails +- Code test coverage + - Introduced test for filtering remote workflows by keyword +- Linting updates + - Now properly searches for conda packages in default channels + - Now correctly validates version pinning for packages from PyPI + - Updates for changes to `process.container` definition ### Other -* Bump `conda` to 4.6.7 in base nf-core Dockerfile +- Bump `conda` to 4.6.7 in base nf-core Dockerfile ## [v1.4 - Tantalum Butterfly](https://github.com/nf-core/tools/releases/tag/1.4) - [2018-12-12] ### Template pipeline -* Institutional custom config profiles moved to github `nf-core/configs` - * These will now be maintained centrally as opposed to being shipped with the pipelines in `conf/` - * Load `base.config` by default for all profiles - * Removed profiles named `standard` and `none` - * Added parameter `--igenomesIgnore` so `igenomes.config` is not loaded if parameter clashes are observed - * Added parameter `--custom_config_version` for custom config version control. Can use this parameter to provide commit id for reproducibility. Defaults to `master` - * Deleted custom configs from template in `conf/` directory i.e. `uzh.config`, `binac.config` and `cfc.config` -* `multiqc_config` and `output_md` are now put into channels instead of using the files directly (see issue [#222](https://github.com/nf-core/tools/issues/222)) -* Added `local.md` to cookiecutter template in `docs/configuration/`. This was referenced in `README.md` but not present. -* Major overhaul of docs to add/remove parameters, unify linking of files and added description for providing custom configs where necessary -* Travis: Pull the `dev` tagged docker image for testing -* Removed UPPMAX-specific documentation from the template. +- Institutional custom config profiles moved to github `nf-core/configs` + - These will now be maintained centrally as opposed to being shipped with the pipelines in `conf/` + - Load `base.config` by default for all profiles + - Removed profiles named `standard` and `none` + - Added parameter `--igenomesIgnore` so `igenomes.config` is not loaded if parameter clashes are observed + - Added parameter `--custom_config_version` for custom config version control. Can use this parameter to provide commit id for reproducibility. Defaults to `master` + - Deleted custom configs from template in `conf/` directory i.e. `uzh.config`, `binac.config` and `cfc.config` +- `multiqc_config` and `output_md` are now put into channels instead of using the files directly (see issue [#222](https://github.com/nf-core/tools/issues/222)) +- Added `local.md` to cookiecutter template in `docs/configuration/`. This was referenced in `README.md` but not present. +- Major overhaul of docs to add/remove parameters, unify linking of files and added description for providing custom configs where necessary +- Travis: Pull the `dev` tagged docker image for testing +- Removed UPPMAX-specific documentation from the template. ### Tools helper code -* Make Travis CI tests fail on pull requests if the `CHANGELOG.md` file hasn't been updated -* Minor bugfixing in Python code (eg. removing unused import statements) -* Made the web requests caching work on multi-user installations -* Handle exception if nextflow isn't installed -* Linting: Update for Travis: Pull the `dev` tagged docker image for testing +- Make Travis CI tests fail on pull requests if the `CHANGELOG.md` file hasn't been updated +- Minor bugfixing in Python code (eg. removing unused import statements) +- Made the web requests caching work on multi-user installations +- Handle exception if nextflow isn't installed +- Linting: Update for Travis: Pull the `dev` tagged docker image for testing ## [v1.3 - Citreous Swordfish](https://github.com/nf-core/tools/releases/tag/1.3) - [2018-11-21] -* `nf-core create` command line interface updated - * Interactive prompts for required arguments if not given - * New flag for workflow author -* Updated channel order for bioconda/conda-forge channels in environment.yaml -* Increased code coverage for sub command `create` and `licenses` -* Fixed nasty dependency hell issue between `pytest` and `py` package in Python 3.4.x -* Introduced `.coveragerc` for pytest-cov configuration, which excludes the pipeline template now from being reported -* Fix [189](https://github.com/nf-core/tools/issues/189): Check for given conda and PyPi package dependencies, if their versions exist -* Added profiles for `cfc`,`binac`, `uzh` that can be synced across pipelines - * Ordering alphabetically for profiles now -* Added `pip install --upgrade pip` to `.travis.yml` to update pip in the Travis CI environment +- `nf-core create` command line interface updated + - Interactive prompts for required arguments if not given + - New flag for workflow author +- Updated channel order for bioconda/conda-forge channels in environment.yaml +- Increased code coverage for sub command `create` and `licenses` +- Fixed nasty dependency hell issue between `pytest` and `py` package in Python 3.4.x +- Introduced `.coveragerc` for pytest-cov configuration, which excludes the pipeline template now from being reported +- Fix [189](https://github.com/nf-core/tools/issues/189): Check for given conda and PyPi package dependencies, if their versions exist +- Added profiles for `cfc`,`binac`, `uzh` that can be synced across pipelines + - Ordering alphabetically for profiles now +- Added `pip install --upgrade pip` to `.travis.yml` to update pip in the Travis CI environment ## [v1.2](https://github.com/nf-core/tools/releases/tag/1.2) - [2018-10-01] -* Updated the `nf-core release` command - * Now called `nf-core bump-versions` instead - * New flag `--nextflow` to change the required nextflow version instead -* Template updates - * Simpler installation of the `nf-core` helper tool, now directly from PyPI - * Bump minimum nextflow version to `0.32.0` - required for built in `manifest.nextflowVersion` check and access to `workflow.manifest` variables from within nextflow scripts - * New `withName` syntax for configs - * Travis tests fail if PRs come against the `master` branch, slightly refactored - * Improved GitHub contributing instructions and pull request / issue templates -* New lint tests - * `.travis.yml` test for PRs made against the `master` branch - * Automatic `--release` option not used if the travis repo is `nf-core/tools` - * Warnings if depreciated variables `params.version` and `params.nf_required_version` are found -* New `nf-core licences` subcommand to show licence for each conda package in a workflow -* `nf-core list` now has options for sorting pipeline nicely -* Latest version of conda used in nf-core base docker image -* Updated PyPI deployment to correctly parse the markdown readme (hopefully!) -* New GitHub contributing instructions and pull request template +- Updated the `nf-core release` command + - Now called `nf-core bump-versions` instead + - New flag `--nextflow` to change the required nextflow version instead +- Template updates + - Simpler installation of the `nf-core` helper tool, now directly from PyPI + - Bump minimum nextflow version to `0.32.0` - required for built in `manifest.nextflowVersion` check and access to `workflow.manifest` variables from within nextflow scripts + - New `withName` syntax for configs + - Travis tests fail if PRs come against the `master` branch, slightly refactored + - Improved GitHub contributing instructions and pull request / issue templates +- New lint tests + - `.travis.yml` test for PRs made against the `master` branch + - Automatic `--release` option not used if the travis repo is `nf-core/tools` + - Warnings if depreciated variables `params.version` and `params.nf_required_version` are found +- New `nf-core licences` subcommand to show licence for each conda package in a workflow +- `nf-core list` now has options for sorting pipeline nicely +- Latest version of conda used in nf-core base docker image +- Updated PyPI deployment to correctly parse the markdown readme (hopefully!) +- New GitHub contributing instructions and pull request template ## [v1.1](https://github.com/nf-core/tools/releases/tag/1.1) - [2018-08-14] Very large release containing lots of work from the first nf-core hackathon, held in SciLifeLab Stockholm. -* The [Cookiecutter template](https://github.com/nf-core/cookiecutter) has been merged into tools - * The old repo above has been archived - * New pipelines are now created using the command `nf-core create` - * The nf-core template and associated linting are now controlled under the same version system -* Large number of template updates and associated linting changes - * New simplified cookiecutter variable usage - * Refactored documentation - simplified and reduced duplication - * Better `manifest` variables instead of `params` for pipeline name and version - * New integrated nextflow version checking - * Updated travis docker pull command to use tagging to allow release tests to pass - * Reverted Docker and Singularity syntax to use `ENV` hack again -* Improved Python readme parsing for PyPI -* Updated Travis tests to check that the correct `dev` branch is being targeted -* New sync tool to automate pipeline updates - * Once initial merges are complete, a nf-core bot account will create PRs for future template updates +- The [Cookiecutter template](https://github.com/nf-core/cookiecutter) has been merged into tools + - The old repo above has been archived + - New pipelines are now created using the command `nf-core create` + - The nf-core template and associated linting are now controlled under the same version system +- Large number of template updates and associated linting changes + - New simplified cookiecutter variable usage + - Refactored documentation - simplified and reduced duplication + - Better `manifest` variables instead of `params` for pipeline name and version + - New integrated nextflow version checking + - Updated travis docker pull command to use tagging to allow release tests to pass + - Reverted Docker and Singularity syntax to use `ENV` hack again +- Improved Python readme parsing for PyPI +- Updated Travis tests to check that the correct `dev` branch is being targeted +- New sync tool to automate pipeline updates + - Once initial merges are complete, a nf-core bot account will create PRs for future template updates ## [v1.0.1](https://github.com/nf-core/tools/releases/tag/1.0.1) - [2018-07-18] @@ -916,7 +917,7 @@ The version 1.0 of nf-core tools cannot be installed from PyPi. This patch fixes Initial release of the nf-core helper tools package. Currently includes four subcommands: -* `nf-core list`: List nf-core pipelines with local info -* `nf-core download`: Download a pipeline and singularity container -* `nf-core lint`: Check pipeline against nf-core guidelines -* `nf-core release`: Update nf-core pipeline version number +- `nf-core list`: List nf-core pipelines with local info +- `nf-core download`: Download a pipeline and singularity container +- `nf-core lint`: Check pipeline against nf-core guidelines +- `nf-core release`: Update nf-core pipeline version number diff --git a/README.md b/README.md index a418150e26..be0054533c 100644 --- a/README.md +++ b/README.md @@ -14,29 +14,30 @@ A python package with helper tools for the nf-core community. ## Table of contents -* [`nf-core` tools installation](#installation) -* [`nf-core list` - List available pipelines](#listing-pipelines) -* [`nf-core launch` - Run a pipeline with interactive parameter prompts](#launch-a-pipeline) -* [`nf-core download` - Download pipeline for offline use](#downloading-pipelines-for-offline-use) -* [`nf-core licences` - List software licences in a pipeline](#pipeline-software-licences) -* [`nf-core create` - Create a new pipeline with the nf-core template](#creating-a-new-pipeline) -* [`nf-core lint` - Check pipeline code against nf-core guidelines](#linting-a-workflow) -* [`nf-core schema` - Work with pipeline schema files](#pipeline-schema) -* [`nf-core bump-version` - Update nf-core pipeline version number](#bumping-a-pipeline-version-number) -* [`nf-core sync` - Synchronise pipeline TEMPLATE branches](#sync-a-pipeline-with-the-template) -* [`nf-core modules` - commands for dealing with DSL2 modules](#modules) - * [`modules list` - List available modules](#list-modules) - * [`modules list remote` - List remote modules](#list-remote-modules) - * [`modules list local` - List installed modules](#list-installed-modules) - * [`modules install` - Install modules in a pipeline](#install-modules-in-a-pipeline) - * [`modules update` - Update modules in a pipeline](#update-modules-in-a-pipeline) - * [`modules remove` - Remove a module from a pipeline](#remove-a-module-from-a-pipeline) - * [`modules create` - Create a module from the template](#create-a-new-module) - * [`modules create-test-yml` - Create the `test.yml` file for a module](#create-a-module-test-config-file) - * [`modules lint` - Check a module against nf-core guidelines](#check-a-module-against-nf-core-guidelines) - * [`modules bump-versions` - Bump software versions of modules](#bump-bioconda-and-container-versions-of-modules-in) - -* [Citation](#citation) +- [`nf-core` tools installation](#installation) +- [`nf-core list` - List available pipelines](#listing-pipelines) +- [`nf-core launch` - Run a pipeline with interactive parameter prompts](#launch-a-pipeline) +- [`nf-core download` - Download pipeline for offline use](#downloading-pipelines-for-offline-use) +- [`nf-core licences` - List software licences in a pipeline](#pipeline-software-licences) +- [`nf-core create` - Create a new pipeline with the nf-core template](#creating-a-new-pipeline) +- [`nf-core lint` - Check pipeline code against nf-core guidelines](#linting-a-workflow) +- [`nf-core schema` - Work with pipeline schema files](#pipeline-schema) +- [`nf-core bump-version` - Update nf-core pipeline version number](#bumping-a-pipeline-version-number) +- [`nf-core sync` - Synchronise pipeline TEMPLATE branches](#sync-a-pipeline-with-the-template) +- [`nf-core modules` - commands for dealing with DSL2 modules](#modules) + + - [`modules list` - List available modules](#list-modules) + - [`modules list remote` - List remote modules](#list-remote-modules) + - [`modules list local` - List installed modules](#list-installed-modules) + - [`modules install` - Install modules in a pipeline](#install-modules-in-a-pipeline) + - [`modules update` - Update modules in a pipeline](#update-modules-in-a-pipeline) + - [`modules remove` - Remove a module from a pipeline](#remove-a-module-from-a-pipeline) + - [`modules create` - Create a module from the template](#create-a-new-module) + - [`modules create-test-yml` - Create the `test.yml` file for a module](#create-a-module-test-config-file) + - [`modules lint` - Check a module against nf-core guidelines](#check-a-module-against-nf-core-guidelines) + - [`modules bump-versions` - Bump software versions of modules](#bump-bioconda-and-container-versions-of-modules-in) + +- [Citation](#citation) The nf-core tools package is written in Python and can be imported and used within other packages. For documentation of the internal Python functions, please refer to the [Tools Python API docs](https://nf-co.re/tools-docs/). @@ -80,10 +81,10 @@ You can use this container on the command line as follows: docker run -itv `pwd`:`pwd` -w `pwd` -u $(id -u):$(id -g) nfcore/tools ``` -* `-i` and `-t` are needed for the interactive cli prompts to work (this tells Docker to use a pseudo-tty with stdin attached) -* The `-v` argument tells Docker to bind your current working directory (`pwd`) to the same path inside the container, so that files created there will be saved to your local file system outside of the container. -* `-w` sets the working directory in the container to this path, so that it's the same as your working directory outside of the container. -* `-u` sets your local user account as the user inside the container, so that any files created have the correct ownership permissions +- `-i` and `-t` are needed for the interactive cli prompts to work (this tells Docker to use a pseudo-tty with stdin attached) +- The `-v` argument tells Docker to bind your current working directory (`pwd`) to the same path inside the container, so that files created there will be saved to your local file system outside of the container. +- `-w` sets the working directory in the container to this path, so that it's the same as your working directory outside of the container. +- `-u` sets your local user account as the user inside the container, so that any files created have the correct ownership permissions After the above base command, you can use the regular command line flags that you would use with other types of installation. For example, to launch the `viralrecon` pipeline: @@ -181,7 +182,7 @@ export NFCORE_NO_VERSION_CHECK=1 ## Listing pipelines -The command `nf-core list` shows all available nf-core pipelines along with their latest version, when that was published and how recently the pipeline code was pulled to your local system (if at all). +The command `nf-core list` shows all available nf-core pipelines along with their latest version, when that was published and how recently the pipeline code was pulled to your local system (if at all). An example of the output from the command is as follows: @@ -331,25 +332,25 @@ Do you want to run this command now? [y/n]: ### Launch tool options -* `-r`, `--revision` - * Specify a pipeline release (or branch / git commit sha) of the project to run -* `-i`, `--id` - * You can use the web GUI for nf-core pipelines by clicking _"Launch"_ on the website. Once filled in you will be given an ID to use with this command which is used to retrieve your inputs. -* `-c`, `--command-only` - * If you prefer not to save your inputs in a JSON file and use `-params-file`, this option will specify all entered params directly in the nextflow command. -* `-p`, `--params-in PATH` - * To use values entered in a previous pipeline run, you can supply the `nf-params.json` file previously generated. - * This will overwrite the pipeline schema defaults before the wizard is launched. -* `-o`, `--params-out PATH` - * Path to save parameters JSON file to. (Default: `nf-params.json`) -* `-a`, `--save-all` - * Without this option the pipeline will ignore any values that match the pipeline schema defaults. - * This option saves _all_ parameters found to the JSON file. -* `-h`, `--show-hidden` - * A pipeline JSON schema can define some parameters as 'hidden' if they are rarely used or for internal pipeline use only. - * This option forces the wizard to show all parameters, including those labelled as 'hidden'. -* `--url` - * Change the URL used for the graphical interface, useful for development work on the website. +- `-r`, `--revision` + - Specify a pipeline release (or branch / git commit sha) of the project to run +- `-i`, `--id` + - You can use the web GUI for nf-core pipelines by clicking _"Launch"_ on the website. Once filled in you will be given an ID to use with this command which is used to retrieve your inputs. +- `-c`, `--command-only` + - If you prefer not to save your inputs in a JSON file and use `-params-file`, this option will specify all entered params directly in the nextflow command. +- `-p`, `--params-in PATH` + - To use values entered in a previous pipeline run, you can supply the `nf-params.json` file previously generated. + - This will overwrite the pipeline schema defaults before the wizard is launched. +- `-o`, `--params-out PATH` + - Path to save parameters JSON file to. (Default: `nf-params.json`) +- `-a`, `--save-all` + - Without this option the pipeline will ignore any values that match the pipeline schema defaults. + - This option saves _all_ parameters found to the JSON file. +- `-h`, `--show-hidden` + - A pipeline JSON schema can define some parameters as 'hidden' if they are rarely used or for internal pipeline use only. + - This option forces the wizard to show all parameters, including those labelled as 'hidden'. +- `--url` + - Change the URL used for the graphical interface, useful for development work on the website. ## Downloading pipelines for offline use @@ -482,7 +483,7 @@ Once a full list of containers is found, they are processed in the following ord 2. If found in `$NXF_SINGULARITY_CACHEDIR` and `--singularity-cache-only` is _not_ specified, they are copied to the output directory 3. If they start with `http` they are downloaded directly within Python (default 4 at a time, you can customise this with `--parallel-downloads`) 4. If they look like a Docker image name, they are fetched using a `singularity pull` command - * This requires Singularity to be installed on the system and is substantially slower + - This requires Singularity to be installed on the system and is substantially slower Note that compressing many GBs of binary files can be slow, so specifying `--compress none` is recommended when downloading Singularity images. @@ -641,7 +642,7 @@ You can use the `-k` / `--key` flag to run only named tests for faster debugging ### Linting documentation Each test result name on the left is a terminal hyperlink. -In most terminals you can ctrl + click ( cmd + click) these +In most terminals you can ctrl + click ( cmd + click) these links to open documentation specific to this test in your browser. Alternatively visit and find your test to read more. @@ -656,8 +657,8 @@ Here you can list the names of any tests that you would like to disable and set ```yaml lint: - actions_awsfulltest: False - pipeline_todos: False + actions_awsfulltest: False + pipeline_todos: False ``` Some lint tests allow greater granularity, for example skipping a test only for a specific file. @@ -665,9 +666,9 @@ This is documented in the test-specific docs but generally involves passing a li ```yaml lint: - files_exist: + files_exist: - CODE_OF_CONDUCT.md - files_unchanged: + files_unchanged: - assets/email_template.html - CODE_OF_CONDUCT.md ``` @@ -693,9 +694,9 @@ Pipeline schema files are built according to the [JSONSchema specification](http To help developers working with pipeline schema, nf-core tools has three `schema` sub-commands: -* `nf-core schema validate` -* `nf-core schema build` -* `nf-core schema lint` +- `nf-core schema validate` +- `nf-core schema build` +- `nf-core schema lint` ### Validate pipeline parameters @@ -763,10 +764,10 @@ $ nf-core schema build nf-core-testpipeline There are four flags that you can use with this command: -* `--dir `: Specify a pipeline directory other than the current working directory -* `--no-prompts`: Make changes without prompting for confirmation each time. Does not launch web tool. -* `--web-only`: Skips comparison of the schema against the pipeline parameters and only launches the web tool. -* `--url `: Supply a custom URL for the online tool. Useful when testing locally. +- `--dir `: Specify a pipeline directory other than the current working directory +- `--no-prompts`: Make changes without prompting for confirmation each time. Does not launch web tool. +- `--web-only`: Skips comparison of the schema against the pipeline parameters and only launches the web tool. +- `--url `: Supply a custom URL for the online tool. Useful when testing locally. ### Linting a pipeline schema @@ -906,8 +907,8 @@ The nf-core DSL2 modules repository is at The modules supercommand comes with two flags for specifying a custom remote: -* `--github-repository `: Specify the repository from which the modules should be fetched. Defaults to `nf-core/modules`. -* `--branch `: Specify the branch from which the modules shoudl be fetched. Defaults to `master`. +- `--github-repository `: Specify the repository from which the modules should be fetched. Defaults to `nf-core/modules`. +- `--branch `: Specify the branch from which the modules shoudl be fetched. Defaults to `master`. Note that a custom remote must follow a similar directory structure to that of `nf-core/moduleś` for the `nf-core modules` commands to work properly. @@ -1072,9 +1073,9 @@ You can pass the module name as an optional argument to `nf-core modules install There are three additional flags that you can use when installing a module: -* `--force`: Overwrite a previously installed version of the module. -* `--prompt`: Select the module version using a cli prompt. -* `--sha `: Install the module at a specific commit from the `nf-core/modules` repository. +- `--force`: Overwrite a previously installed version of the module. +- `--prompt`: Select the module version using a cli prompt. +- `--sha `: Install the module at a specific commit from the `nf-core/modules` repository. ### Update modules in a pipeline @@ -1099,12 +1100,12 @@ You can pass the module name as an optional argument to `nf-core modules update` There are five additional flags that you can use with this command: -* `--force`: Reinstall module even if it appears to be up to date -* `--prompt`: Select the module version using a cli prompt. -* `--sha `: Install the module at a specific commit from the `nf-core/modules` repository. -* `--diff`: Show the diff between the installed files and the new version before installing. -* `--diff-file `: Specify where the diffs between the local and remote versions of a module should be written -* `--all`: Use this flag to run the command on all modules in the pipeline. +- `--force`: Reinstall module even if it appears to be up to date +- `--prompt`: Select the module version using a cli prompt. +- `--sha `: Install the module at a specific commit from the `nf-core/modules` repository. +- `--diff`: Show the diff between the installed files and the new version before installing. +- `--diff-file `: Specify where the diffs between the local and remote versions of a module should be written +- `--all`: Use this flag to run the command on all modules in the pipeline. If you don't want to update certain modules or want to update them to specific versions, you can make use of the `.nf-core.yml` configuration file. For example, you can prevent the `star/align` module installed from `nf-core/modules` from being updated by adding the following to the `.nf-core.yml` file: diff --git a/docs/api/_src/_static/css/custom.css b/docs/api/_src/_static/css/custom.css index c892c07aa6..e892dd9998 100644 --- a/docs/api/_src/_static/css/custom.css +++ b/docs/api/_src/_static/css/custom.css @@ -1,212 +1,437 @@ @media (prefers-color-scheme: light) { - a, a:visited { - color: #246EB9; - } - a:hover, a:focus, a:active { - color: #C03221; - } - .wy-nav-side{ - background-color: #ededed; - } - .wy-nav-top, .wy-side-nav-search, .wy-menu-vertical a:active { - background-color: #32AD65; - } - .wy-menu-vertical a{ - color: #343434; - } - .wy-menu-vertical a:hover{ - background-color: #abacab85; - } - .wy-menu-vertical header, .wy-menu-vertical p.caption { - color:#32AD65; - } - - .wy-side-nav-search input[type=text]{ - border: none; - } - - code, .rst-content code.literal{ - background-color: rgba(220,220,220,0.4); - color: #C03221; - border: none; - } - .rst-content .note .admonition-title{ - background-color: #72757bfc - } + a, + a:visited { + color: #246eb9; + } + a:hover, + a:focus, + a:active { + color: #c03221; + } + .wy-nav-side { + background-color: #ededed; + } + .wy-nav-top, + .wy-side-nav-search, + .wy-menu-vertical a:active { + background-color: #32ad65; + } + .wy-menu-vertical a { + color: #343434; + } + .wy-menu-vertical a:hover { + background-color: #abacab85; + } + .wy-menu-vertical header, + .wy-menu-vertical p.caption { + color: #32ad65; + } + + .wy-side-nav-search input[type="text"] { + border: none; + } + + code, + .rst-content code.literal { + background-color: rgba(220, 220, 220, 0.4); + color: #c03221; + border: none; + } + .rst-content .note .admonition-title { + background-color: #72757bfc; + } - html.writer-html4 .rst-content dl:not(.docutils)>dt, html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple)>dt { - background-color: #32AD65; - border:none; - } - .rst-content .method>dt>code { - color: #f5f6f7; - } - .rst-content div[class^=highlight], .rst-content pre.literal-block{ - border:none; - } - .highlight{ - background-color: #242424; - color: #e5e6e7; - } - .highlight .hll{ - color: #e5e6e7; - background-color: #585b60fc; - } - .highlight .k, .highlight .nt, .highlight .no{ - color: #246EB9; - } - .highlight .s, .highlight .s1, .highlight .s2, .highlight .na{ - color: #32AD65; - } - .highlight .nb, .highlight .o, .highlight .cm{ - color: #FFBE0B; - } - .highlight .c1{ - color: #88898afc; - } - .highlight .nv, .py>span{ - color: #C03221; - } - .py .sig-param, .py .sig-paren, .property .pre{ - color: #EEF0F2; - } + html.writer-html4 .rst-content dl:not(.docutils) > dt, + html.writer-html5 + .rst-content + dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) + > dt { + background-color: #32ad65; + border: none; + } + .rst-content .method > dt > code { + color: #f5f6f7; + } + .rst-content div[class^="highlight"], + .rst-content pre.literal-block { + border: none; + } + .highlight { + background-color: #242424; + color: #e5e6e7; + } + .highlight .hll { + color: #e5e6e7; + background-color: #585b60fc; + } + .highlight .k, + .highlight .nt, + .highlight .no { + color: #246eb9; + } + .highlight .s, + .highlight .s1, + .highlight .s2, + .highlight .na { + color: #32ad65; + } + .highlight .nb, + .highlight .o, + .highlight .cm { + color: #ffbe0b; + } + .highlight .c1 { + color: #88898afc; + } + .highlight .nv, + .py > span { + color: #c03221; + } + .py .sig-param, + .py .sig-paren, + .property .pre { + color: #eef0f2; + } - .method > .py .sig-param, .method > .py .sig-paren{ - color: #32AD65; - } - .btn.btn-neutral { - background-color: #e5e6e7d4 !important; - } - .rst-content .hint .admonition-title, .rst-content .hint .wy-alert-title, .rst-content .important .admonition-title, .rst-content .important .wy-alert-title, .rst-content .tip .admonition-title, .rst-content .tip .wy-alert-title, .rst-content .wy-alert-success.admonition-todo .admonition-title, .rst-content .wy-alert-success.admonition-todo .wy-alert-title, .rst-content .wy-alert-success.admonition .admonition-title, .rst-content .wy-alert-success.admonition .wy-alert-title, .rst-content .wy-alert-success.attention .admonition-title, .rst-content .wy-alert-success.attention .wy-alert-title, .rst-content .wy-alert-success.caution .admonition-title, .rst-content .wy-alert-success.caution .wy-alert-title, .rst-content .wy-alert-success.danger .admonition-title, .rst-content .wy-alert-success.danger .wy-alert-title, .rst-content .wy-alert-success.error .admonition-title, .rst-content .wy-alert-success.error .wy-alert-title, .rst-content .wy-alert-success.note .admonition-title, .rst-content .wy-alert-success.note .wy-alert-title, .rst-content .wy-alert-success.seealso .admonition-title, .rst-content .wy-alert-success.seealso .wy-alert-title, .rst-content .wy-alert-success.warning .admonition-title, .rst-content .wy-alert-success.warning .wy-alert-title, .rst-content .wy-alert.wy-alert-success .admonition-title, .wy-alert.wy-alert-success .rst-content .admonition-title, .wy-alert.wy-alert-success .wy-alert-title{ - background-color: #32AD65; - color: #e5e6e7; - } + .method > .py .sig-param, + .method > .py .sig-paren { + color: #32ad65; + } + .btn.btn-neutral { + background-color: #e5e6e7d4 !important; + } + .rst-content .hint .admonition-title, + .rst-content .hint .wy-alert-title, + .rst-content .important .admonition-title, + .rst-content .important .wy-alert-title, + .rst-content .tip .admonition-title, + .rst-content .tip .wy-alert-title, + .rst-content .wy-alert-success.admonition-todo .admonition-title, + .rst-content .wy-alert-success.admonition-todo .wy-alert-title, + .rst-content .wy-alert-success.admonition .admonition-title, + .rst-content .wy-alert-success.admonition .wy-alert-title, + .rst-content .wy-alert-success.attention .admonition-title, + .rst-content .wy-alert-success.attention .wy-alert-title, + .rst-content .wy-alert-success.caution .admonition-title, + .rst-content .wy-alert-success.caution .wy-alert-title, + .rst-content .wy-alert-success.danger .admonition-title, + .rst-content .wy-alert-success.danger .wy-alert-title, + .rst-content .wy-alert-success.error .admonition-title, + .rst-content .wy-alert-success.error .wy-alert-title, + .rst-content .wy-alert-success.note .admonition-title, + .rst-content .wy-alert-success.note .wy-alert-title, + .rst-content .wy-alert-success.seealso .admonition-title, + .rst-content .wy-alert-success.seealso .wy-alert-title, + .rst-content .wy-alert-success.warning .admonition-title, + .rst-content .wy-alert-success.warning .wy-alert-title, + .rst-content .wy-alert.wy-alert-success .admonition-title, + .wy-alert.wy-alert-success .rst-content .admonition-title, + .wy-alert.wy-alert-success .wy-alert-title { + background-color: #32ad65; + color: #e5e6e7; + } - .rst-content .note, .rst-content .seealso, .rst-content .wy-alert-info.admonition, .rst-content .wy-alert-info.admonition-todo, .rst-content .wy-alert-info.attention, .rst-content .wy-alert-info.caution, .rst-content .wy-alert-info.danger, .rst-content .wy-alert-info.error, .rst-content .wy-alert-info.hint, .rst-content .wy-alert-info.important, .rst-content .wy-alert-info.tip, .rst-content .wy-alert-info.warning, .wy-alert.wy-alert-info, - .rst-content .hint, .rst-content .important, .rst-content .tip, .rst-content .wy-alert-success.admonition, .rst-content .wy-alert-success.admonition-todo, .rst-content .wy-alert-success.attention, .rst-content .wy-alert-success.caution, .rst-content .wy-alert-success.danger, .rst-content .wy-alert-success.error, .rst-content .wy-alert-success.note, .rst-content .wy-alert-success.seealso, .rst-content .wy-alert-success.warning, .wy-alert.wy-alert-success{ - color: #343434; - background-color: #e3e3e3; - border: none; - } + .rst-content .note, + .rst-content .seealso, + .rst-content .wy-alert-info.admonition, + .rst-content .wy-alert-info.admonition-todo, + .rst-content .wy-alert-info.attention, + .rst-content .wy-alert-info.caution, + .rst-content .wy-alert-info.danger, + .rst-content .wy-alert-info.error, + .rst-content .wy-alert-info.hint, + .rst-content .wy-alert-info.important, + .rst-content .wy-alert-info.tip, + .rst-content .wy-alert-info.warning, + .wy-alert.wy-alert-info, + .rst-content .hint, + .rst-content .important, + .rst-content .tip, + .rst-content .wy-alert-success.admonition, + .rst-content .wy-alert-success.admonition-todo, + .rst-content .wy-alert-success.attention, + .rst-content .wy-alert-success.caution, + .rst-content .wy-alert-success.danger, + .rst-content .wy-alert-success.error, + .rst-content .wy-alert-success.note, + .rst-content .wy-alert-success.seealso, + .rst-content .wy-alert-success.warning, + .wy-alert.wy-alert-success { + color: #343434; + background-color: #e3e3e3; + border: none; + } - -.rst-content .admonition-todo .admonition-title, .rst-content .admonition-todo .wy-alert-title, .rst-content .attention .admonition-title, .rst-content .attention .wy-alert-title, .rst-content .caution .admonition-title, .rst-content .caution .wy-alert-title, .rst-content .warning .admonition-title, .rst-content .warning .wy-alert-title, .rst-content .wy-alert-warning.admonition .admonition-title, .rst-content .wy-alert-warning.admonition .wy-alert-title, .rst-content .wy-alert-warning.danger .admonition-title, .rst-content .wy-alert-warning.danger .wy-alert-title, .rst-content .wy-alert-warning.error .admonition-title, .rst-content .wy-alert-warning.error .wy-alert-title, .rst-content .wy-alert-warning.hint .admonition-title, .rst-content .wy-alert-warning.hint .wy-alert-title, .rst-content .seealso .admonition-title, .rst-content .seealso .wy-alert-title, .rst-content .wy-alert-warning.important .admonition-title, .rst-content .wy-alert-warning.important .wy-alert-title, .rst-content .wy-alert-warning.note .admonition-title, .rst-content .wy-alert-warning.note .wy-alert-title, .rst-content .wy-alert-warning.seealso .admonition-title, .rst-content .wy-alert-warning.seealso .wy-alert-title, .rst-content .wy-alert-warning.tip .admonition-title, .rst-content .wy-alert-warning.tip .wy-alert-title, .rst-content .wy-alert.wy-alert-warning .admonition-title, .wy-alert.wy-alert-warning .rst-content .admonition-title, .wy-alert.wy-alert-warning .wy-alert-title { - background-color: #246EB9; - } -.rst-content .admonition-todo, .rst-content .attention, .rst-content .caution, .rst-content .warning, .rst-content .wy-alert-warning.admonition, .rst-content .wy-alert-warning.danger, .rst-content .wy-alert-warning.error, .rst-content .wy-alert-warning.hint, .rst-content .wy-alert-warning.important, .rst-content .wy-alert-warning.note, .rst-content .wy-alert-warning.seealso, .rst-content .wy-alert-warning.tip, .wy-alert.wy-alert-warning { + .rst-content .admonition-todo .admonition-title, + .rst-content .admonition-todo .wy-alert-title, + .rst-content .attention .admonition-title, + .rst-content .attention .wy-alert-title, + .rst-content .caution .admonition-title, + .rst-content .caution .wy-alert-title, + .rst-content .warning .admonition-title, + .rst-content .warning .wy-alert-title, + .rst-content .wy-alert-warning.admonition .admonition-title, + .rst-content .wy-alert-warning.admonition .wy-alert-title, + .rst-content .wy-alert-warning.danger .admonition-title, + .rst-content .wy-alert-warning.danger .wy-alert-title, + .rst-content .wy-alert-warning.error .admonition-title, + .rst-content .wy-alert-warning.error .wy-alert-title, + .rst-content .wy-alert-warning.hint .admonition-title, + .rst-content .wy-alert-warning.hint .wy-alert-title, + .rst-content .seealso .admonition-title, + .rst-content .seealso .wy-alert-title, + .rst-content .wy-alert-warning.important .admonition-title, + .rst-content .wy-alert-warning.important .wy-alert-title, + .rst-content .wy-alert-warning.note .admonition-title, + .rst-content .wy-alert-warning.note .wy-alert-title, + .rst-content .wy-alert-warning.seealso .admonition-title, + .rst-content .wy-alert-warning.seealso .wy-alert-title, + .rst-content .wy-alert-warning.tip .admonition-title, + .rst-content .wy-alert-warning.tip .wy-alert-title, + .rst-content .wy-alert.wy-alert-warning .admonition-title, + .wy-alert.wy-alert-warning .rst-content .admonition-title, + .wy-alert.wy-alert-warning .wy-alert-title { + background-color: #246eb9; + } + .rst-content .admonition-todo, + .rst-content .attention, + .rst-content .caution, + .rst-content .warning, + .rst-content .wy-alert-warning.admonition, + .rst-content .wy-alert-warning.danger, + .rst-content .wy-alert-warning.error, + .rst-content .wy-alert-warning.hint, + .rst-content .wy-alert-warning.important, + .rst-content .wy-alert-warning.note, + .rst-content .wy-alert-warning.seealso, + .rst-content .wy-alert-warning.tip, + .wy-alert.wy-alert-warning { color: #343434; - background-color: #e3e3e3; - } - - + background-color: #e3e3e3; + } } @media (prefers-color-scheme: dark) { - .wy-nav-content-wrap { - background-color: #181a1b - } - .wy-nav-top, .wy-side-nav-search, .wy-menu-vertical a:active { - background-color: #32AD65; - } - .wy-menu-vertical header, .wy-menu-vertical p.caption { - color:#32AD65; - } - .wy-nav-side{ - background-color: #2E2E2E; - } - .wy-nav-content{ - background-color: #343434; - color: #e5e6e7; - } - a, a:visited { - color: #6fb2e8; - } - a:hover, a:focus, a:active { - color: #db9444; - } - code, .rst-content code.literal{ - background-color: rgba(220,220,220,0.1); - color: #db9444; - border: none; - } - .wy-side-nav-search input[type=text]{ - border: none; - background-color: #e5e6e7; - } - .wy-side-nav-search>div.version{ - color: #e5e6e7c9; - } - .wy-side-nav-search .wy-dropdown>a, .wy-side-nav-search>a { - color: #e5e6e7; - } - - html.writer-html4 .rst-content dl:not(.docutils)>dt, html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple)>dt { - background-color: #32AD65; - border:none; - } - html.writer-html4 .rst-content dl:not(.docutils) dl:not(.field-list)>dt, html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) dl:not(.field-list)>dt{ - background-color:#777; - color:#e5e6e7a6; - } - .rst-content code { - color: #e5e6e7d4; - } - .rst-content .method>dt>code { - color: #e5e6e7; - } - .sig-paren{ - color: #db9444; - } - .rst-content .note .admonition-title{ - background-color: #72757bfc - } - .rst-content div[class^=highlight], .rst-content pre.literal-block{ - border:none; - } + .wy-nav-content-wrap { + background-color: #181a1b; + } + .wy-nav-top, + .wy-side-nav-search, + .wy-menu-vertical a:active { + background-color: #32ad65; + } + .wy-menu-vertical header, + .wy-menu-vertical p.caption { + color: #32ad65; + } + .wy-nav-side { + background-color: #2e2e2e; + } + .wy-nav-content { + background-color: #343434; + color: #e5e6e7; + } + a, + a:visited { + color: #6fb2e8; + } + a:hover, + a:focus, + a:active { + color: #db9444; + } + code, + .rst-content code.literal { + background-color: rgba(220, 220, 220, 0.1); + color: #db9444; + border: none; + } + .wy-side-nav-search input[type="text"] { + border: none; + background-color: #e5e6e7; + } + .wy-side-nav-search > div.version { + color: #e5e6e7c9; + } + .wy-side-nav-search .wy-dropdown > a, + .wy-side-nav-search > a { + color: #e5e6e7; + } + + html.writer-html4 .rst-content dl:not(.docutils) > dt, + html.writer-html5 + .rst-content + dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) + > dt { + background-color: #32ad65; + border: none; + } + html.writer-html4 .rst-content dl:not(.docutils) dl:not(.field-list) > dt, + html.writer-html5 + .rst-content + dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) + dl:not(.field-list) + > dt { + background-color: #777; + color: #e5e6e7a6; + } + .rst-content code { + color: #e5e6e7d4; + } + .rst-content .method > dt > code { + color: #e5e6e7; + } + .sig-paren { + color: #db9444; + } + .rst-content .note .admonition-title { + background-color: #72757bfc; + } + .rst-content div[class^="highlight"], + .rst-content pre.literal-block { + border: none; + } - .highlight{ - background-color: #242424; - color: #e5e6e7; - } - .highlight .hll{ - color: #e5e6e7; - background-color: #585b60fc; - } - .highlight .k, .highlight .nt, .highlight .no{ - color: #6fb2e8; - } - .highlight .s, .highlight .s1, .highlight .s2, .highlight .na{ - color: #32AD65; - } - .highlight .nb, .highlight .o, .highlight .cm{ - color: #db9444; - } - .highlight .c1{ - color: #88898afc; - } - .highlight .nv{ - color: #f9d977; - } - .btn.btn-neutral { - background-color: #e5e6e7d4 !important; - } + .highlight { + background-color: #242424; + color: #e5e6e7; + } + .highlight .hll { + color: #e5e6e7; + background-color: #585b60fc; + } + .highlight .k, + .highlight .nt, + .highlight .no { + color: #6fb2e8; + } + .highlight .s, + .highlight .s1, + .highlight .s2, + .highlight .na { + color: #32ad65; + } + .highlight .nb, + .highlight .o, + .highlight .cm { + color: #db9444; + } + .highlight .c1 { + color: #88898afc; + } + .highlight .nv { + color: #f9d977; + } + .btn.btn-neutral { + background-color: #e5e6e7d4 !important; + } - .rst-content .hint .admonition-title, .rst-content .hint .wy-alert-title, .rst-content .important .admonition-title, .rst-content .important .wy-alert-title, .rst-content .tip .admonition-title, .rst-content .tip .wy-alert-title, .rst-content .wy-alert-success.admonition-todo .admonition-title, .rst-content .wy-alert-success.admonition-todo .wy-alert-title, .rst-content .wy-alert-success.admonition .admonition-title, .rst-content .wy-alert-success.admonition .wy-alert-title, .rst-content .wy-alert-success.attention .admonition-title, .rst-content .wy-alert-success.attention .wy-alert-title, .rst-content .wy-alert-success.caution .admonition-title, .rst-content .wy-alert-success.caution .wy-alert-title, .rst-content .wy-alert-success.danger .admonition-title, .rst-content .wy-alert-success.danger .wy-alert-title, .rst-content .wy-alert-success.error .admonition-title, .rst-content .wy-alert-success.error .wy-alert-title, .rst-content .wy-alert-success.note .admonition-title, .rst-content .wy-alert-success.note .wy-alert-title, .rst-content .wy-alert-success.seealso .admonition-title, .rst-content .wy-alert-success.seealso .wy-alert-title, .rst-content .wy-alert-success.warning .admonition-title, .rst-content .wy-alert-success.warning .wy-alert-title, .rst-content .wy-alert.wy-alert-success .admonition-title, .wy-alert.wy-alert-success .rst-content .admonition-title, .wy-alert.wy-alert-success .wy-alert-title{ - background-color: #32AD65; - color: #e5e6e7; - } + .rst-content .hint .admonition-title, + .rst-content .hint .wy-alert-title, + .rst-content .important .admonition-title, + .rst-content .important .wy-alert-title, + .rst-content .tip .admonition-title, + .rst-content .tip .wy-alert-title, + .rst-content .wy-alert-success.admonition-todo .admonition-title, + .rst-content .wy-alert-success.admonition-todo .wy-alert-title, + .rst-content .wy-alert-success.admonition .admonition-title, + .rst-content .wy-alert-success.admonition .wy-alert-title, + .rst-content .wy-alert-success.attention .admonition-title, + .rst-content .wy-alert-success.attention .wy-alert-title, + .rst-content .wy-alert-success.caution .admonition-title, + .rst-content .wy-alert-success.caution .wy-alert-title, + .rst-content .wy-alert-success.danger .admonition-title, + .rst-content .wy-alert-success.danger .wy-alert-title, + .rst-content .wy-alert-success.error .admonition-title, + .rst-content .wy-alert-success.error .wy-alert-title, + .rst-content .wy-alert-success.note .admonition-title, + .rst-content .wy-alert-success.note .wy-alert-title, + .rst-content .wy-alert-success.seealso .admonition-title, + .rst-content .wy-alert-success.seealso .wy-alert-title, + .rst-content .wy-alert-success.warning .admonition-title, + .rst-content .wy-alert-success.warning .wy-alert-title, + .rst-content .wy-alert.wy-alert-success .admonition-title, + .wy-alert.wy-alert-success .rst-content .admonition-title, + .wy-alert.wy-alert-success .wy-alert-title { + background-color: #32ad65; + color: #e5e6e7; + } - .rst-content .note, .rst-content .seealso, .rst-content .wy-alert-info.admonition, .rst-content .wy-alert-info.admonition-todo, .rst-content .wy-alert-info.attention, .rst-content .wy-alert-info.caution, .rst-content .wy-alert-info.danger, .rst-content .wy-alert-info.error, .rst-content .wy-alert-info.hint, .rst-content .wy-alert-info.important, .rst-content .wy-alert-info.tip, .rst-content .wy-alert-info.warning, .wy-alert.wy-alert-info, - .rst-content .hint, .rst-content .important, .rst-content .tip, .rst-content .wy-alert-success.admonition, .rst-content .wy-alert-success.admonition-todo, .rst-content .wy-alert-success.attention, .rst-content .wy-alert-success.caution, .rst-content .wy-alert-success.danger, .rst-content .wy-alert-success.error, .rst-content .wy-alert-success.note, .rst-content .wy-alert-success.seealso, .rst-content .wy-alert-success.warning, .wy-alert.wy-alert-success{ - color: #343434; - background-color: #e3e3e3; - border: none; - } + .rst-content .note, + .rst-content .seealso, + .rst-content .wy-alert-info.admonition, + .rst-content .wy-alert-info.admonition-todo, + .rst-content .wy-alert-info.attention, + .rst-content .wy-alert-info.caution, + .rst-content .wy-alert-info.danger, + .rst-content .wy-alert-info.error, + .rst-content .wy-alert-info.hint, + .rst-content .wy-alert-info.important, + .rst-content .wy-alert-info.tip, + .rst-content .wy-alert-info.warning, + .wy-alert.wy-alert-info, + .rst-content .hint, + .rst-content .important, + .rst-content .tip, + .rst-content .wy-alert-success.admonition, + .rst-content .wy-alert-success.admonition-todo, + .rst-content .wy-alert-success.attention, + .rst-content .wy-alert-success.caution, + .rst-content .wy-alert-success.danger, + .rst-content .wy-alert-success.error, + .rst-content .wy-alert-success.note, + .rst-content .wy-alert-success.seealso, + .rst-content .wy-alert-success.warning, + .wy-alert.wy-alert-success { + color: #343434; + background-color: #e3e3e3; + border: none; + } - .rst-content .admonition-todo .admonition-title, .rst-content .admonition-todo .wy-alert-title, .rst-content .attention .admonition-title, .rst-content .attention .wy-alert-title, .rst-content .caution .admonition-title, .rst-content .caution .wy-alert-title, .rst-content .warning .admonition-title, .rst-content .warning .wy-alert-title, .rst-content .wy-alert-warning.admonition .admonition-title, .rst-content .wy-alert-warning.admonition .wy-alert-title, .rst-content .wy-alert-warning.danger .admonition-title, .rst-content .wy-alert-warning.danger .wy-alert-title, .rst-content .wy-alert-warning.error .admonition-title, .rst-content .wy-alert-warning.error .wy-alert-title, .rst-content .wy-alert-warning.hint .admonition-title, .rst-content .wy-alert-warning.hint .wy-alert-title, .rst-content .seealso .admonition-title, .rst-content .seealso .wy-alert-title, .rst-content .wy-alert-warning.important .admonition-title, .rst-content .wy-alert-warning.important .wy-alert-title, .rst-content .wy-alert-warning.note .admonition-title, .rst-content .wy-alert-warning.note .wy-alert-title, .rst-content .wy-alert-warning.seealso .admonition-title, .rst-content .wy-alert-warning.seealso .wy-alert-title, .rst-content .wy-alert-warning.tip .admonition-title, .rst-content .wy-alert-warning.tip .wy-alert-title, .rst-content .wy-alert.wy-alert-warning .admonition-title, .wy-alert.wy-alert-warning .rst-content .admonition-title, .wy-alert.wy-alert-warning .wy-alert-title { - background-color: #f4a25b; - } - .rst-content .admonition-todo, .rst-content .attention, .rst-content .caution, .rst-content .warning, .rst-content .wy-alert-warning.admonition, .rst-content .wy-alert-warning.danger, .rst-content .wy-alert-warning.error, .rst-content .wy-alert-warning.hint, .rst-content .wy-alert-warning.important, .rst-content .wy-alert-warning.note, .rst-content .wy-alert-warning.seealso, .rst-content .wy-alert-warning.tip, .wy-alert.wy-alert-warning { + .rst-content .admonition-todo .admonition-title, + .rst-content .admonition-todo .wy-alert-title, + .rst-content .attention .admonition-title, + .rst-content .attention .wy-alert-title, + .rst-content .caution .admonition-title, + .rst-content .caution .wy-alert-title, + .rst-content .warning .admonition-title, + .rst-content .warning .wy-alert-title, + .rst-content .wy-alert-warning.admonition .admonition-title, + .rst-content .wy-alert-warning.admonition .wy-alert-title, + .rst-content .wy-alert-warning.danger .admonition-title, + .rst-content .wy-alert-warning.danger .wy-alert-title, + .rst-content .wy-alert-warning.error .admonition-title, + .rst-content .wy-alert-warning.error .wy-alert-title, + .rst-content .wy-alert-warning.hint .admonition-title, + .rst-content .wy-alert-warning.hint .wy-alert-title, + .rst-content .seealso .admonition-title, + .rst-content .seealso .wy-alert-title, + .rst-content .wy-alert-warning.important .admonition-title, + .rst-content .wy-alert-warning.important .wy-alert-title, + .rst-content .wy-alert-warning.note .admonition-title, + .rst-content .wy-alert-warning.note .wy-alert-title, + .rst-content .wy-alert-warning.seealso .admonition-title, + .rst-content .wy-alert-warning.seealso .wy-alert-title, + .rst-content .wy-alert-warning.tip .admonition-title, + .rst-content .wy-alert-warning.tip .wy-alert-title, + .rst-content .wy-alert.wy-alert-warning .admonition-title, + .wy-alert.wy-alert-warning .rst-content .admonition-title, + .wy-alert.wy-alert-warning .wy-alert-title { + background-color: #f4a25b; + } + .rst-content .admonition-todo, + .rst-content .attention, + .rst-content .caution, + .rst-content .warning, + .rst-content .wy-alert-warning.admonition, + .rst-content .wy-alert-warning.danger, + .rst-content .wy-alert-warning.error, + .rst-content .wy-alert-warning.hint, + .rst-content .wy-alert-warning.important, + .rst-content .wy-alert-warning.note, + .rst-content .wy-alert-warning.seealso, + .rst-content .wy-alert-warning.tip, + .wy-alert.wy-alert-warning { color: #343434; - background-color: #e3e3e3; - } + background-color: #e3e3e3; + } } diff --git a/docs/api/_src/_templates/layout.html b/docs/api/_src/_templates/layout.html index 35ead5f53f..42341a97c1 100644 --- a/docs/api/_src/_templates/layout.html +++ b/docs/api/_src/_templates/layout.html @@ -1,6 +1,4 @@ -{% extends '!layout.html' %} - -{% block header %} +{% extends '!layout.html' %} {% block header %} - -{% endblock %} \ No newline at end of file + +{% endblock %} From 0786955108777fb7f216352f535f528565ffd026 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 17:13:12 +0100 Subject: [PATCH 055/110] Nicer workflow name --- .github/workflows/create-lint-wf.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/create-lint-wf.yml b/.github/workflows/create-lint-wf.yml index 28347d1b8a..e4ace9ce9e 100644 --- a/.github/workflows/create-lint-wf.yml +++ b/.github/workflows/create-lint-wf.yml @@ -1,4 +1,4 @@ -name: Create a pipeline and run `nf-core lint`` +name: Create a pipeline and run nf-core linting on: [push, pull_request] env: From 81bf944f29ec6e6a66de4874470e84afe466ac19 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 19:10:48 +0100 Subject: [PATCH 056/110] Prettier: ignore testing directory --- .prettierignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.prettierignore b/.prettierignore index 26b9d5cd73..1057f5429e 100644 --- a/.prettierignore +++ b/.prettierignore @@ -1,4 +1,5 @@ docs/api/_build +testing # Skip the templates as the Jinja2 tags confuse prettier # Don't worry - we run prettier on the compiled pipeline in create-lint-wf.yml From 756ffebf02c04881695bd553f5b939d9dc1b1913 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 19:17:36 +0100 Subject: [PATCH 057/110] Run new compiled pipeline linting commands in the correct directory --- .github/workflows/create-lint-wf.yml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/.github/workflows/create-lint-wf.yml b/.github/workflows/create-lint-wf.yml index e4ace9ce9e..2296242291 100644 --- a/.github/workflows/create-lint-wf.yml +++ b/.github/workflows/create-lint-wf.yml @@ -59,17 +59,19 @@ jobs: # Build a module from the template - name: nf-core modules create - run: nf-core --log-file log.txt modules create bpipe --author @nf-core-bot --label process_low --meta + run: nf-core --log-file log.txt modules create bpipe --dir nf-core-testpipeline --author @nf-core-bot --label process_low --meta # Run all the linters - name: nf-core lint run: nf-core --log-file log.txt lint --dir nf-core-testpipeline --fail-ignored - name: Run Prettier --check - run: prettier --check ${GITHUB_WORKSPACE} + run: prettier --check nf-core-testpipeline - name: Check code lints with Black uses: psf/black@stable + with: + src: "./nf-core-testpipeline" # Run the other nf-core commands - name: nf-core list From 847ca2a0215884970a84e9437d4669e1a61e0df3 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 21:05:59 +0100 Subject: [PATCH 058/110] Add .prettierrc.yaml to the pipeline-template --- .prettierrc.yaml | 1 + nf_core/pipeline-template/.prettierrc.yaml | 2 ++ 2 files changed, 3 insertions(+) create mode 100644 nf_core/pipeline-template/.prettierrc.yaml diff --git a/.prettierrc.yaml b/.prettierrc.yaml index c81f9a7660..ef9b1b85a7 100644 --- a/.prettierrc.yaml +++ b/.prettierrc.yaml @@ -1 +1,2 @@ printWidth: 120 +tabWidth: 2 diff --git a/nf_core/pipeline-template/.prettierrc.yaml b/nf_core/pipeline-template/.prettierrc.yaml new file mode 100644 index 0000000000..ef9b1b85a7 --- /dev/null +++ b/nf_core/pipeline-template/.prettierrc.yaml @@ -0,0 +1,2 @@ +printWidth: 120 +tabWidth: 2 From 23d33ca52194a8a449626136e8b84dce93c299c2 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 21:09:47 +0100 Subject: [PATCH 059/110] Remove yamllint and markdownlint configs from the pipeline template --- nf_core/pipeline-template/.markdownlint.yml | 14 -------------- nf_core/pipeline-template/.yamllint.yml | 6 ------ 2 files changed, 20 deletions(-) delete mode 100644 nf_core/pipeline-template/.markdownlint.yml delete mode 100644 nf_core/pipeline-template/.yamllint.yml diff --git a/nf_core/pipeline-template/.markdownlint.yml b/nf_core/pipeline-template/.markdownlint.yml deleted file mode 100644 index 9e605fcfab..0000000000 --- a/nf_core/pipeline-template/.markdownlint.yml +++ /dev/null @@ -1,14 +0,0 @@ -# Markdownlint configuration file -default: true -line-length: false -ul-indent: - indent: 4 -no-duplicate-header: - siblings_only: true -no-inline-html: - allowed_elements: - - img - - p - - kbd - - details - - summary diff --git a/nf_core/pipeline-template/.yamllint.yml b/nf_core/pipeline-template/.yamllint.yml deleted file mode 100644 index d466deec92..0000000000 --- a/nf_core/pipeline-template/.yamllint.yml +++ /dev/null @@ -1,6 +0,0 @@ -extends: default - -rules: - document-start: disable - line-length: disable - truthy: disable From 73f4ff2f906eb499a4d5d2469ff5e898bb82d958 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 21:10:00 +0100 Subject: [PATCH 060/110] Run prettier on the pipeline template folder --- .../pipeline-template/.github/CONTRIBUTING.md | 12 +- .../.github/ISSUE_TEMPLATE/bug_report.yml | 1 - .../.github/PULL_REQUEST_TEMPLATE.md | 4 +- nf_core/pipeline-template/.gitpod.yml | 18 +- nf_core/pipeline-template/CITATIONS.md | 27 +- nf_core/pipeline-template/README.md | 29 +- .../assets/email_template.html | 141 +++-- .../assets/schema_input.json | 69 ++- nf_core/pipeline-template/docs/README.md | 8 +- nf_core/pipeline-template/docs/output.md | 28 +- nf_core/pipeline-template/docs/usage.md | 98 ++-- nf_core/pipeline-template/modules.json | 30 +- .../custom/dumpsoftwareversions/meta.yml | 2 +- .../pipeline-template/nextflow_schema.json | 522 +++++++++--------- 14 files changed, 519 insertions(+), 470 deletions(-) diff --git a/nf_core/pipeline-template/.github/CONTRIBUTING.md b/nf_core/pipeline-template/.github/CONTRIBUTING.md index edc89666ab..2524037294 100644 --- a/nf_core/pipeline-template/.github/CONTRIBUTING.md +++ b/nf_core/pipeline-template/.github/CONTRIBUTING.md @@ -16,7 +16,7 @@ Contributions to the code are even more welcome ;) If you'd like to write some code for {{ name }}, the standard workflow is as follows: 1. Check that there isn't already an issue about your idea in the [{{ name }} issues](https://github.com/{{ name }}/issues) to avoid duplicating work - * If there isn't one already, please create one so that others know you're working on this + - If there isn't one already, please create one so that others know you're working on this 2. [Fork](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) the [{{ name }} repository](https://github.com/{{ name }}) to your GitHub account 3. Make the necessary changes / additions within your forked repository following [Pipeline conventions](#pipeline-contribution-conventions) 4. Use `nf-core schema build` and add any new parameters to the pipeline JSON schema (requires [nf-core tools](https://github.com/nf-core/tools) >= 1.10). @@ -49,9 +49,9 @@ These tests are run both with the latest available version of `Nextflow` and als :warning: Only in the unlikely and regretful event of a release happening with a bug. -* On your own fork, make a new branch `patch` based on `upstream/master`. -* Fix the bug, and bump version (X.Y.Z+1). -* A PR should be made on `master` from patch to directly this particular bug. +- On your own fork, make a new branch `patch` based on `upstream/master`. +- Fix the bug, and bump version (X.Y.Z+1). +- A PR should be made on `master` from patch to directly this particular bug. ## Getting help @@ -92,8 +92,8 @@ The process resources can be passed on to the tool dynamically within the proces Please use the following naming schemes, to make it easy to understand what is going where. -* initial process channel: `ch_output_from_` -* intermediate and terminal channels: `ch__for_` +- initial process channel: `ch_output_from_` +- intermediate and terminal channels: `ch__for_` ### Nextflow version bumping diff --git a/nf_core/pipeline-template/.github/ISSUE_TEMPLATE/bug_report.yml b/nf_core/pipeline-template/.github/ISSUE_TEMPLATE/bug_report.yml index a9ab8e4513..209c6e2d9f 100644 --- a/nf_core/pipeline-template/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/nf_core/pipeline-template/.github/ISSUE_TEMPLATE/bug_report.yml @@ -2,7 +2,6 @@ name: Bug report description: Report something that is broken or incorrect labels: bug body: - - type: markdown attributes: value: | diff --git a/nf_core/pipeline-template/.github/PULL_REQUEST_TEMPLATE.md b/nf_core/pipeline-template/.github/PULL_REQUEST_TEMPLATE.md index 1ef9d3eaad..3047cba874 100644 --- a/nf_core/pipeline-template/.github/PULL_REQUEST_TEMPLATE.md +++ b/nf_core/pipeline-template/.github/PULL_REQUEST_TEMPLATE.md @@ -16,8 +16,8 @@ Learn more about contributing: [CONTRIBUTING.md](https://github.com/{{ name }}/t - [ ] This comment contains a description of changes (with reason). - [ ] If you've fixed a bug or added code that should be tested, add tests! - - [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/{{ name }}/tree/master/.github/CONTRIBUTING.md) - - [ ] If necessary, also make a PR on the {{ name }} _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository. + - [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/{{ name }}/tree/master/.github/CONTRIBUTING.md) + - [ ] If necessary, also make a PR on the {{ name }} _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository. - [ ] Make sure your code lints (`nf-core lint`). - [ ] Ensure the test suite passes (`nextflow run . -profile test,docker --outdir `). - [ ] Usage Documentation in `docs/usage.md` is updated. diff --git a/nf_core/pipeline-template/.gitpod.yml b/nf_core/pipeline-template/.gitpod.yml index 2b9ef8bf95..c452ee93e5 100644 --- a/nf_core/pipeline-template/.gitpod.yml +++ b/nf_core/pipeline-template/.gitpod.yml @@ -1,14 +1,14 @@ image: nfcore/gitpod:latest vscode: - extensions: # based on nf-core.nf-core-extensionpack - - codezombiech.gitignore # Language support for .gitignore files + extensions: # based on nf-core.nf-core-extensionpack + - codezombiech.gitignore # Language support for .gitignore files # - cssho.vscode-svgviewer # SVG viewer - - davidanson.vscode-markdownlint # Markdown/CommonMark linting and style checking for Visual Studio Code - - eamodio.gitlens # Quickly glimpse into whom, why, and when a line or code block was changed - - EditorConfig.EditorConfig # override user/workspace settings with settings found in .editorconfig files - - Gruntfuggly.todo-tree # Display TODO and FIXME in a tree view in the activity bar - - mechatroner.rainbow-csv # Highlight columns in csv files in different colors + - davidanson.vscode-markdownlint # Markdown/CommonMark linting and style checking for Visual Studio Code + - eamodio.gitlens # Quickly glimpse into whom, why, and when a line or code block was changed + - EditorConfig.EditorConfig # override user/workspace settings with settings found in .editorconfig files + - Gruntfuggly.todo-tree # Display TODO and FIXME in a tree view in the activity bar + - mechatroner.rainbow-csv # Highlight columns in csv files in different colors # - nextflow.nextflow # Nextflow syntax highlighting - - oderwat.indent-rainbow # Highlight indentation level - - streetsidesoftware.code-spell-checker # Spelling checker for source code + - oderwat.indent-rainbow # Highlight indentation level + - streetsidesoftware.code-spell-checker # Spelling checker for source code diff --git a/nf_core/pipeline-template/CITATIONS.md b/nf_core/pipeline-template/CITATIONS.md index 323c681a0a..740b045103 100644 --- a/nf_core/pipeline-template/CITATIONS.md +++ b/nf_core/pipeline-template/CITATIONS.md @@ -10,23 +10,26 @@ ## Pipeline tools -* [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) +- [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) -* [MultiQC](https://pubmed.ncbi.nlm.nih.gov/27312411/) - > Ewels P, Magnusson M, Lundin S, Käller M. MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics. 2016 Oct 1;32(19):3047-8. doi: 10.1093/bioinformatics/btw354. Epub 2016 Jun 16. PubMed PMID: 27312411; PubMed Central PMCID: PMC5039924. +- [MultiQC](https://pubmed.ncbi.nlm.nih.gov/27312411/) + > Ewels P, Magnusson M, Lundin S, Käller M. MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics. 2016 Oct 1;32(19):3047-8. doi: 10.1093/bioinformatics/btw354. Epub 2016 Jun 16. PubMed PMID: 27312411; PubMed Central PMCID: PMC5039924. ## Software packaging/containerisation tools -* [Anaconda](https://anaconda.com) - > Anaconda Software Distribution. Computer software. Vers. 2-2.4.0. Anaconda, Nov. 2016. Web. +- [Anaconda](https://anaconda.com) -* [Bioconda](https://pubmed.ncbi.nlm.nih.gov/29967506/) - > Grüning B, Dale R, Sjödin A, Chapman BA, Rowe J, Tomkins-Tinch CH, Valieris R, Köster J; Bioconda Team. Bioconda: sustainable and comprehensive software distribution for the life sciences. Nat Methods. 2018 Jul;15(7):475-476. doi: 10.1038/s41592-018-0046-7. PubMed PMID: 29967506. + > Anaconda Software Distribution. Computer software. Vers. 2-2.4.0. Anaconda, Nov. 2016. Web. -* [BioContainers](https://pubmed.ncbi.nlm.nih.gov/28379341/) - > da Veiga Leprevost F, Grüning B, Aflitos SA, Röst HL, Uszkoreit J, Barsnes H, Vaudel M, Moreno P, Gatto L, Weber J, Bai M, Jimenez RC, Sachsenberg T, Pfeuffer J, Alvarez RV, Griss J, Nesvizhskii AI, Perez-Riverol Y. BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics. 2017 Aug 15;33(16):2580-2582. doi: 10.1093/bioinformatics/btx192. PubMed PMID: 28379341; PubMed Central PMCID: PMC5870671. +- [Bioconda](https://pubmed.ncbi.nlm.nih.gov/29967506/) -* [Docker](https://dl.acm.org/doi/10.5555/2600239.2600241) + > Grüning B, Dale R, Sjödin A, Chapman BA, Rowe J, Tomkins-Tinch CH, Valieris R, Köster J; Bioconda Team. Bioconda: sustainable and comprehensive software distribution for the life sciences. Nat Methods. 2018 Jul;15(7):475-476. doi: 10.1038/s41592-018-0046-7. PubMed PMID: 29967506. -* [Singularity](https://pubmed.ncbi.nlm.nih.gov/28494014/) - > Kurtzer GM, Sochat V, Bauer MW. Singularity: Scientific containers for mobility of compute. PLoS One. 2017 May 11;12(5):e0177459. doi: 10.1371/journal.pone.0177459. eCollection 2017. PubMed PMID: 28494014; PubMed Central PMCID: PMC5426675. +- [BioContainers](https://pubmed.ncbi.nlm.nih.gov/28379341/) + + > da Veiga Leprevost F, Grüning B, Aflitos SA, Röst HL, Uszkoreit J, Barsnes H, Vaudel M, Moreno P, Gatto L, Weber J, Bai M, Jimenez RC, Sachsenberg T, Pfeuffer J, Alvarez RV, Griss J, Nesvizhskii AI, Perez-Riverol Y. BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics. 2017 Aug 15;33(16):2580-2582. doi: 10.1093/bioinformatics/btx192. PubMed PMID: 28379341; PubMed Central PMCID: PMC5870671. + +- [Docker](https://dl.acm.org/doi/10.5555/2600239.2600241) + +- [Singularity](https://pubmed.ncbi.nlm.nih.gov/28494014/) + > Kurtzer GM, Sochat V, Bauer MW. Singularity: Scientific containers for mobility of compute. PLoS One. 2017 May 11;12(5):e0177459. doi: 10.1371/journal.pone.0177459. eCollection 2017. PubMed PMID: 28494014; PubMed Central PMCID: PMC5426675. diff --git a/nf_core/pipeline-template/README.md b/nf_core/pipeline-template/README.md index e0049a72e9..9ce1bd191e 100644 --- a/nf_core/pipeline-template/README.md +++ b/nf_core/pipeline-template/README.md @@ -1,4 +1,4 @@ -# ![{{ name }}](docs/images/{{ name_noslash }}_logo_light.png#gh-light-mode-only) ![{{ name }}](docs/images/{{ name_noslash }}_logo_dark.png#gh-dark-mode-only) +# ![{{ name }}](docs/images/{{ name_noslash }}\_logo_light.png#gh-light-mode-only) ![{{ name }}](docs/images/{{ name_noslash }}\_logo_dark.png#gh-dark-mode-only) [![GitHub Actions CI Status](https://github.com/{{ name }}/workflows/nf-core%20CI/badge.svg)](https://github.com/{{ name }}/actions?query=workflow%3A%22nf-core+CI%22) [![GitHub Actions Linting Status](https://github.com/{{ name }}/workflows/nf-core%20linting/badge.svg)](https://github.com/{{ name }}/actions?query=workflow%3A%22nf-core+linting%22) @@ -17,11 +17,13 @@ ## Introduction + **{{ name }}** is a bioinformatics best-practice analysis pipeline for {{ description }}. The pipeline is built using [Nextflow](https://www.nextflow.io), a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. The [Nextflow DSL2](https://www.nextflow.io/docs/latest/dsl2.html) implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. Where possible, these processes have been submitted to and installed from [nf-core/modules](https://github.com/nf-core/modules) in order to make them available to all nf-core pipelines, and to everyone within the Nextflow community! + On release, automated continuous integration tests run the pipeline on a full-sized dataset on the AWS cloud infrastructure. This ensures that the pipeline runs on AWS, has sensible resource allocation defaults set to run on real-world datasets, and permits the persistent storage of results to benchmark between pipeline releases and other analysis sources. The results obtained from the full-sized test can be viewed on the [nf-core website](https://nf-co.re/{{ short_name }}/results). ## Pipeline summary @@ -39,24 +41,24 @@ On release, automated continuous integration tests run the pipeline on a full-si 3. Download the pipeline and test it on a minimal dataset with a single command: - ```console - nextflow run {{ name }} -profile test,YOURPROFILE --outdir - ``` + ```console + nextflow run {{ name }} -profile test,YOURPROFILE --outdir + ``` - Note that some form of configuration will be needed so that Nextflow knows how to fetch the required software. This is usually done in the form of a config profile (`YOURPROFILE` in the example command above). You can chain multiple config profiles in a comma-separated string. + Note that some form of configuration will be needed so that Nextflow knows how to fetch the required software. This is usually done in the form of a config profile (`YOURPROFILE` in the example command above). You can chain multiple config profiles in a comma-separated string. - > * The pipeline comes with config profiles called `docker`, `singularity`, `podman`, `shifter`, `charliecloud` and `conda` which instruct the pipeline to use the named tool for software management. For example, `-profile test,docker`. - > * Please check [nf-core/configs](https://github.com/nf-core/configs#documentation) to see if a custom config file to run nf-core pipelines already exists for your Institute. If so, you can simply use `-profile ` in your command. This will enable either `docker` or `singularity` and set the appropriate execution settings for your local compute environment. - > * If you are using `singularity`, please use the [`nf-core download`](https://nf-co.re/tools/#downloading-pipelines-for-offline-use) command to download images first, before running the pipeline. Setting the [`NXF_SINGULARITY_CACHEDIR` or `singularity.cacheDir`](https://www.nextflow.io/docs/latest/singularity.html?#singularity-docker-hub) Nextflow options enables you to store and re-use the images from a central location for future pipeline runs. - > * If you are using `conda`, it is highly recommended to use the [`NXF_CONDA_CACHEDIR` or `conda.cacheDir`](https://www.nextflow.io/docs/latest/conda.html) settings to store the environments in a central location for future pipeline runs. + > - The pipeline comes with config profiles called `docker`, `singularity`, `podman`, `shifter`, `charliecloud` and `conda` which instruct the pipeline to use the named tool for software management. For example, `-profile test,docker`. + > - Please check [nf-core/configs](https://github.com/nf-core/configs#documentation) to see if a custom config file to run nf-core pipelines already exists for your Institute. If so, you can simply use `-profile ` in your command. This will enable either `docker` or `singularity` and set the appropriate execution settings for your local compute environment. + > - If you are using `singularity`, please use the [`nf-core download`](https://nf-co.re/tools/#downloading-pipelines-for-offline-use) command to download images first, before running the pipeline. Setting the [`NXF_SINGULARITY_CACHEDIR` or `singularity.cacheDir`](https://www.nextflow.io/docs/latest/singularity.html?#singularity-docker-hub) Nextflow options enables you to store and re-use the images from a central location for future pipeline runs. + > - If you are using `conda`, it is highly recommended to use the [`NXF_CONDA_CACHEDIR` or `conda.cacheDir`](https://www.nextflow.io/docs/latest/conda.html) settings to store the environments in a central location for future pipeline runs. 4. Start running your own analysis! - + - ```console - nextflow run {{ name }} --input samplesheet.csv --outdir --genome GRCh37 -profile - ``` + ```console + nextflow run {{ name }} --input samplesheet.csv --outdir --genome GRCh37 -profile + ``` ## Documentation @@ -82,6 +84,7 @@ For further information or help, don't hesitate to get in touch on the [Slack `# + An extensive list of references for the tools used by the pipeline can be found in the [`CITATIONS.md`](CITATIONS.md) file. You can cite the `nf-core` publication as follows: diff --git a/nf_core/pipeline-template/assets/email_template.html b/nf_core/pipeline-template/assets/email_template.html index ecff600d44..069da09361 100644 --- a/nf_core/pipeline-template/assets/email_template.html +++ b/nf_core/pipeline-template/assets/email_template.html @@ -1,53 +1,110 @@ - - - - + + + + - - {{ name }} Pipeline Report - - -
+ + {{ name }} Pipeline Report + + +
+ - +

{{ name }} v${version}

+

Run Name: $runName

-

{{ name }} v${version}

-

Run Name: $runName

- -<% if (!success){ - out << """ -
-

{{ name }} execution completed unsuccessfully!

+ <% if (!success){ out << """ +
+

{{ name }} execution completed unsuccessfully!

The exit status of the task that caused the workflow execution to fail was: $exitStatus.

The full error message was:

- 
${errorReport}
-
- """ -} else { - out << """ -
+ 
${errorReport}
+
+ """ } else { out << """ +
{{ name }} execution completed successfully! -
- """ -} -%> +
+ """ } %> -

The workflow was completed at $dateComplete (duration: $duration)

-

The command used to launch the workflow was as follows:

-
$commandLine
+

The workflow was completed at $dateComplete (duration: $duration)

+

The command used to launch the workflow was as follows:

+ 
+$commandLine
-

Pipeline Configuration:

- - - <% out << summary.collect{ k,v -> "" }.join("\n") %> - -
$k
$v
+

Pipeline Configuration:

+ + + <% out << summary.collect{ k,v -> " + + + + + " }.join("\n") %> + +
+ $k + + 
$v
+
-

{{ name }}

-

https://github.com/{{ name }}

- -
- - +

{{ name }}

+

https://github.com/{{ name }}

+
+ diff --git a/nf_core/pipeline-template/assets/schema_input.json b/nf_core/pipeline-template/assets/schema_input.json index 5ce14d5825..8f6b7f047e 100644 --- a/nf_core/pipeline-template/assets/schema_input.json +++ b/nf_core/pipeline-template/assets/schema_input.json @@ -1,39 +1,36 @@ { - "$schema": "http://json-schema.org/draft-07/schema", - "$id": "https://raw.githubusercontent.com/{{ name }}/master/assets/schema_input.json", - "title": "{{ name }} pipeline - params.input schema", - "description": "Schema for the file provided with params.input", - "type": "array", - "items": { - "type": "object", - "properties": { - "sample": { - "type": "string", - "pattern": "^\\S+$", - "errorMessage": "Sample name must be provided and cannot contain spaces" - }, - "fastq_1": { - "type": "string", - "pattern": "^\\S+\\.f(ast)?q\\.gz$", - "errorMessage": "FastQ file for reads 1 must be provided, cannot contain spaces and must have extension '.fq.gz' or '.fastq.gz'" - }, - "fastq_2": { - "errorMessage": "FastQ file for reads 2 cannot contain spaces and must have extension '.fq.gz' or '.fastq.gz'", - "anyOf": [ - { - "type": "string", - "pattern": "^\\S+\\.f(ast)?q\\.gz$" - }, - { - "type": "string", - "maxLength": 0 - } - ] - } - }, - "required": [ - "sample", - "fastq_1" + "$schema": "http://json-schema.org/draft-07/schema", + "$id": "https://raw.githubusercontent.com/{{ name }}/master/assets/schema_input.json", + "title": "{{ name }} pipeline - params.input schema", + "description": "Schema for the file provided with params.input", + "type": "array", + "items": { + "type": "object", + "properties": { + "sample": { + "type": "string", + "pattern": "^\\S+$", + "errorMessage": "Sample name must be provided and cannot contain spaces" + }, + "fastq_1": { + "type": "string", + "pattern": "^\\S+\\.f(ast)?q\\.gz$", + "errorMessage": "FastQ file for reads 1 must be provided, cannot contain spaces and must have extension '.fq.gz' or '.fastq.gz'" + }, + "fastq_2": { + "errorMessage": "FastQ file for reads 2 cannot contain spaces and must have extension '.fq.gz' or '.fastq.gz'", + "anyOf": [ + { + "type": "string", + "pattern": "^\\S+\\.f(ast)?q\\.gz$" + }, + { + "type": "string", + "maxLength": 0 + } ] - } + } + }, + "required": ["sample", "fastq_1"] + } } diff --git a/nf_core/pipeline-template/docs/README.md b/nf_core/pipeline-template/docs/README.md index 6251d9c618..3b78de94cf 100644 --- a/nf_core/pipeline-template/docs/README.md +++ b/nf_core/pipeline-template/docs/README.md @@ -2,9 +2,9 @@ The {{ name }} documentation is split into the following pages: -* [Usage](usage.md) - * An overview of how the pipeline works, how to run it and a description of all of the different command-line flags. -* [Output](output.md) - * An overview of the different results produced by the pipeline and how to interpret them. +- [Usage](usage.md) + - An overview of how the pipeline works, how to run it and a description of all of the different command-line flags. +- [Output](output.md) + - An overview of the different results produced by the pipeline and how to interpret them. You can find a lot more documentation about installing, configuring and running nf-core pipelines on the website: [https://nf-co.re](https://nf-co.re) diff --git a/nf_core/pipeline-template/docs/output.md b/nf_core/pipeline-template/docs/output.md index 4ef9a4ea01..09eb6839d9 100644 --- a/nf_core/pipeline-template/docs/output.md +++ b/nf_core/pipeline-template/docs/output.md @@ -12,18 +12,18 @@ The directories listed below will be created in the results directory after the The pipeline is built using [Nextflow](https://www.nextflow.io/) and processes data using the following steps: -* [FastQC](#fastqc) - Raw read QC -* [MultiQC](#multiqc) - Aggregate report describing results and QC from the whole pipeline -* [Pipeline information](#pipeline-information) - Report metrics generated during the workflow execution +- [FastQC](#fastqc) - Raw read QC +- [MultiQC](#multiqc) - Aggregate report describing results and QC from the whole pipeline +- [Pipeline information](#pipeline-information) - Report metrics generated during the workflow execution ### FastQC
Output files -* `fastqc/` - * `*_fastqc.html`: FastQC report containing quality metrics. - * `*_fastqc.zip`: Zip archive containing the FastQC report, tab-delimited data file and plot images. +- `fastqc/` + - `*_fastqc.html`: FastQC report containing quality metrics. + - `*_fastqc.zip`: Zip archive containing the FastQC report, tab-delimited data file and plot images.
@@ -42,10 +42,10 @@ The pipeline is built using [Nextflow](https://www.nextflow.io/) and processes d
Output files -* `multiqc/` - * `multiqc_report.html`: a standalone HTML file that can be viewed in your web browser. - * `multiqc_data/`: directory containing parsed statistics from the different tools used in the pipeline. - * `multiqc_plots/`: directory containing static images from the report in various formats. +- `multiqc/` + - `multiqc_report.html`: a standalone HTML file that can be viewed in your web browser. + - `multiqc_data/`: directory containing parsed statistics from the different tools used in the pipeline. + - `multiqc_plots/`: directory containing static images from the report in various formats.
@@ -58,10 +58,10 @@ Results generated by MultiQC collate pipeline QC from supported tools e.g. FastQ
Output files -* `pipeline_info/` - * Reports generated by Nextflow: `execution_report.html`, `execution_timeline.html`, `execution_trace.txt` and `pipeline_dag.dot`/`pipeline_dag.svg`. - * Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.yml`. The `pipeline_report*` files will only be present if the `--email` / `--email_on_fail` parameter's are used when running the pipeline. - * Reformatted samplesheet files used as input to the pipeline: `samplesheet.valid.csv`. +- `pipeline_info/` + - Reports generated by Nextflow: `execution_report.html`, `execution_timeline.html`, `execution_trace.txt` and `pipeline_dag.dot`/`pipeline_dag.svg`. + - Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.yml`. The `pipeline_report*` files will only be present if the `--email` / `--email_on_fail` parameter's are used when running the pipeline. + - Reformatted samplesheet files used as input to the pipeline: `samplesheet.valid.csv`.
diff --git a/nf_core/pipeline-template/docs/usage.md b/nf_core/pipeline-template/docs/usage.md index 00da1fca8c..7169b46082 100644 --- a/nf_core/pipeline-template/docs/usage.md +++ b/nf_core/pipeline-template/docs/usage.md @@ -44,11 +44,11 @@ TREATMENT_REP3,AEG588A6_S6_L003_R1_001.fastq.gz, TREATMENT_REP3,AEG588A6_S6_L004_R1_001.fastq.gz, ``` -| Column | Description | -|----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `sample` | Custom sample name. This entry will be identical for multiple sequencing libraries/runs from the same sample. Spaces in sample names are automatically converted to underscores (`_`). | -| `fastq_1` | Full path to FastQ file for Illumina short reads 1. File has to be gzipped and have the extension ".fastq.gz" or ".fq.gz". | -| `fastq_2` | Full path to FastQ file for Illumina short reads 2. File has to be gzipped and have the extension ".fastq.gz" or ".fq.gz". | +| Column | Description | +| --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `sample` | Custom sample name. This entry will be identical for multiple sequencing libraries/runs from the same sample. Spaces in sample names are automatically converted to underscores (`_`). | +| `fastq_1` | Full path to FastQ file for Illumina short reads 1. File has to be gzipped and have the extension ".fastq.gz" or ".fq.gz". | +| `fastq_2` | Full path to FastQ file for Illumina short reads 2. File has to be gzipped and have the extension ".fastq.gz" or ".fq.gz". | An [example samplesheet](../assets/samplesheet.csv) has been provided with the pipeline. @@ -106,21 +106,21 @@ They are loaded in sequence, so later profiles can overwrite earlier profiles. If `-profile` is not specified, the pipeline will run locally and expect all software to be installed and available on the `PATH`. This is _not_ recommended. -* `docker` - * A generic configuration profile to be used with [Docker](https://docker.com/) -* `singularity` - * A generic configuration profile to be used with [Singularity](https://sylabs.io/docs/) -* `podman` - * A generic configuration profile to be used with [Podman](https://podman.io/) -* `shifter` - * A generic configuration profile to be used with [Shifter](https://nersc.gitlab.io/development/shifter/how-to-use/) -* `charliecloud` - * A generic configuration profile to be used with [Charliecloud](https://hpc.github.io/charliecloud/) -* `conda` - * A generic configuration profile to be used with [Conda](https://conda.io/docs/). Please only use Conda as a last resort i.e. when it's not possible to run the pipeline with Docker, Singularity, Podman, Shifter or Charliecloud. -* `test` - * A profile with a complete configuration for automated testing - * Includes links to test data so needs no other parameters +- `docker` + - A generic configuration profile to be used with [Docker](https://docker.com/) +- `singularity` + - A generic configuration profile to be used with [Singularity](https://sylabs.io/docs/) +- `podman` + - A generic configuration profile to be used with [Podman](https://podman.io/) +- `shifter` + - A generic configuration profile to be used with [Shifter](https://nersc.gitlab.io/development/shifter/how-to-use/) +- `charliecloud` + - A generic configuration profile to be used with [Charliecloud](https://hpc.github.io/charliecloud/) +- `conda` + - A generic configuration profile to be used with [Conda](https://conda.io/docs/). Please only use Conda as a last resort i.e. when it's not possible to run the pipeline with Docker, Singularity, Podman, Shifter or Charliecloud. +- `test` + - A profile with a complete configuration for automated testing + - Includes links to test data so needs no other parameters ### `-resume` @@ -197,35 +197,35 @@ The [Nextflow DSL2](https://www.nextflow.io/docs/latest/dsl2.html) implementatio 2. Find the latest version of the Biocontainer available on [Quay.io](https://quay.io/repository/biocontainers/pangolin?tag=latest&tab=tags) 3. Create the custom config accordingly: - * For Docker: - - ```nextflow - process { - withName: PANGOLIN { - container = 'quay.io/biocontainers/pangolin:3.0.5--pyhdfd78af_0' - } - } - ``` - - * For Singularity: - - ```nextflow - process { - withName: PANGOLIN { - container = 'https://depot.galaxyproject.org/singularity/pangolin:3.0.5--pyhdfd78af_0' - } - } - ``` - - * For Conda: - - ```nextflow - process { - withName: PANGOLIN { - conda = 'bioconda::pangolin=3.0.5' - } - } - ``` + - For Docker: + + ```nextflow + process { + withName: PANGOLIN { + container = 'quay.io/biocontainers/pangolin:3.0.5--pyhdfd78af_0' + } + } + ``` + + - For Singularity: + + ```nextflow + process { + withName: PANGOLIN { + container = 'https://depot.galaxyproject.org/singularity/pangolin:3.0.5--pyhdfd78af_0' + } + } + ``` + + - For Conda: + + ```nextflow + process { + withName: PANGOLIN { + conda = 'bioconda::pangolin=3.0.5' + } + } + ``` > **NB:** If you wish to periodically update individual tool-specific results (e.g. Pangolin) generated by the pipeline then you must ensure to keep the `work/` directory otherwise the `-resume` ability of the pipeline will be compromised and it will restart from scratch. diff --git a/nf_core/pipeline-template/modules.json b/nf_core/pipeline-template/modules.json index 2c377919a0..8c2d2a4071 100644 --- a/nf_core/pipeline-template/modules.json +++ b/nf_core/pipeline-template/modules.json @@ -1,17 +1,17 @@ { - "name": "{{ name }}", - "homePage": "https://github.com/{{ name }}", - "repos": { - "nf-core/modules": { - "custom/dumpsoftwareversions": { - "git_sha": "20d8250d9f39ddb05dfb437603aaf99b5c0b2b41" - }, - "fastqc": { - "git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d" - }, - "multiqc": { - "git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d" - } - } + "name": "{{ name }}", + "homePage": "https://github.com/{{ name }}", + "repos": { + "nf-core/modules": { + "custom/dumpsoftwareversions": { + "git_sha": "20d8250d9f39ddb05dfb437603aaf99b5c0b2b41" + }, + "fastqc": { + "git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d" + }, + "multiqc": { + "git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d" + } } -} \ No newline at end of file + } +} diff --git a/nf_core/pipeline-template/modules/nf-core/modules/custom/dumpsoftwareversions/meta.yml b/nf_core/pipeline-template/modules/nf-core/modules/custom/dumpsoftwareversions/meta.yml index 5b5b8a6026..60b546a012 100644 --- a/nf_core/pipeline-template/modules/nf-core/modules/custom/dumpsoftwareversions/meta.yml +++ b/nf_core/pipeline-template/modules/nf-core/modules/custom/dumpsoftwareversions/meta.yml @@ -8,7 +8,7 @@ tools: description: Custom module used to dump software versions within the nf-core pipeline template homepage: https://github.com/nf-core/tools documentation: https://github.com/nf-core/tools - licence: ['MIT'] + licence: ["MIT"] input: - versions: type: file diff --git a/nf_core/pipeline-template/nextflow_schema.json b/nf_core/pipeline-template/nextflow_schema.json index 598f8a84b8..531c92cfff 100644 --- a/nf_core/pipeline-template/nextflow_schema.json +++ b/nf_core/pipeline-template/nextflow_schema.json @@ -1,279 +1,269 @@ { - "$schema": "http://json-schema.org/draft-07/schema", - "$id": "https://raw.githubusercontent.com/{{ name }}/master/nextflow_schema.json", - "title": "{{ name }} pipeline parameters", - "description": "{{ description }}", - "type": "object", - "definitions": { - "input_output_options": { - "title": "Input/output options", - "type": "object", - "fa_icon": "fas fa-terminal", - "description": "Define where the pipeline should find input data and save output data.", - "required": [ - "input", - "outdir" - ], - "properties": { - "input": { - "type": "string", - "format": "file-path", - "mimetype": "text/csv", - "pattern": "^\\S+\\.csv$", - "schema": "assets/schema_input.json", - "description": "Path to comma-separated file containing information about the samples in the experiment.", - "help_text": "You will need to create a design file with information about the samples in your experiment before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row. See [usage docs](https://nf-co.re/{{ short_name }}/usage#samplesheet-input).", - "fa_icon": "fas fa-file-csv" - }, - "outdir": { - "type": "string", - "format": "directory-path", - "description": "The output directory where the results will be saved. You have to use absolute paths to storage on Cloud infrastructure.", - "fa_icon": "fas fa-folder-open" - }, - "email": { - "type": "string", - "description": "Email address for completion summary.", - "fa_icon": "fas fa-envelope", - "help_text": "Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file (`~/.nextflow/config`) then you don't need to specify this on the command line for every run.", - "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" - }, - "multiqc_title": { - "type": "string", - "description": "MultiQC report title. Printed as page header, used for filename if not otherwise specified.", - "fa_icon": "fas fa-file-signature" - } - } + "$schema": "http://json-schema.org/draft-07/schema", + "$id": "https://raw.githubusercontent.com/{{ name }}/master/nextflow_schema.json", + "title": "{{ name }} pipeline parameters", + "description": "{{ description }}", + "type": "object", + "definitions": { + "input_output_options": { + "title": "Input/output options", + "type": "object", + "fa_icon": "fas fa-terminal", + "description": "Define where the pipeline should find input data and save output data.", + "required": ["input", "outdir"], + "properties": { + "input": { + "type": "string", + "format": "file-path", + "mimetype": "text/csv", + "pattern": "^\\S+\\.csv$", + "schema": "assets/schema_input.json", + "description": "Path to comma-separated file containing information about the samples in the experiment.", + "help_text": "You will need to create a design file with information about the samples in your experiment before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row. See [usage docs](https://nf-co.re/{{ short_name }}/usage#samplesheet-input).", + "fa_icon": "fas fa-file-csv" }, - "reference_genome_options": { - "title": "Reference genome options", - "type": "object", - "fa_icon": "fas fa-dna", - "description": "Reference genome related files and options required for the workflow.", - "properties": { - "genome": { - "type": "string", - "description": "Name of iGenomes reference.", - "fa_icon": "fas fa-book", - "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. `--genome GRCh38`. \n\nSee the [nf-core website docs](https://nf-co.re/usage/reference_genomes) for more details." - }, - "fasta": { - "type": "string", - "format": "file-path", - "mimetype": "text/plain", - "pattern": "^\\S+\\.fn?a(sta)?(\\.gz)?$", - "description": "Path to FASTA genome file.", - "help_text": "This parameter is *mandatory* if `--genome` is not specified. If you don't have a BWA index available this will be generated for you automatically. Combine with `--save_reference` to save BWA index for future runs.", - "fa_icon": "far fa-file-code" - }, - "igenomes_base": { - "type": "string", - "format": "directory-path", - "description": "Directory / URL base for iGenomes references.", - "default": "s3://ngi-igenomes/igenomes", - "fa_icon": "fas fa-cloud-download-alt", - "hidden": true - }, - "igenomes_ignore": { - "type": "boolean", - "description": "Do not load the iGenomes reference config.", - "fa_icon": "fas fa-ban", - "hidden": true, - "help_text": "Do not load `igenomes.config` when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in `igenomes.config`." - } - } + "outdir": { + "type": "string", + "format": "directory-path", + "description": "The output directory where the results will be saved. You have to use absolute paths to storage on Cloud infrastructure.", + "fa_icon": "fas fa-folder-open" }, - "institutional_config_options": { - "title": "Institutional config options", - "type": "object", - "fa_icon": "fas fa-university", - "description": "Parameters used to describe centralised config profiles. These should not be edited.", - "help_text": "The centralised nf-core configuration profiles use a handful of pipeline parameters to describe themselves. This information is then printed to the Nextflow log when you run a pipeline. You should not need to change these values when you run a pipeline.", - "properties": { - "custom_config_version": { - "type": "string", - "description": "Git commit id for Institutional configs.", - "default": "master", - "hidden": true, - "fa_icon": "fas fa-users-cog" - }, - "custom_config_base": { - "type": "string", - "description": "Base directory for Institutional configs.", - "default": "https://raw.githubusercontent.com/nf-core/configs/master", - "hidden": true, - "help_text": "If you're running offline, Nextflow will not be able to fetch the institutional config files from the internet. If you don't need them, then this is not a problem. If you do need them, you should download the files from the repo and tell Nextflow where to find them with this parameter.", - "fa_icon": "fas fa-users-cog" - }, - "config_profile_name": { - "type": "string", - "description": "Institutional config name.", - "hidden": true, - "fa_icon": "fas fa-users-cog" - }, - "config_profile_description": { - "type": "string", - "description": "Institutional config description.", - "hidden": true, - "fa_icon": "fas fa-users-cog" - }, - "config_profile_contact": { - "type": "string", - "description": "Institutional config contact information.", - "hidden": true, - "fa_icon": "fas fa-users-cog" - }, - "config_profile_url": { - "type": "string", - "description": "Institutional config URL link.", - "hidden": true, - "fa_icon": "fas fa-users-cog" - } - } + "email": { + "type": "string", + "description": "Email address for completion summary.", + "fa_icon": "fas fa-envelope", + "help_text": "Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file (`~/.nextflow/config`) then you don't need to specify this on the command line for every run.", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" }, - "max_job_request_options": { - "title": "Max job request options", - "type": "object", - "fa_icon": "fab fa-acquisitions-incorporated", - "description": "Set the top limit for requested resources for any single job.", - "help_text": "If you are running on a smaller system, a pipeline step requesting more resources than are available may cause the Nextflow to stop the run with an error. These options allow you to cap the maximum resources requested by any single job so that the pipeline will run on your system.\n\nNote that you can not _increase_ the resources requested by any job using these options. For that you will need your own configuration file. See [the nf-core website](https://nf-co.re/usage/configuration) for details.", - "properties": { - "max_cpus": { - "type": "integer", - "description": "Maximum number of CPUs that can be requested for any single job.", - "default": 16, - "fa_icon": "fas fa-microchip", - "hidden": true, - "help_text": "Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. `--max_cpus 1`" - }, - "max_memory": { - "type": "string", - "description": "Maximum amount of memory that can be requested for any single job.", - "default": "128.GB", - "fa_icon": "fas fa-memory", - "pattern": "^\\d+(\\.\\d+)?\\.?\\s*(K|M|G|T)?B$", - "hidden": true, - "help_text": "Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. `--max_memory '8.GB'`" - }, - "max_time": { - "type": "string", - "description": "Maximum amount of time that can be requested for any single job.", - "default": "240.h", - "fa_icon": "far fa-clock", - "pattern": "^(\\d+\\.?\\s*(s|m|h|day)\\s*)+$", - "hidden": true, - "help_text": "Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. `--max_time '2.h'`" - } - } + "multiqc_title": { + "type": "string", + "description": "MultiQC report title. Printed as page header, used for filename if not otherwise specified.", + "fa_icon": "fas fa-file-signature" + } + } + }, + "reference_genome_options": { + "title": "Reference genome options", + "type": "object", + "fa_icon": "fas fa-dna", + "description": "Reference genome related files and options required for the workflow.", + "properties": { + "genome": { + "type": "string", + "description": "Name of iGenomes reference.", + "fa_icon": "fas fa-book", + "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. `--genome GRCh38`. \n\nSee the [nf-core website docs](https://nf-co.re/usage/reference_genomes) for more details." + }, + "fasta": { + "type": "string", + "format": "file-path", + "mimetype": "text/plain", + "pattern": "^\\S+\\.fn?a(sta)?(\\.gz)?$", + "description": "Path to FASTA genome file.", + "help_text": "This parameter is *mandatory* if `--genome` is not specified. If you don't have a BWA index available this will be generated for you automatically. Combine with `--save_reference` to save BWA index for future runs.", + "fa_icon": "far fa-file-code" + }, + "igenomes_base": { + "type": "string", + "format": "directory-path", + "description": "Directory / URL base for iGenomes references.", + "default": "s3://ngi-igenomes/igenomes", + "fa_icon": "fas fa-cloud-download-alt", + "hidden": true }, - "generic_options": { - "title": "Generic options", - "type": "object", - "fa_icon": "fas fa-file-import", - "description": "Less common options for the pipeline, typically set in a config file.", - "help_text": "These options are common to all nf-core pipelines and allow you to customise some of the core preferences for how the pipeline runs.\n\nTypically these options would be set in a Nextflow config file loaded for all pipeline runs, such as `~/.nextflow/config`.", - "properties": { - "help": { - "type": "boolean", - "description": "Display help text.", - "fa_icon": "fas fa-question-circle", - "hidden": true - }, - "publish_dir_mode": { - "type": "string", - "default": "copy", - "description": "Method used to save pipeline results to output directory.", - "help_text": "The Nextflow `publishDir` option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See [Nextflow docs](https://www.nextflow.io/docs/latest/process.html#publishdir) for details.", - "fa_icon": "fas fa-copy", - "enum": [ - "symlink", - "rellink", - "link", - "copy", - "copyNoFollow", - "move" - ], - "hidden": true - }, - "email_on_fail": { - "type": "string", - "description": "Email address for completion summary, only when pipeline fails.", - "fa_icon": "fas fa-exclamation-triangle", - "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$", - "help_text": "An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully.", - "hidden": true - }, - "plaintext_email": { - "type": "boolean", - "description": "Send plain-text email instead of HTML.", - "fa_icon": "fas fa-remove-format", - "hidden": true - }, - "max_multiqc_email_size": { - "type": "string", - "description": "File size limit when attaching MultiQC reports to summary emails.", - "pattern": "^\\d+(\\.\\d+)?\\.?\\s*(K|M|G|T)?B$", - "default": "25.MB", - "fa_icon": "fas fa-file-upload", - "hidden": true - }, - "monochrome_logs": { - "type": "boolean", - "description": "Do not use coloured log outputs.", - "fa_icon": "fas fa-palette", - "hidden": true - }, - "multiqc_config": { - "type": "string", - "description": "Custom config file to supply to MultiQC.", - "fa_icon": "fas fa-cog", - "hidden": true - }, - "tracedir": { - "type": "string", - "description": "Directory to keep pipeline Nextflow logs and reports.", - "default": "${params.outdir}/pipeline_info", - "fa_icon": "fas fa-cogs", - "hidden": true - }, - "validate_params": { - "type": "boolean", - "description": "Boolean whether to validate parameters against the schema at runtime", - "default": true, - "fa_icon": "fas fa-check-square", - "hidden": true - }, - "show_hidden_params": { - "type": "boolean", - "fa_icon": "far fa-eye-slash", - "description": "Show all params when using `--help`", - "hidden": true, - "help_text": "By default, parameters set as _hidden_ in the schema are not shown on the command line when a user runs with `--help`. Specifying this option will tell the pipeline to show all parameters." - }, - "enable_conda": { - "type": "boolean", - "description": "Run this workflow with Conda. You can also use '-profile conda' instead of providing this parameter.", - "hidden": true, - "fa_icon": "fas fa-bacon" - } - } + "igenomes_ignore": { + "type": "boolean", + "description": "Do not load the iGenomes reference config.", + "fa_icon": "fas fa-ban", + "hidden": true, + "help_text": "Do not load `igenomes.config` when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in `igenomes.config`." } + } }, - "allOf": [ - { - "$ref": "#/definitions/input_output_options" + "institutional_config_options": { + "title": "Institutional config options", + "type": "object", + "fa_icon": "fas fa-university", + "description": "Parameters used to describe centralised config profiles. These should not be edited.", + "help_text": "The centralised nf-core configuration profiles use a handful of pipeline parameters to describe themselves. This information is then printed to the Nextflow log when you run a pipeline. You should not need to change these values when you run a pipeline.", + "properties": { + "custom_config_version": { + "type": "string", + "description": "Git commit id for Institutional configs.", + "default": "master", + "hidden": true, + "fa_icon": "fas fa-users-cog" }, - { - "$ref": "#/definitions/reference_genome_options" + "custom_config_base": { + "type": "string", + "description": "Base directory for Institutional configs.", + "default": "https://raw.githubusercontent.com/nf-core/configs/master", + "hidden": true, + "help_text": "If you're running offline, Nextflow will not be able to fetch the institutional config files from the internet. If you don't need them, then this is not a problem. If you do need them, you should download the files from the repo and tell Nextflow where to find them with this parameter.", + "fa_icon": "fas fa-users-cog" }, - { - "$ref": "#/definitions/institutional_config_options" + "config_profile_name": { + "type": "string", + "description": "Institutional config name.", + "hidden": true, + "fa_icon": "fas fa-users-cog" }, - { - "$ref": "#/definitions/max_job_request_options" + "config_profile_description": { + "type": "string", + "description": "Institutional config description.", + "hidden": true, + "fa_icon": "fas fa-users-cog" }, - { - "$ref": "#/definitions/generic_options" + "config_profile_contact": { + "type": "string", + "description": "Institutional config contact information.", + "hidden": true, + "fa_icon": "fas fa-users-cog" + }, + "config_profile_url": { + "type": "string", + "description": "Institutional config URL link.", + "hidden": true, + "fa_icon": "fas fa-users-cog" } - ] + } + }, + "max_job_request_options": { + "title": "Max job request options", + "type": "object", + "fa_icon": "fab fa-acquisitions-incorporated", + "description": "Set the top limit for requested resources for any single job.", + "help_text": "If you are running on a smaller system, a pipeline step requesting more resources than are available may cause the Nextflow to stop the run with an error. These options allow you to cap the maximum resources requested by any single job so that the pipeline will run on your system.\n\nNote that you can not _increase_ the resources requested by any job using these options. For that you will need your own configuration file. See [the nf-core website](https://nf-co.re/usage/configuration) for details.", + "properties": { + "max_cpus": { + "type": "integer", + "description": "Maximum number of CPUs that can be requested for any single job.", + "default": 16, + "fa_icon": "fas fa-microchip", + "hidden": true, + "help_text": "Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. `--max_cpus 1`" + }, + "max_memory": { + "type": "string", + "description": "Maximum amount of memory that can be requested for any single job.", + "default": "128.GB", + "fa_icon": "fas fa-memory", + "pattern": "^\\d+(\\.\\d+)?\\.?\\s*(K|M|G|T)?B$", + "hidden": true, + "help_text": "Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. `--max_memory '8.GB'`" + }, + "max_time": { + "type": "string", + "description": "Maximum amount of time that can be requested for any single job.", + "default": "240.h", + "fa_icon": "far fa-clock", + "pattern": "^(\\d+\\.?\\s*(s|m|h|day)\\s*)+$", + "hidden": true, + "help_text": "Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. `--max_time '2.h'`" + } + } + }, + "generic_options": { + "title": "Generic options", + "type": "object", + "fa_icon": "fas fa-file-import", + "description": "Less common options for the pipeline, typically set in a config file.", + "help_text": "These options are common to all nf-core pipelines and allow you to customise some of the core preferences for how the pipeline runs.\n\nTypically these options would be set in a Nextflow config file loaded for all pipeline runs, such as `~/.nextflow/config`.", + "properties": { + "help": { + "type": "boolean", + "description": "Display help text.", + "fa_icon": "fas fa-question-circle", + "hidden": true + }, + "publish_dir_mode": { + "type": "string", + "default": "copy", + "description": "Method used to save pipeline results to output directory.", + "help_text": "The Nextflow `publishDir` option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See [Nextflow docs](https://www.nextflow.io/docs/latest/process.html#publishdir) for details.", + "fa_icon": "fas fa-copy", + "enum": ["symlink", "rellink", "link", "copy", "copyNoFollow", "move"], + "hidden": true + }, + "email_on_fail": { + "type": "string", + "description": "Email address for completion summary, only when pipeline fails.", + "fa_icon": "fas fa-exclamation-triangle", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$", + "help_text": "An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully.", + "hidden": true + }, + "plaintext_email": { + "type": "boolean", + "description": "Send plain-text email instead of HTML.", + "fa_icon": "fas fa-remove-format", + "hidden": true + }, + "max_multiqc_email_size": { + "type": "string", + "description": "File size limit when attaching MultiQC reports to summary emails.", + "pattern": "^\\d+(\\.\\d+)?\\.?\\s*(K|M|G|T)?B$", + "default": "25.MB", + "fa_icon": "fas fa-file-upload", + "hidden": true + }, + "monochrome_logs": { + "type": "boolean", + "description": "Do not use coloured log outputs.", + "fa_icon": "fas fa-palette", + "hidden": true + }, + "multiqc_config": { + "type": "string", + "description": "Custom config file to supply to MultiQC.", + "fa_icon": "fas fa-cog", + "hidden": true + }, + "tracedir": { + "type": "string", + "description": "Directory to keep pipeline Nextflow logs and reports.", + "default": "${params.outdir}/pipeline_info", + "fa_icon": "fas fa-cogs", + "hidden": true + }, + "validate_params": { + "type": "boolean", + "description": "Boolean whether to validate parameters against the schema at runtime", + "default": true, + "fa_icon": "fas fa-check-square", + "hidden": true + }, + "show_hidden_params": { + "type": "boolean", + "fa_icon": "far fa-eye-slash", + "description": "Show all params when using `--help`", + "hidden": true, + "help_text": "By default, parameters set as _hidden_ in the schema are not shown on the command line when a user runs with `--help`. Specifying this option will tell the pipeline to show all parameters." + }, + "enable_conda": { + "type": "boolean", + "description": "Run this workflow with Conda. You can also use '-profile conda' instead of providing this parameter.", + "hidden": true, + "fa_icon": "fas fa-bacon" + } + } + } + }, + "allOf": [ + { + "$ref": "#/definitions/input_output_options" + }, + { + "$ref": "#/definitions/reference_genome_options" + }, + { + "$ref": "#/definitions/institutional_config_options" + }, + { + "$ref": "#/definitions/max_job_request_options" + }, + { + "$ref": "#/definitions/generic_options" + } + ] } From a53ebfb1ad9ed7b46f71d94fe4883f42ea2aa70b Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 21:14:57 +0100 Subject: [PATCH 061/110] Manually fix remaining prettier errors in template files that won't compile --- .../pipeline-template/.github/workflows/branch.yml | 2 +- nf_core/pipeline-template/.github/workflows/ci.yml | 8 ++++---- .../.github/workflows/linting.yml | 11 +++++------ nf_core/pipeline-template/README.md | 2 +- .../pipeline-template/assets/multiqc_config.yaml | 14 +++++++------- 5 files changed, 18 insertions(+), 19 deletions(-) diff --git a/nf_core/pipeline-template/.github/workflows/branch.yml b/nf_core/pipeline-template/.github/workflows/branch.yml index 5281ada9cb..416723230d 100644 --- a/nf_core/pipeline-template/.github/workflows/branch.yml +++ b/nf_core/pipeline-template/.github/workflows/branch.yml @@ -15,7 +15,7 @@ jobs: run: | { [[ {% raw %}${{github.event.pull_request.head.repo.full_name }}{% endraw %} == {{ name }} ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] -{% raw %} +{%- raw %} # If the above check failed, post a comment on the PR explaining the failure # NOTE - this doesn't currently work if the PR is coming from a fork, due to limitations in GitHub actions secrets - name: Post PR comment diff --git a/nf_core/pipeline-template/.github/workflows/ci.yml b/nf_core/pipeline-template/.github/workflows/ci.yml index 94edf32438..3bedf3d6db 100644 --- a/nf_core/pipeline-template/.github/workflows/ci.yml +++ b/nf_core/pipeline-template/.github/workflows/ci.yml @@ -23,11 +23,11 @@ jobs: # Nextflow versions include: # Test pipeline minimum Nextflow version - - NXF_VER: '21.10.3' - NXF_EDGE: '' + - NXF_VER: "21.10.3" + NXF_EDGE: "" # Test latest edge release of Nextflow - - NXF_VER: '' - NXF_EDGE: '1' + - NXF_VER: "" + NXF_EDGE: "1" steps: - name: Check out pipeline code uses: actions/checkout@v2 diff --git a/nf_core/pipeline-template/.github/workflows/linting.yml b/nf_core/pipeline-template/.github/workflows/linting.yml index 28b0c97095..e4a6097f4e 100644 --- a/nf_core/pipeline-template/.github/workflows/linting.yml +++ b/nf_core/pipeline-template/.github/workflows/linting.yml @@ -62,11 +62,11 @@ jobs: steps: - name: Checkout uses: actions/checkout@master - - name: 'Yamllint' + - name: "Yamllint" uses: karancode/yamllint-github-action@master with: - yamllint_file_or_dir: '.' - yamllint_config_filepath: '.yamllint.yml' + yamllint_file_or_dir: "." + yamllint_config_filepath: ".yamllint.yml" # If the above check failed, post a comment on the PR explaining the failure - name: Post PR comment @@ -97,7 +97,6 @@ jobs: nf-core: runs-on: ubuntu-latest steps: - - name: Check out pipeline code uses: actions/checkout@v2 @@ -110,8 +109,8 @@ jobs: - uses: actions/setup-python@v1 with: - python-version: '3.6' - architecture: 'x64' + python-version: "3.6" + architecture: "x64" - name: Install dependencies run: | diff --git a/nf_core/pipeline-template/README.md b/nf_core/pipeline-template/README.md index 9ce1bd191e..2cf3f836b6 100644 --- a/nf_core/pipeline-template/README.md +++ b/nf_core/pipeline-template/README.md @@ -1,4 +1,4 @@ -# ![{{ name }}](docs/images/{{ name_noslash }}\_logo_light.png#gh-light-mode-only) ![{{ name }}](docs/images/{{ name_noslash }}\_logo_dark.png#gh-dark-mode-only) +# ![{{ name }}](docs/images/{{ name_noslash }}_logo_light.png#gh-light-mode-only) ![{{ name }}](docs/images/{{ name_noslash }}_logo_dark.png#gh-dark-mode-only) [![GitHub Actions CI Status](https://github.com/{{ name }}/workflows/nf-core%20CI/badge.svg)](https://github.com/{{ name }}/actions?query=workflow%3A%22nf-core+CI%22) [![GitHub Actions Linting Status](https://github.com/{{ name }}/workflows/nf-core%20linting/badge.svg)](https://github.com/{{ name }}/actions?query=workflow%3A%22nf-core+linting%22) diff --git a/nf_core/pipeline-template/assets/multiqc_config.yaml b/nf_core/pipeline-template/assets/multiqc_config.yaml index bbc9a14e1c..cceea6f1d9 100644 --- a/nf_core/pipeline-template/assets/multiqc_config.yaml +++ b/nf_core/pipeline-template/assets/multiqc_config.yaml @@ -1,11 +1,11 @@ report_comment: > - This report has been generated by the {{ name }} - analysis pipeline. For information about how to interpret these results, please see the - documentation. + This report has been generated by the {{ name }} + analysis pipeline. For information about how to interpret these results, please see the + documentation. report_section_order: - software_versions: - order: -1000 - {{ name.lower().replace('/', '-') }}-summary: - order: -1001 + software_versions: + order: -1000 + {{ name.lower().replace('/', '-') }}-summary: + order: -1001 export_plots: true From 181c9bb15e6eb59f19825c57cb51c740229083f5 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 21:16:29 +0100 Subject: [PATCH 062/110] Template: Remove old linting, add Prettier linting CI --- .../.github/workflows/linting.yml | 73 ++----------------- 1 file changed, 7 insertions(+), 66 deletions(-) diff --git a/nf_core/pipeline-template/.github/workflows/linting.yml b/nf_core/pipeline-template/.github/workflows/linting.yml index e4a6097f4e..ad9b42be4d 100644 --- a/nf_core/pipeline-template/.github/workflows/linting.yml +++ b/nf_core/pipeline-template/.github/workflows/linting.yml @@ -8,41 +8,6 @@ on: types: [published] jobs: - Markdown: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v2 - - uses: actions/setup-node@v2 - - name: Install markdownlint - run: npm install -g markdownlint-cli - - name: Run Markdownlint - run: markdownlint . - - # If the above check failed, post a comment on the PR explaining the failure - - name: Post PR comment - if: failure() - uses: mshick/add-pr-comment@v1 - with: - message: | - ## Markdown linting is failing - - To keep the code consistent with lots of contributors, we run automated code consistency checks. - To fix this CI test, please run: - - * Install `markdownlint-cli` - * On Mac: `brew install markdownlint-cli` - * Everything else: [Install `npm`](https://www.npmjs.com/get-npm) then [install `markdownlint-cli`](https://www.npmjs.com/package/markdownlint-cli) (`npm install -g markdownlint-cli`) - * Fix the markdown errors - * Automatically: `markdownlint . --fix` - * Manually resolve anything left from `markdownlint .` - - Once you push these changes the test should pass, and you can hide this comment :+1: - - We highly recommend setting up markdownlint in your code editor so that this formatting is done automatically on save. Ask about it on Slack for help! - - Thanks again for your contribution! - repo-token: ${{ secrets.GITHUB_TOKEN }} - allow-repeats: false EditorConfig: runs-on: ubuntu-latest @@ -57,42 +22,18 @@ jobs: - name: Run ECLint check run: editorconfig-checker -exclude README.md $(git ls-files | grep -v test) - YAML: + Prettier: runs-on: ubuntu-latest steps: - - name: Checkout - uses: actions/checkout@master - - name: "Yamllint" - uses: karancode/yamllint-github-action@master - with: - yamllint_file_or_dir: "." - yamllint_config_filepath: ".yamllint.yml" - - # If the above check failed, post a comment on the PR explaining the failure - - name: Post PR comment - if: failure() - uses: mshick/add-pr-comment@v1 - with: - message: | - ## YAML linting is failing - - To keep the code consistent with lots of contributors, we run automated code consistency checks. - To fix this CI test, please run: - - * Install `yamllint` - * Install `yamllint` following [this](https://yamllint.readthedocs.io/en/stable/quickstart.html#installing-yamllint) - instructions or alternative install it in your [conda environment](https://anaconda.org/conda-forge/yamllint) - * Fix the markdown errors - * Run the test locally: `yamllint $(find . -type f -name "*.yml" -o -name "*.yaml") -c ./.yamllint.yml` - * Fix any reported errors in your YAML files + - uses: actions/checkout@v2 - Once you push these changes the test should pass, and you can hide this comment :+1: + - uses: actions/setup-node@v2 - We highly recommend setting up yaml-lint in your code editor so that this formatting is done automatically on save. Ask about it on Slack for help! + - name: Install Prettier + run: npm install -g prettier - Thanks again for your contribution! - repo-token: ${{ secrets.GITHUB_TOKEN }} - allow-repeats: false + - name: Run Prettier --check + run: prettier --check ${GITHUB_WORKSPACE} nf-core: runs-on: ubuntu-latest From 8e6bbd82bd4a1612b71d7f73b23b1f4a8f8005c1 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 21:33:17 +0100 Subject: [PATCH 063/110] Try to move templating logic to make YAML files valid. This means that we can run prettier directly on the entire codebase, without ignoring the template. Will probably need quite a lot of sanity checking to make sure that I didn't break stuff! --- .prettierignore | 4 ---- nf_core/module-template/modules/meta.yml | 22 +++++++++---------- nf_core/module-template/tests/test.yml | 12 +++++----- .../.github/workflows/awsfulltest.yml | 3 +-- .../.github/workflows/awstest.yml | 2 +- .../.github/workflows/branch.yml | 11 +++++----- .../.github/workflows/ci.yml | 7 +++--- .../.github/workflows/linting.yml | 7 +++--- .../.github/workflows/linting_comment.yml | 4 +--- .../assets/multiqc_config.yaml | 2 +- 10 files changed, 33 insertions(+), 41 deletions(-) diff --git a/.prettierignore b/.prettierignore index 1057f5429e..069ef5131d 100644 --- a/.prettierignore +++ b/.prettierignore @@ -1,7 +1,3 @@ docs/api/_build testing -# Skip the templates as the Jinja2 tags confuse prettier -# Don't worry - we run prettier on the compiled pipeline in create-lint-wf.yml -nf_core/pipeline-template/ -nf_core/module-template/ diff --git a/nf_core/module-template/modules/meta.yml b/nf_core/module-template/modules/meta.yml index d58df5a371..6acf848a79 100644 --- a/nf_core/module-template/modules/meta.yml +++ b/nf_core/module-template/modules/meta.yml @@ -1,27 +1,27 @@ -name: {{ tool_name_underscore }} +name: "{{ tool_name_underscore }}" ## TODO nf-core: Add a description of the module and list keywords description: write your description here keywords: - sort tools: - - {{ tool }}: + - { { tool } }: ## TODO nf-core: Add a description and other details for the software below - description: {{ tool_description }} - homepage: {{ tool_doc_url }} - documentation: {{ tool_doc_url }} - tool_dev_url: {{ tool_dev_url }} + description: { { tool_description } } + homepage: { { tool_doc_url } } + documentation: { { tool_doc_url } } + tool_dev_url: { { tool_dev_url } } doi: "" - licence: {{ tool_licence }} + licence: { { tool_licence } } ## TODO nf-core: Add a description of all of the variables used as input input: - {% if has_meta -%} + #{% if has_meta %} Only when we have meta - meta: type: map description: | Groovy Map containing sample information e.g. [ id:'test', single_end:false ] - {% endif -%} + # {% endif %} ## TODO nf-core: Delete / customise this example input - bam: type: file @@ -30,13 +30,13 @@ input: ## TODO nf-core: Add a description of all of the variables used as output output: - {% if has_meta -%} + #{% if has_meta -%} Only when we have meta - meta: type: map description: | Groovy Map containing sample information e.g. [ id:'test', single_end:false ] - {% endif -%} + # {% endif -%} - versions: type: file description: File containing software versions diff --git a/nf_core/module-template/tests/test.yml b/nf_core/module-template/tests/test.yml index 8679a44943..9b0c62356d 100644 --- a/nf_core/module-template/tests/test.yml +++ b/nf_core/module-template/tests/test.yml @@ -1,12 +1,12 @@ ## TODO nf-core: Please run the following command to build this file: # nf-core modules create-test-yml {{ tool }}{%- if subtool %}/{{ subtool }}{%- endif %} -- name: {{ tool }}{{ ' '+subtool if subtool else '' }} +- name: "{{ tool }}{{ ' '+subtool if subtool else '' }}" command: nextflow run ./tests/modules/{{ tool_dir }} -entry test_{{ tool_name_underscore }} -c ./tests/config/nextflow.config -c ./tests/modules/{{ tool_dir }}/nextflow.config tags: - - {{ tool }} - {%- if subtool %} - - {{ tool }}/{{ subtool }} - {%- endif %} + - "{{ tool }}" + # {%- if subtool %} + - "{{ tool }}/{{ subtool }}" + # {%- endif %} files: - - path: output/{{ tool }}/test.bam + - path: "output/{{ tool }}/test.bam" md5sum: e667c7caad0bc4b7ac383fd023c654fc diff --git a/nf_core/pipeline-template/.github/workflows/awsfulltest.yml b/nf_core/pipeline-template/.github/workflows/awsfulltest.yml index 874149a936..31273a1bd5 100644 --- a/nf_core/pipeline-template/.github/workflows/awsfulltest.yml +++ b/nf_core/pipeline-template/.github/workflows/awsfulltest.yml @@ -17,8 +17,7 @@ jobs: uses: nf-core/tower-action@v3 # TODO nf-core: You can customise AWS full pipeline tests as required # Add full size test data (but still relatively small datasets for few samples) - # on the `test_full.config` test runs with only one set of parameters - {%- raw %} + # on the `test_full.config` test runs with only one set of parameters {% raw %} with: workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }} access_token: ${{ secrets.TOWER_ACCESS_TOKEN }} diff --git a/nf_core/pipeline-template/.github/workflows/awstest.yml b/nf_core/pipeline-template/.github/workflows/awstest.yml index 99f64ec34a..6a58539b80 100644 --- a/nf_core/pipeline-template/.github/workflows/awstest.yml +++ b/nf_core/pipeline-template/.github/workflows/awstest.yml @@ -10,9 +10,9 @@ jobs: if: github.repository == '{{ name }}' runs-on: ubuntu-latest steps: + # Launch workflow using Tower CLI tool action {%- raw %} - name: Launch workflow via tower uses: nf-core/tower-action@v3 - {%- raw %} with: workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }} access_token: ${{ secrets.TOWER_ACCESS_TOKEN }} diff --git a/nf_core/pipeline-template/.github/workflows/branch.yml b/nf_core/pipeline-template/.github/workflows/branch.yml index 416723230d..40da8e6829 100644 --- a/nf_core/pipeline-template/.github/workflows/branch.yml +++ b/nf_core/pipeline-template/.github/workflows/branch.yml @@ -5,17 +5,19 @@ on: pull_request_target: branches: [master] +env: + REPO_NAME: "{{ name }}" + jobs: test: runs-on: ubuntu-latest steps: - # PRs to the nf-core repo master branch are only ok if coming from the nf-core repo `dev` or any `patch` branches + # PRs to the nf-core repo master branch are only ok if coming from the nf-core repo `dev` or any `patch` branches. {% raw %} - name: Check PRs - if: github.repository == '{{ name }}' + if: github.repository == env.REPO_NAME run: | - { [[ {% raw %}${{github.event.pull_request.head.repo.full_name }}{% endraw %} == {{ name }} ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] + { [[ ${{github.event.pull_request.head.repo.full_name }} == env.REPO_NAME ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] -{%- raw %} # If the above check failed, post a comment on the PR explaining the failure # NOTE - this doesn't currently work if the PR is coming from a fork, due to limitations in GitHub actions secrets - name: Post PR comment @@ -43,4 +45,3 @@ jobs: Thanks again for your contribution! repo-token: ${{ secrets.GITHUB_TOKEN }} allow-repeats: false -{%- endraw %} diff --git a/nf_core/pipeline-template/.github/workflows/ci.yml b/nf_core/pipeline-template/.github/workflows/ci.yml index 3bedf3d6db..348a6ac25c 100644 --- a/nf_core/pipeline-template/.github/workflows/ci.yml +++ b/nf_core/pipeline-template/.github/workflows/ci.yml @@ -11,12 +11,13 @@ on: env: NXF_ANSI_LOG: false CAPSULE_LOG: none + REPO_NAME: "{{ name }}" jobs: test: name: Run pipeline with test data - # Only run on push if this is the nf-core dev branch (merged PRs) - if: {% raw %}${{{% endraw %} github.event_name != 'push' || (github.event_name == 'push' && github.repository == '{{ name }}') {% raw %}}}{% endraw %} + # Only run on push if this is the nf-core dev branch (merged PRs) {% raw %} + if: ${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == env.REPO_NAME) }} runs-on: ubuntu-latest strategy: matrix: @@ -34,12 +35,10 @@ jobs: - name: Install Nextflow env: - {% raw -%} NXF_VER: ${{ matrix.NXF_VER }} # Uncomment only if the edge release is more recent than the latest stable release # See https://github.com/nextflow-io/nextflow/issues/2467 # NXF_EDGE: ${{ matrix.NXF_EDGE }} - {%- endraw %} run: | wget -qO- get.nextflow.io | bash sudo mv nextflow /usr/local/bin/ diff --git a/nf_core/pipeline-template/.github/workflows/linting.yml b/nf_core/pipeline-template/.github/workflows/linting.yml index ad9b42be4d..7609d3f786 100644 --- a/nf_core/pipeline-template/.github/workflows/linting.yml +++ b/nf_core/pipeline-template/.github/workflows/linting.yml @@ -1,6 +1,7 @@ -{% raw %}name: nf-core linting +name: nf-core linting # This workflow is triggered on pushes and PRs to the repository. -# It runs the `nf-core lint` and markdown lint tests to ensure that the code meets the nf-core guidelines +# It runs the `nf-core lint` and markdown lint tests to ensure +# that the code meets the nf-core guidelines. {% raw %} on: push: pull_request: @@ -8,7 +9,6 @@ on: types: [published] jobs: - EditorConfig: runs-on: ubuntu-latest steps: @@ -78,4 +78,3 @@ jobs: lint_log.txt lint_results.md PR_number.txt -{%- endraw %} diff --git a/nf_core/pipeline-template/.github/workflows/linting_comment.yml b/nf_core/pipeline-template/.github/workflows/linting_comment.yml index edd997325e..15cc1d6bcc 100644 --- a/nf_core/pipeline-template/.github/workflows/linting_comment.yml +++ b/nf_core/pipeline-template/.github/workflows/linting_comment.yml @@ -1,7 +1,6 @@ -{% raw -%} name: nf-core linting comment # This workflow is triggered after the linting action is complete -# It posts an automated comment to the PR, even if the PR is coming from a fork +# It posts an automated comment to the PR, even if the PR is coming from a fork {% raw -%} on: workflow_run: @@ -27,4 +26,3 @@ jobs: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} number: ${{ steps.pr_number.outputs.pr_number }} path: linting-logs/lint_results.md -{%- endraw %} diff --git a/nf_core/pipeline-template/assets/multiqc_config.yaml b/nf_core/pipeline-template/assets/multiqc_config.yaml index cceea6f1d9..e12f6b16cb 100644 --- a/nf_core/pipeline-template/assets/multiqc_config.yaml +++ b/nf_core/pipeline-template/assets/multiqc_config.yaml @@ -5,7 +5,7 @@ report_comment: > report_section_order: software_versions: order: -1000 - {{ name.lower().replace('/', '-') }}-summary: + "{{ name.lower().replace('/', '-') }}-summary": order: -1001 export_plots: true From 70c27cc2187a564cb04c407a27256f857d36ed13 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 21:38:03 +0100 Subject: [PATCH 064/110] Simpler workaround for CI tests by just using YAML quotes --- .../pipeline-template/.github/workflows/branch.yml | 11 ++++------- nf_core/pipeline-template/.github/workflows/ci.yml | 7 +++---- 2 files changed, 7 insertions(+), 11 deletions(-) diff --git a/nf_core/pipeline-template/.github/workflows/branch.yml b/nf_core/pipeline-template/.github/workflows/branch.yml index 40da8e6829..eacfac0e3f 100644 --- a/nf_core/pipeline-template/.github/workflows/branch.yml +++ b/nf_core/pipeline-template/.github/workflows/branch.yml @@ -5,20 +5,17 @@ on: pull_request_target: branches: [master] -env: - REPO_NAME: "{{ name }}" - jobs: test: runs-on: ubuntu-latest steps: - # PRs to the nf-core repo master branch are only ok if coming from the nf-core repo `dev` or any `patch` branches. {% raw %} + # PRs to the nf-core repo master branch are only ok if coming from the nf-core repo `dev` or any `patch` branches - name: Check PRs - if: github.repository == env.REPO_NAME + if: github.repository == '{{ name }}' run: | - { [[ ${{github.event.pull_request.head.repo.full_name }} == env.REPO_NAME ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] + "{ [[ {% raw %}${{github.event.pull_request.head.repo.full_name }}{% endraw %} == {{ name }} ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]]" - # If the above check failed, post a comment on the PR explaining the failure + # If the above check failed, post a comment on the PR explaining the failure {% raw %} # NOTE - this doesn't currently work if the PR is coming from a fork, due to limitations in GitHub actions secrets - name: Post PR comment if: failure() diff --git a/nf_core/pipeline-template/.github/workflows/ci.yml b/nf_core/pipeline-template/.github/workflows/ci.yml index 348a6ac25c..5afb1aef51 100644 --- a/nf_core/pipeline-template/.github/workflows/ci.yml +++ b/nf_core/pipeline-template/.github/workflows/ci.yml @@ -11,13 +11,12 @@ on: env: NXF_ANSI_LOG: false CAPSULE_LOG: none - REPO_NAME: "{{ name }}" jobs: test: name: Run pipeline with test data - # Only run on push if this is the nf-core dev branch (merged PRs) {% raw %} - if: ${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == env.REPO_NAME) }} + # Only run on push if this is the nf-core dev branch (merged PRs) + if: "{% raw %}${{{% endraw %} github.event_name != 'push' || (github.event_name == 'push' && github.repository == '{{ name }}') {% raw %}}}{% endraw %}" runs-on: ubuntu-latest strategy: matrix: @@ -26,7 +25,7 @@ jobs: # Test pipeline minimum Nextflow version - NXF_VER: "21.10.3" NXF_EDGE: "" - # Test latest edge release of Nextflow + # Test latest edge release of Nextflow {% raw %} - NXF_VER: "" NXF_EDGE: "1" steps: From 5736a0566ee29477d17f81bf58070bc07a6b45c1 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 21:44:02 +0100 Subject: [PATCH 065/110] Work around Prettier auto-formatting in pipeline readme --- nf_core/create.py | 2 ++ nf_core/pipeline-template/README.md | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/nf_core/create.py b/nf_core/create.py index 31404a241d..3fa3baeda0 100644 --- a/nf_core/create.py +++ b/nf_core/create.py @@ -39,6 +39,8 @@ def __init__(self, name, description, author, version="1.0dev", no_git=False, fo self.name = f"nf-core/{self.short_name}" self.name_noslash = self.name.replace("/", "-") self.name_docker = self.name.replace("nf-core", "nfcore") + self.logo_light = f"{self.name}_logo_light.png" + self.logo_dark = f"{self.name}_logo_dark.png" self.description = description self.author = author self.version = version diff --git a/nf_core/pipeline-template/README.md b/nf_core/pipeline-template/README.md index 2cf3f836b6..faab299a13 100644 --- a/nf_core/pipeline-template/README.md +++ b/nf_core/pipeline-template/README.md @@ -1,4 +1,4 @@ -# ![{{ name }}](docs/images/{{ name_noslash }}_logo_light.png#gh-light-mode-only) ![{{ name }}](docs/images/{{ name_noslash }}_logo_dark.png#gh-dark-mode-only) +# ![{{ name }}](docs/images/{{ logo_light }}#gh-light-mode-only) ![{{ name }}](docs/images/{{ logo_dark }}#gh-dark-mode-only) [![GitHub Actions CI Status](https://github.com/{{ name }}/workflows/nf-core%20CI/badge.svg)](https://github.com/{{ name }}/actions?query=workflow%3A%22nf-core+CI%22) [![GitHub Actions Linting Status](https://github.com/{{ name }}/workflows/nf-core%20linting/badge.svg)](https://github.com/{{ name }}/actions?query=workflow%3A%22nf-core+linting%22) From 861555b7db5d68cc48da5a68c5b5b31bde8d2b0f Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 23:20:47 +0100 Subject: [PATCH 066/110] Trailing whitespace after jinja2 tags --- nf_core/pipeline-template/.github/workflows/awsfulltest.yml | 2 +- nf_core/pipeline-template/.github/workflows/branch.yml | 3 ++- nf_core/pipeline-template/.github/workflows/ci.yml | 4 +++- nf_core/pipeline-template/.github/workflows/linting.yml | 4 +++- .../pipeline-template/.github/workflows/linting_comment.yml | 3 ++- 5 files changed, 11 insertions(+), 5 deletions(-) diff --git a/nf_core/pipeline-template/.github/workflows/awsfulltest.yml b/nf_core/pipeline-template/.github/workflows/awsfulltest.yml index 31273a1bd5..4851b2875c 100644 --- a/nf_core/pipeline-template/.github/workflows/awsfulltest.yml +++ b/nf_core/pipeline-template/.github/workflows/awsfulltest.yml @@ -17,7 +17,7 @@ jobs: uses: nf-core/tower-action@v3 # TODO nf-core: You can customise AWS full pipeline tests as required # Add full size test data (but still relatively small datasets for few samples) - # on the `test_full.config` test runs with only one set of parameters {% raw %} + # on the `test_full.config` test runs with only one set of parameters {%- raw %} with: workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }} access_token: ${{ secrets.TOWER_ACCESS_TOKEN }} diff --git a/nf_core/pipeline-template/.github/workflows/branch.yml b/nf_core/pipeline-template/.github/workflows/branch.yml index eacfac0e3f..8f10abac98 100644 --- a/nf_core/pipeline-template/.github/workflows/branch.yml +++ b/nf_core/pipeline-template/.github/workflows/branch.yml @@ -15,7 +15,7 @@ jobs: run: | "{ [[ {% raw %}${{github.event.pull_request.head.repo.full_name }}{% endraw %} == {{ name }} ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]]" - # If the above check failed, post a comment on the PR explaining the failure {% raw %} + # If the above check failed, post a comment on the PR explaining the failure {%- raw %} # NOTE - this doesn't currently work if the PR is coming from a fork, due to limitations in GitHub actions secrets - name: Post PR comment if: failure() @@ -42,3 +42,4 @@ jobs: Thanks again for your contribution! repo-token: ${{ secrets.GITHUB_TOKEN }} allow-repeats: false +# {%- endraw %} diff --git a/nf_core/pipeline-template/.github/workflows/ci.yml b/nf_core/pipeline-template/.github/workflows/ci.yml index 5afb1aef51..0cd52074b7 100644 --- a/nf_core/pipeline-template/.github/workflows/ci.yml +++ b/nf_core/pipeline-template/.github/workflows/ci.yml @@ -25,7 +25,7 @@ jobs: # Test pipeline minimum Nextflow version - NXF_VER: "21.10.3" NXF_EDGE: "" - # Test latest edge release of Nextflow {% raw %} + # Test latest edge release of Nextflow {%- raw %} - NXF_VER: "" NXF_EDGE: "1" steps: @@ -48,3 +48,5 @@ jobs: # Remember that you can parallelise this by using strategy.matrix run: | nextflow run ${GITHUB_WORKSPACE} -profile test,docker --outdir ./results + +# {%- endraw %} diff --git a/nf_core/pipeline-template/.github/workflows/linting.yml b/nf_core/pipeline-template/.github/workflows/linting.yml index 7609d3f786..48d643f657 100644 --- a/nf_core/pipeline-template/.github/workflows/linting.yml +++ b/nf_core/pipeline-template/.github/workflows/linting.yml @@ -1,7 +1,7 @@ name: nf-core linting # This workflow is triggered on pushes and PRs to the repository. # It runs the `nf-core lint` and markdown lint tests to ensure -# that the code meets the nf-core guidelines. {% raw %} +# that the code meets the nf-core guidelines. {%- raw %} on: push: pull_request: @@ -78,3 +78,5 @@ jobs: lint_log.txt lint_results.md PR_number.txt + +# {%- endraw %} diff --git a/nf_core/pipeline-template/.github/workflows/linting_comment.yml b/nf_core/pipeline-template/.github/workflows/linting_comment.yml index 15cc1d6bcc..f3c6e56501 100644 --- a/nf_core/pipeline-template/.github/workflows/linting_comment.yml +++ b/nf_core/pipeline-template/.github/workflows/linting_comment.yml @@ -1,6 +1,6 @@ name: nf-core linting comment # This workflow is triggered after the linting action is complete -# It posts an automated comment to the PR, even if the PR is coming from a fork {% raw -%} +# It posts an automated comment to the PR, even if the PR is coming from a fork {%- raw %} on: workflow_run: @@ -26,3 +26,4 @@ jobs: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} number: ${{ steps.pr_number.outputs.pr_number }} path: linting-logs/lint_results.md +# {%- endraw %} From 6acf5d58ea8bd4fe75395a4d04bd6c79db1563c4 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 23:24:45 +0100 Subject: [PATCH 067/110] Update nf-core pipeline lint tests --- nf_core/lint/files_exist.py | 8 ++++++-- nf_core/lint/files_unchanged.py | 4 ++-- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/nf_core/lint/files_exist.py b/nf_core/lint/files_exist.py index 05f6de108d..33becc668b 100644 --- a/nf_core/lint/files_exist.py +++ b/nf_core/lint/files_exist.py @@ -20,7 +20,8 @@ def files_exist(self): .gitattributes .gitignore - .markdownlint.yml + .nf-core.yml + .prettierrc.yml .github/.dockstore.yml .github/CONTRIBUTING.md .github/ISSUE_TEMPLATE/bug_report.yml @@ -80,6 +81,7 @@ def files_exist(self): .github/ISSUE_TEMPLATE/bug_report.md .github/ISSUE_TEMPLATE/feature_request.md docs/images/nf-core-PIPELINE_logo.png + .markdownlint.yml Files that *should not* be present: @@ -100,7 +102,8 @@ def files_exist(self): files_fail = [ [".gitattributes"], [".gitignore"], - [".markdownlint.yml"], + [".nf-core.yaml"], + [".prettierrc.yaml"], ["CHANGELOG.md"], ["CITATIONS.md"], ["CODE_OF_CONDUCT.md"], @@ -160,6 +163,7 @@ def files_exist(self): os.path.join(".github", "ISSUE_TEMPLATE", "bug_report.md"), os.path.join(".github", "ISSUE_TEMPLATE", "feature_request.md"), os.path.join("docs", "images", f"nf-core-{short_name}_logo.png"), + ".markdownlint.yml", ] files_warn_ifexists = [".travis.yml"] diff --git a/nf_core/lint/files_unchanged.py b/nf_core/lint/files_unchanged.py index cd034fa24b..887406c6f0 100644 --- a/nf_core/lint/files_unchanged.py +++ b/nf_core/lint/files_unchanged.py @@ -19,7 +19,7 @@ def files_unchanged(self): Files that must be unchanged:: .gitattributes - .markdownlint.yml + .prettierrc.yml .github/.dockstore.yml .github/CONTRIBUTING.md .github/ISSUE_TEMPLATE/bug_report.yml @@ -75,7 +75,7 @@ def files_unchanged(self): # List of lists. Passes if any of the files in the sublist are found. files_exact = [ [".gitattributes"], - [".markdownlint.yml"], + [".prettierrc.yml"], ["CODE_OF_CONDUCT.md"], ["LICENSE", "LICENSE.md", "LICENCE", "LICENCE.md"], # NB: British / American spelling [os.path.join(".github", ".dockstore.yml")], From 9a8113720b761305825957d7ddb509229edafdba Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 23:26:48 +0100 Subject: [PATCH 068/110] yaml/yml typo --- nf_core/lint/files_exist.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/lint/files_exist.py b/nf_core/lint/files_exist.py index 33becc668b..ff7f8a8a70 100644 --- a/nf_core/lint/files_exist.py +++ b/nf_core/lint/files_exist.py @@ -102,7 +102,7 @@ def files_exist(self): files_fail = [ [".gitattributes"], [".gitignore"], - [".nf-core.yaml"], + [".nf-core.yml"], [".prettierrc.yaml"], ["CHANGELOG.md"], ["CITATIONS.md"], From cfb217f19d340bb961d9de95c7fd75c08f9a486d Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 23:28:02 +0100 Subject: [PATCH 069/110] .prettierrc yaml -> yml --- .prettierrc.yaml => .prettierrc.yml | 0 nf_core/lint/files_exist.py | 2 +- nf_core/pipeline-template/{.prettierrc.yaml => .prettierrc.yml} | 0 3 files changed, 1 insertion(+), 1 deletion(-) rename .prettierrc.yaml => .prettierrc.yml (100%) rename nf_core/pipeline-template/{.prettierrc.yaml => .prettierrc.yml} (100%) diff --git a/.prettierrc.yaml b/.prettierrc.yml similarity index 100% rename from .prettierrc.yaml rename to .prettierrc.yml diff --git a/nf_core/lint/files_exist.py b/nf_core/lint/files_exist.py index ff7f8a8a70..f00c676196 100644 --- a/nf_core/lint/files_exist.py +++ b/nf_core/lint/files_exist.py @@ -103,7 +103,7 @@ def files_exist(self): [".gitattributes"], [".gitignore"], [".nf-core.yml"], - [".prettierrc.yaml"], + [".prettierrc.yml"], ["CHANGELOG.md"], ["CITATIONS.md"], ["CODE_OF_CONDUCT.md"], diff --git a/nf_core/pipeline-template/.prettierrc.yaml b/nf_core/pipeline-template/.prettierrc.yml similarity index 100% rename from nf_core/pipeline-template/.prettierrc.yaml rename to nf_core/pipeline-template/.prettierrc.yml From fb5211a6b274bb0d75d159aa7f6b439787f47067 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 17 Mar 2022 23:44:42 +0100 Subject: [PATCH 070/110] Update modules before running nf-core lint --- .github/workflows/create-lint-wf.yml | 13 +++++++++---- nf_core/__main__.py | 13 ++++++++++--- nf_core/modules/update.py | 12 +++++++++--- 3 files changed, 28 insertions(+), 10 deletions(-) diff --git a/.github/workflows/create-lint-wf.yml b/.github/workflows/create-lint-wf.yml index 2296242291..6b0ce125e7 100644 --- a/.github/workflows/create-lint-wf.yml +++ b/.github/workflows/create-lint-wf.yml @@ -61,10 +61,7 @@ jobs: - name: nf-core modules create run: nf-core --log-file log.txt modules create bpipe --dir nf-core-testpipeline --author @nf-core-bot --label process_low --meta - # Run all the linters - - name: nf-core lint - run: nf-core --log-file log.txt lint --dir nf-core-testpipeline --fail-ignored - + # Run code style linters - name: Run Prettier --check run: prettier --check nf-core-testpipeline @@ -73,6 +70,14 @@ jobs: with: src: "./nf-core-testpipeline" + # Update modules to the latest version + - name: nf-core modules update + run: nf-core --log-file log.txt modules update --dir nf-core-testpipeline --all --no-preview + + # Run nf-core linting + - name: nf-core lint + run: nf-core --log-file log.txt lint --dir nf-core-testpipeline --fail-ignored + # Run the other nf-core commands - name: nf-core list run: nf-core --log-file log.txt list diff --git a/nf_core/__main__.py b/nf_core/__main__.py index 7ce4a18a06..8affd23bfe 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -464,14 +464,21 @@ def install(ctx, tool, dir, prompt, force, sha): @click.option("-s", "--sha", type=str, metavar="", help="Install module at commit SHA") @click.option("-a", "--all", is_flag=True, default=False, help="Update all modules installed in pipeline") @click.option( - "-x", + "-x/-y", + "--preview/--no-preview", + is_flag=True, + default=None, + help="Preview / no preview of changes before applying", +) +@click.option( + "-p", "--save-diff", type=str, metavar="", default=None, help="Save diffs to a file instead of updating in place", ) -def update(ctx, tool, dir, force, prompt, sha, all, save_diff): +def update(ctx, tool, dir, force, prompt, sha, all, preview, save_diff): """ Update DSL2 modules within a pipeline. @@ -479,7 +486,7 @@ def update(ctx, tool, dir, force, prompt, sha, all, save_diff): """ try: module_install = nf_core.modules.ModuleUpdate( - dir, force=force, prompt=prompt, sha=sha, update_all=all, save_diff_fn=save_diff + dir, force=force, prompt=prompt, sha=sha, update_all=all, show_diff=preview, save_diff_fn=save_diff ) module_install.modules_repo = ctx.obj["modules_repo_obj"] exit_status = module_install.update(tool) diff --git a/nf_core/modules/update.py b/nf_core/modules/update.py index 910af7ebaa..1967ac55a4 100644 --- a/nf_core/modules/update.py +++ b/nf_core/modules/update.py @@ -22,13 +22,15 @@ class ModuleUpdate(ModuleCommand): - def __init__(self, pipeline_dir, force=False, prompt=False, sha=None, update_all=False, save_diff_fn=None): + def __init__( + self, pipeline_dir, force=False, prompt=False, sha=None, update_all=False, show_diff=None, save_diff_fn=None + ): super().__init__(pipeline_dir) self.force = force self.prompt = prompt self.sha = sha self.update_all = update_all - self.show_diff = False + self.show_diff = show_diff self.save_diff_fn = save_diff_fn def update(self, module): @@ -187,8 +189,12 @@ def update(self, module): if not modules_json: return False + # If --preview is true, don't save to a patch file + if self.show_diff: + self.show_diff_fn = False + # Ask if we should show the diffs (unless a filename was already given on the command line) - if not self.save_diff_fn: + if not self.save_diff_fn and self.show_diff is None: diff_type = questionary.select( "Do you want to view diffs of the proposed changes?", choices=[ From 369180574052f80221ad3e5e1df93cea796e03b9 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 00:08:50 +0100 Subject: [PATCH 071/110] Don't test pipeline template with black --- .github/workflows/create-lint-wf.yml | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/.github/workflows/create-lint-wf.yml b/.github/workflows/create-lint-wf.yml index 6b0ce125e7..30920e7431 100644 --- a/.github/workflows/create-lint-wf.yml +++ b/.github/workflows/create-lint-wf.yml @@ -61,15 +61,10 @@ jobs: - name: nf-core modules create run: nf-core --log-file log.txt modules create bpipe --dir nf-core-testpipeline --author @nf-core-bot --label process_low --meta - # Run code style linters + # Run code style linting - name: Run Prettier --check run: prettier --check nf-core-testpipeline - - name: Check code lints with Black - uses: psf/black@stable - with: - src: "./nf-core-testpipeline" - # Update modules to the latest version - name: nf-core modules update run: nf-core --log-file log.txt modules update --dir nf-core-testpipeline --all --no-preview From 6e60a8d59b25efa3089578423a40d02ba99356ac Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 00:22:18 +0100 Subject: [PATCH 072/110] Move sync test to before we change anything --- .github/workflows/create-lint-wf.yml | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/workflows/create-lint-wf.yml b/.github/workflows/create-lint-wf.yml index 30920e7431..d26333540b 100644 --- a/.github/workflows/create-lint-wf.yml +++ b/.github/workflows/create-lint-wf.yml @@ -57,6 +57,10 @@ jobs: - name: nf-core create run: nf-core --log-file log.txt create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface" + # Try syncing it before we change anything + - name: nf-core sync + run: nf-core --log-file log.txt sync --dir nf-core-testpipeline/ + # Build a module from the template - name: nf-core modules create run: nf-core --log-file log.txt modules create bpipe --dir nf-core-testpipeline --author @nf-core-bot --label process_low --meta @@ -80,9 +84,6 @@ jobs: # - name: nf-core licences # run: nf-core --log-file log.txt licences nf-core-testpipeline - - name: nf-core sync - run: nf-core --log-file log.txt sync --dir nf-core-testpipeline/ - - name: nf-core schema run: nf-core --log-file log.txt schema build --dir nf-core-testpipeline/ --no-prompts From 04a3b8e3af6f3c40af179cd44040109506f49775 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 00:24:34 +0100 Subject: [PATCH 073/110] YAML quotes for prettier + jinja parsing --- nf_core/module-template/modules/meta.yml | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/nf_core/module-template/modules/meta.yml b/nf_core/module-template/modules/meta.yml index 6acf848a79..b2a2cf7b93 100644 --- a/nf_core/module-template/modules/meta.yml +++ b/nf_core/module-template/modules/meta.yml @@ -4,14 +4,14 @@ description: write your description here keywords: - sort tools: - - { { tool } }: + - "{{ tool }}": ## TODO nf-core: Add a description and other details for the software below - description: { { tool_description } } - homepage: { { tool_doc_url } } - documentation: { { tool_doc_url } } - tool_dev_url: { { tool_dev_url } } + description: "{{ tool_description }}" + homepage: "{{ tool_doc_url }}" + documentation: "{{ tool_doc_url }}" + tool_dev_url: "{{ tool_dev_url }}" doi: "" - licence: { { tool_licence } } + licence: "{{ tool_licence }}" ## TODO nf-core: Add a description of all of the variables used as input input: From 7841b4a619443c94411520e62c71a939b50b3a8e Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 00:30:52 +0100 Subject: [PATCH 074/110] Fix obscure bug that was breaking pytest --- nf_core/module-template/modules/meta.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/module-template/modules/meta.yml b/nf_core/module-template/modules/meta.yml index b2a2cf7b93..23c8b06da8 100644 --- a/nf_core/module-template/modules/meta.yml +++ b/nf_core/module-template/modules/meta.yml @@ -36,7 +36,7 @@ output: description: | Groovy Map containing sample information e.g. [ id:'test', single_end:false ] - # {% endif -%} + # {% endif %} - versions: type: file description: File containing software versions From bf9e7af09574310b6f9cfb4424ba7df10bbae855 Mon Sep 17 00:00:00 2001 From: Edmund Miller Date: Thu, 17 Mar 2022 19:42:56 -0500 Subject: [PATCH 075/110] build(gitpod): Swap markdownlint for prettier --- .gitpod.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.gitpod.yml b/.gitpod.yml index c8d7fce6e3..64994f3c08 100644 --- a/.gitpod.yml +++ b/.gitpod.yml @@ -6,7 +6,7 @@ vscode: extensions: # based on nf-core.nf-core-extensionpack - codezombiech.gitignore # Language support for .gitignore files # - cssho.vscode-svgviewer # SVG viewer - - davidanson.vscode-markdownlint # Markdown/CommonMark linting and style checking for Visual Studio Code + - esbenp.prettier-vscode # Markdown/CommonMark linting and style checking for Visual Studio Code - eamodio.gitlens # Quickly glimpse into whom, why, and when a line or code block was changed - EditorConfig.EditorConfig # override user/workspace settings with settings found in .editorconfig files - Gruntfuggly.todo-tree # Display TODO and FIXME in a tree view in the activity bar @@ -14,4 +14,3 @@ vscode: # - nextflow.nextflow # Nextflow syntax highlighting - oderwat.indent-rainbow # Highlight indentation level - streetsidesoftware.code-spell-checker # Spelling checker for source code - - DavidAnson.vscode-markdownlint # Linter for markdown files From 207d7c6147b85a0b7dd09e9dce019497b1d4bd59 Mon Sep 17 00:00:00 2001 From: Edmund Miller Date: Thu, 17 Mar 2022 19:53:33 -0500 Subject: [PATCH 076/110] style(editorconfig): Use 2 spaces for markdown https://github.com/nf-core/tools/pull/1470#issuecomment-1071028358 --- nf_core/pipeline-template/.editorconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/.editorconfig b/nf_core/pipeline-template/.editorconfig index 95549501a2..d5f52fddc6 100644 --- a/nf_core/pipeline-template/.editorconfig +++ b/nf_core/pipeline-template/.editorconfig @@ -8,7 +8,7 @@ trim_trailing_whitespace = true indent_size = 4 indent_style = space -[*.{yml,yaml}] +[*.{md,yml,yaml}] indent_size = 2 [*.json] From 64994e31e9ee5f0d5c38535d31b509b61c631af7 Mon Sep 17 00:00:00 2001 From: Sarah G <99276764+G-Sarah@users.noreply.github.com> Date: Fri, 18 Mar 2022 09:41:08 +0100 Subject: [PATCH 077/110] Rephrase conda warning Co-authored-by: Phil Ewels --- nf_core/pipeline-template/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/README.md b/nf_core/pipeline-template/README.md index 2355cff750..48b353ee7f 100644 --- a/nf_core/pipeline-template/README.md +++ b/nf_core/pipeline-template/README.md @@ -35,7 +35,7 @@ On release, automated continuous integration tests run the pipeline on a full-si 1. Install [`Nextflow`](https://www.nextflow.io/docs/latest/getstarted.html#installation) (`>=21.10.3`) -2. Install any of [`Docker`](https://docs.docker.com/engine/installation/), [`Singularity`](https://www.sylabs.io/guides/3.0/user-guide/) (you can follow [this tutorial](https://singularity-tutorial.github.io/01-installation/)), [`Podman`](https://podman.io/), [`Shifter`](https://nersc.gitlab.io/development/shifter/how-to-use/) or [`Charliecloud`](https://hpc.github.io/charliecloud/) for full pipeline reproducibility _(you can use [`Conda`](https://conda.io/miniconda.html) for workflow software but please only use it as a last resort for running the pipeline; see [docs](https://nf-co.re/usage/configuration#basic-configuration-profiles))_. +2. Install any of [`Docker`](https://docs.docker.com/engine/installation/), [`Singularity`](https://www.sylabs.io/guides/3.0/user-guide/) (you can follow [this tutorial](https://singularity-tutorial.github.io/01-installation/)), [`Podman`](https://podman.io/), [`Shifter`](https://nersc.gitlab.io/development/shifter/how-to-use/) or [`Charliecloud`](https://hpc.github.io/charliecloud/) for full pipeline reproducibility _(you can use [`Conda`](https://conda.io/miniconda.html) both to install Nextflow itself and also to manage software within pipelines. Please only use it within pipelines as a last resort; see [docs](https://nf-co.re/usage/configuration#basic-configuration-profiles))_. 3. Download the pipeline and test it on a minimal dataset with a single command: From 24e022cd1b0966cea48e23f8629e2cf84ec77344 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 10:39:07 +0100 Subject: [PATCH 078/110] Change indentation rules to minimise diffs --- .prettierrc.yml | 7 +++++++ CHANGELOG.md | 2 ++ nf_core/pipeline-template/.editorconfig | 5 +---- nf_core/pipeline-template/.prettierrc.yml | 7 +++++++ 4 files changed, 17 insertions(+), 4 deletions(-) diff --git a/.prettierrc.yml b/.prettierrc.yml index ef9b1b85a7..2866c05d40 100644 --- a/.prettierrc.yml +++ b/.prettierrc.yml @@ -1,2 +1,9 @@ printWidth: 120 tabWidth: 2 + +# Override tabWidth for JSON - set to 2 spaces +overrides: + - files: + - "*.json" + options: + tabWidth: 4 diff --git a/CHANGELOG.md b/CHANGELOG.md index 85c41a48d5..64daa7b208 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,8 @@ ### Template +- Replace `markdownlint` and `yamllint` with [_Prettier_](https://prettier.io) for linting formatting / whitespace. + ### General - Convert nf-core/tools API / lint test documentation to MyST ([#1245](https://github.com/nf-core/tools/pull/1245)) diff --git a/nf_core/pipeline-template/.editorconfig b/nf_core/pipeline-template/.editorconfig index d5f52fddc6..b6b3190776 100644 --- a/nf_core/pipeline-template/.editorconfig +++ b/nf_core/pipeline-template/.editorconfig @@ -8,12 +8,9 @@ trim_trailing_whitespace = true indent_size = 4 indent_style = space -[*.{md,yml,yaml}] +[*.{md,yml,yaml,html,css,scss,js}] indent_size = 2 -[*.json] -insert_final_newline = unset - # These files are edited and tested upstream in nf-core/modules [/modules/nf-core/**] charset = unset diff --git a/nf_core/pipeline-template/.prettierrc.yml b/nf_core/pipeline-template/.prettierrc.yml index ef9b1b85a7..2866c05d40 100644 --- a/nf_core/pipeline-template/.prettierrc.yml +++ b/nf_core/pipeline-template/.prettierrc.yml @@ -1,2 +1,9 @@ printWidth: 120 tabWidth: 2 + +# Override tabWidth for JSON - set to 2 spaces +overrides: + - files: + - "*.json" + options: + tabWidth: 4 From d668dd25734d9cab369373e7599787c090317420 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 10:42:40 +0100 Subject: [PATCH 079/110] Reformat JSON to 4 spaces with prettier --- .../assets/schema_input.json | 68 +-- nf_core/pipeline-template/modules.json | 28 +- .../pipeline-template/nextflow_schema.json | 512 +++++++++--------- 3 files changed, 304 insertions(+), 304 deletions(-) diff --git a/nf_core/pipeline-template/assets/schema_input.json b/nf_core/pipeline-template/assets/schema_input.json index 8f6b7f047e..509048bd8a 100644 --- a/nf_core/pipeline-template/assets/schema_input.json +++ b/nf_core/pipeline-template/assets/schema_input.json @@ -1,36 +1,36 @@ { - "$schema": "http://json-schema.org/draft-07/schema", - "$id": "https://raw.githubusercontent.com/{{ name }}/master/assets/schema_input.json", - "title": "{{ name }} pipeline - params.input schema", - "description": "Schema for the file provided with params.input", - "type": "array", - "items": { - "type": "object", - "properties": { - "sample": { - "type": "string", - "pattern": "^\\S+$", - "errorMessage": "Sample name must be provided and cannot contain spaces" - }, - "fastq_1": { - "type": "string", - "pattern": "^\\S+\\.f(ast)?q\\.gz$", - "errorMessage": "FastQ file for reads 1 must be provided, cannot contain spaces and must have extension '.fq.gz' or '.fastq.gz'" - }, - "fastq_2": { - "errorMessage": "FastQ file for reads 2 cannot contain spaces and must have extension '.fq.gz' or '.fastq.gz'", - "anyOf": [ - { - "type": "string", - "pattern": "^\\S+\\.f(ast)?q\\.gz$" - }, - { - "type": "string", - "maxLength": 0 - } - ] - } - }, - "required": ["sample", "fastq_1"] - } + "$schema": "http://json-schema.org/draft-07/schema", + "$id": "https://raw.githubusercontent.com/{{ name }}/master/assets/schema_input.json", + "title": "{{ name }} pipeline - params.input schema", + "description": "Schema for the file provided with params.input", + "type": "array", + "items": { + "type": "object", + "properties": { + "sample": { + "type": "string", + "pattern": "^\\S+$", + "errorMessage": "Sample name must be provided and cannot contain spaces" + }, + "fastq_1": { + "type": "string", + "pattern": "^\\S+\\.f(ast)?q\\.gz$", + "errorMessage": "FastQ file for reads 1 must be provided, cannot contain spaces and must have extension '.fq.gz' or '.fastq.gz'" + }, + "fastq_2": { + "errorMessage": "FastQ file for reads 2 cannot contain spaces and must have extension '.fq.gz' or '.fastq.gz'", + "anyOf": [ + { + "type": "string", + "pattern": "^\\S+\\.f(ast)?q\\.gz$" + }, + { + "type": "string", + "maxLength": 0 + } + ] + } + }, + "required": ["sample", "fastq_1"] + } } diff --git a/nf_core/pipeline-template/modules.json b/nf_core/pipeline-template/modules.json index 8c2d2a4071..29a763e917 100644 --- a/nf_core/pipeline-template/modules.json +++ b/nf_core/pipeline-template/modules.json @@ -1,17 +1,17 @@ { - "name": "{{ name }}", - "homePage": "https://github.com/{{ name }}", - "repos": { - "nf-core/modules": { - "custom/dumpsoftwareversions": { - "git_sha": "20d8250d9f39ddb05dfb437603aaf99b5c0b2b41" - }, - "fastqc": { - "git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d" - }, - "multiqc": { - "git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d" - } + "name": "{{ name }}", + "homePage": "https://github.com/{{ name }}", + "repos": { + "nf-core/modules": { + "custom/dumpsoftwareversions": { + "git_sha": "20d8250d9f39ddb05dfb437603aaf99b5c0b2b41" + }, + "fastqc": { + "git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d" + }, + "multiqc": { + "git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d" + } + } } - } } diff --git a/nf_core/pipeline-template/nextflow_schema.json b/nf_core/pipeline-template/nextflow_schema.json index 531c92cfff..084a5de44c 100644 --- a/nf_core/pipeline-template/nextflow_schema.json +++ b/nf_core/pipeline-template/nextflow_schema.json @@ -1,269 +1,269 @@ { - "$schema": "http://json-schema.org/draft-07/schema", - "$id": "https://raw.githubusercontent.com/{{ name }}/master/nextflow_schema.json", - "title": "{{ name }} pipeline parameters", - "description": "{{ description }}", - "type": "object", - "definitions": { - "input_output_options": { - "title": "Input/output options", - "type": "object", - "fa_icon": "fas fa-terminal", - "description": "Define where the pipeline should find input data and save output data.", - "required": ["input", "outdir"], - "properties": { - "input": { - "type": "string", - "format": "file-path", - "mimetype": "text/csv", - "pattern": "^\\S+\\.csv$", - "schema": "assets/schema_input.json", - "description": "Path to comma-separated file containing information about the samples in the experiment.", - "help_text": "You will need to create a design file with information about the samples in your experiment before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row. See [usage docs](https://nf-co.re/{{ short_name }}/usage#samplesheet-input).", - "fa_icon": "fas fa-file-csv" + "$schema": "http://json-schema.org/draft-07/schema", + "$id": "https://raw.githubusercontent.com/{{ name }}/master/nextflow_schema.json", + "title": "{{ name }} pipeline parameters", + "description": "{{ description }}", + "type": "object", + "definitions": { + "input_output_options": { + "title": "Input/output options", + "type": "object", + "fa_icon": "fas fa-terminal", + "description": "Define where the pipeline should find input data and save output data.", + "required": ["input", "outdir"], + "properties": { + "input": { + "type": "string", + "format": "file-path", + "mimetype": "text/csv", + "pattern": "^\\S+\\.csv$", + "schema": "assets/schema_input.json", + "description": "Path to comma-separated file containing information about the samples in the experiment.", + "help_text": "You will need to create a design file with information about the samples in your experiment before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row. See [usage docs](https://nf-co.re/{{ short_name }}/usage#samplesheet-input).", + "fa_icon": "fas fa-file-csv" + }, + "outdir": { + "type": "string", + "format": "directory-path", + "description": "The output directory where the results will be saved. You have to use absolute paths to storage on Cloud infrastructure.", + "fa_icon": "fas fa-folder-open" + }, + "email": { + "type": "string", + "description": "Email address for completion summary.", + "fa_icon": "fas fa-envelope", + "help_text": "Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file (`~/.nextflow/config`) then you don't need to specify this on the command line for every run.", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" + }, + "multiqc_title": { + "type": "string", + "description": "MultiQC report title. Printed as page header, used for filename if not otherwise specified.", + "fa_icon": "fas fa-file-signature" + } + } }, - "outdir": { - "type": "string", - "format": "directory-path", - "description": "The output directory where the results will be saved. You have to use absolute paths to storage on Cloud infrastructure.", - "fa_icon": "fas fa-folder-open" + "reference_genome_options": { + "title": "Reference genome options", + "type": "object", + "fa_icon": "fas fa-dna", + "description": "Reference genome related files and options required for the workflow.", + "properties": { + "genome": { + "type": "string", + "description": "Name of iGenomes reference.", + "fa_icon": "fas fa-book", + "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. `--genome GRCh38`. \n\nSee the [nf-core website docs](https://nf-co.re/usage/reference_genomes) for more details." + }, + "fasta": { + "type": "string", + "format": "file-path", + "mimetype": "text/plain", + "pattern": "^\\S+\\.fn?a(sta)?(\\.gz)?$", + "description": "Path to FASTA genome file.", + "help_text": "This parameter is *mandatory* if `--genome` is not specified. If you don't have a BWA index available this will be generated for you automatically. Combine with `--save_reference` to save BWA index for future runs.", + "fa_icon": "far fa-file-code" + }, + "igenomes_base": { + "type": "string", + "format": "directory-path", + "description": "Directory / URL base for iGenomes references.", + "default": "s3://ngi-igenomes/igenomes", + "fa_icon": "fas fa-cloud-download-alt", + "hidden": true + }, + "igenomes_ignore": { + "type": "boolean", + "description": "Do not load the iGenomes reference config.", + "fa_icon": "fas fa-ban", + "hidden": true, + "help_text": "Do not load `igenomes.config` when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in `igenomes.config`." + } + } }, - "email": { - "type": "string", - "description": "Email address for completion summary.", - "fa_icon": "fas fa-envelope", - "help_text": "Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file (`~/.nextflow/config`) then you don't need to specify this on the command line for every run.", - "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" + "institutional_config_options": { + "title": "Institutional config options", + "type": "object", + "fa_icon": "fas fa-university", + "description": "Parameters used to describe centralised config profiles. These should not be edited.", + "help_text": "The centralised nf-core configuration profiles use a handful of pipeline parameters to describe themselves. This information is then printed to the Nextflow log when you run a pipeline. You should not need to change these values when you run a pipeline.", + "properties": { + "custom_config_version": { + "type": "string", + "description": "Git commit id for Institutional configs.", + "default": "master", + "hidden": true, + "fa_icon": "fas fa-users-cog" + }, + "custom_config_base": { + "type": "string", + "description": "Base directory for Institutional configs.", + "default": "https://raw.githubusercontent.com/nf-core/configs/master", + "hidden": true, + "help_text": "If you're running offline, Nextflow will not be able to fetch the institutional config files from the internet. If you don't need them, then this is not a problem. If you do need them, you should download the files from the repo and tell Nextflow where to find them with this parameter.", + "fa_icon": "fas fa-users-cog" + }, + "config_profile_name": { + "type": "string", + "description": "Institutional config name.", + "hidden": true, + "fa_icon": "fas fa-users-cog" + }, + "config_profile_description": { + "type": "string", + "description": "Institutional config description.", + "hidden": true, + "fa_icon": "fas fa-users-cog" + }, + "config_profile_contact": { + "type": "string", + "description": "Institutional config contact information.", + "hidden": true, + "fa_icon": "fas fa-users-cog" + }, + "config_profile_url": { + "type": "string", + "description": "Institutional config URL link.", + "hidden": true, + "fa_icon": "fas fa-users-cog" + } + } }, - "multiqc_title": { - "type": "string", - "description": "MultiQC report title. Printed as page header, used for filename if not otherwise specified.", - "fa_icon": "fas fa-file-signature" - } - } - }, - "reference_genome_options": { - "title": "Reference genome options", - "type": "object", - "fa_icon": "fas fa-dna", - "description": "Reference genome related files and options required for the workflow.", - "properties": { - "genome": { - "type": "string", - "description": "Name of iGenomes reference.", - "fa_icon": "fas fa-book", - "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. `--genome GRCh38`. \n\nSee the [nf-core website docs](https://nf-co.re/usage/reference_genomes) for more details." - }, - "fasta": { - "type": "string", - "format": "file-path", - "mimetype": "text/plain", - "pattern": "^\\S+\\.fn?a(sta)?(\\.gz)?$", - "description": "Path to FASTA genome file.", - "help_text": "This parameter is *mandatory* if `--genome` is not specified. If you don't have a BWA index available this will be generated for you automatically. Combine with `--save_reference` to save BWA index for future runs.", - "fa_icon": "far fa-file-code" - }, - "igenomes_base": { - "type": "string", - "format": "directory-path", - "description": "Directory / URL base for iGenomes references.", - "default": "s3://ngi-igenomes/igenomes", - "fa_icon": "fas fa-cloud-download-alt", - "hidden": true + "max_job_request_options": { + "title": "Max job request options", + "type": "object", + "fa_icon": "fab fa-acquisitions-incorporated", + "description": "Set the top limit for requested resources for any single job.", + "help_text": "If you are running on a smaller system, a pipeline step requesting more resources than are available may cause the Nextflow to stop the run with an error. These options allow you to cap the maximum resources requested by any single job so that the pipeline will run on your system.\n\nNote that you can not _increase_ the resources requested by any job using these options. For that you will need your own configuration file. See [the nf-core website](https://nf-co.re/usage/configuration) for details.", + "properties": { + "max_cpus": { + "type": "integer", + "description": "Maximum number of CPUs that can be requested for any single job.", + "default": 16, + "fa_icon": "fas fa-microchip", + "hidden": true, + "help_text": "Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. `--max_cpus 1`" + }, + "max_memory": { + "type": "string", + "description": "Maximum amount of memory that can be requested for any single job.", + "default": "128.GB", + "fa_icon": "fas fa-memory", + "pattern": "^\\d+(\\.\\d+)?\\.?\\s*(K|M|G|T)?B$", + "hidden": true, + "help_text": "Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. `--max_memory '8.GB'`" + }, + "max_time": { + "type": "string", + "description": "Maximum amount of time that can be requested for any single job.", + "default": "240.h", + "fa_icon": "far fa-clock", + "pattern": "^(\\d+\\.?\\s*(s|m|h|day)\\s*)+$", + "hidden": true, + "help_text": "Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. `--max_time '2.h'`" + } + } }, - "igenomes_ignore": { - "type": "boolean", - "description": "Do not load the iGenomes reference config.", - "fa_icon": "fas fa-ban", - "hidden": true, - "help_text": "Do not load `igenomes.config` when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in `igenomes.config`." + "generic_options": { + "title": "Generic options", + "type": "object", + "fa_icon": "fas fa-file-import", + "description": "Less common options for the pipeline, typically set in a config file.", + "help_text": "These options are common to all nf-core pipelines and allow you to customise some of the core preferences for how the pipeline runs.\n\nTypically these options would be set in a Nextflow config file loaded for all pipeline runs, such as `~/.nextflow/config`.", + "properties": { + "help": { + "type": "boolean", + "description": "Display help text.", + "fa_icon": "fas fa-question-circle", + "hidden": true + }, + "publish_dir_mode": { + "type": "string", + "default": "copy", + "description": "Method used to save pipeline results to output directory.", + "help_text": "The Nextflow `publishDir` option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See [Nextflow docs](https://www.nextflow.io/docs/latest/process.html#publishdir) for details.", + "fa_icon": "fas fa-copy", + "enum": ["symlink", "rellink", "link", "copy", "copyNoFollow", "move"], + "hidden": true + }, + "email_on_fail": { + "type": "string", + "description": "Email address for completion summary, only when pipeline fails.", + "fa_icon": "fas fa-exclamation-triangle", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$", + "help_text": "An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully.", + "hidden": true + }, + "plaintext_email": { + "type": "boolean", + "description": "Send plain-text email instead of HTML.", + "fa_icon": "fas fa-remove-format", + "hidden": true + }, + "max_multiqc_email_size": { + "type": "string", + "description": "File size limit when attaching MultiQC reports to summary emails.", + "pattern": "^\\d+(\\.\\d+)?\\.?\\s*(K|M|G|T)?B$", + "default": "25.MB", + "fa_icon": "fas fa-file-upload", + "hidden": true + }, + "monochrome_logs": { + "type": "boolean", + "description": "Do not use coloured log outputs.", + "fa_icon": "fas fa-palette", + "hidden": true + }, + "multiqc_config": { + "type": "string", + "description": "Custom config file to supply to MultiQC.", + "fa_icon": "fas fa-cog", + "hidden": true + }, + "tracedir": { + "type": "string", + "description": "Directory to keep pipeline Nextflow logs and reports.", + "default": "${params.outdir}/pipeline_info", + "fa_icon": "fas fa-cogs", + "hidden": true + }, + "validate_params": { + "type": "boolean", + "description": "Boolean whether to validate parameters against the schema at runtime", + "default": true, + "fa_icon": "fas fa-check-square", + "hidden": true + }, + "show_hidden_params": { + "type": "boolean", + "fa_icon": "far fa-eye-slash", + "description": "Show all params when using `--help`", + "hidden": true, + "help_text": "By default, parameters set as _hidden_ in the schema are not shown on the command line when a user runs with `--help`. Specifying this option will tell the pipeline to show all parameters." + }, + "enable_conda": { + "type": "boolean", + "description": "Run this workflow with Conda. You can also use '-profile conda' instead of providing this parameter.", + "hidden": true, + "fa_icon": "fas fa-bacon" + } + } } - } }, - "institutional_config_options": { - "title": "Institutional config options", - "type": "object", - "fa_icon": "fas fa-university", - "description": "Parameters used to describe centralised config profiles. These should not be edited.", - "help_text": "The centralised nf-core configuration profiles use a handful of pipeline parameters to describe themselves. This information is then printed to the Nextflow log when you run a pipeline. You should not need to change these values when you run a pipeline.", - "properties": { - "custom_config_version": { - "type": "string", - "description": "Git commit id for Institutional configs.", - "default": "master", - "hidden": true, - "fa_icon": "fas fa-users-cog" + "allOf": [ + { + "$ref": "#/definitions/input_output_options" }, - "custom_config_base": { - "type": "string", - "description": "Base directory for Institutional configs.", - "default": "https://raw.githubusercontent.com/nf-core/configs/master", - "hidden": true, - "help_text": "If you're running offline, Nextflow will not be able to fetch the institutional config files from the internet. If you don't need them, then this is not a problem. If you do need them, you should download the files from the repo and tell Nextflow where to find them with this parameter.", - "fa_icon": "fas fa-users-cog" + { + "$ref": "#/definitions/reference_genome_options" }, - "config_profile_name": { - "type": "string", - "description": "Institutional config name.", - "hidden": true, - "fa_icon": "fas fa-users-cog" + { + "$ref": "#/definitions/institutional_config_options" }, - "config_profile_description": { - "type": "string", - "description": "Institutional config description.", - "hidden": true, - "fa_icon": "fas fa-users-cog" + { + "$ref": "#/definitions/max_job_request_options" }, - "config_profile_contact": { - "type": "string", - "description": "Institutional config contact information.", - "hidden": true, - "fa_icon": "fas fa-users-cog" - }, - "config_profile_url": { - "type": "string", - "description": "Institutional config URL link.", - "hidden": true, - "fa_icon": "fas fa-users-cog" + { + "$ref": "#/definitions/generic_options" } - } - }, - "max_job_request_options": { - "title": "Max job request options", - "type": "object", - "fa_icon": "fab fa-acquisitions-incorporated", - "description": "Set the top limit for requested resources for any single job.", - "help_text": "If you are running on a smaller system, a pipeline step requesting more resources than are available may cause the Nextflow to stop the run with an error. These options allow you to cap the maximum resources requested by any single job so that the pipeline will run on your system.\n\nNote that you can not _increase_ the resources requested by any job using these options. For that you will need your own configuration file. See [the nf-core website](https://nf-co.re/usage/configuration) for details.", - "properties": { - "max_cpus": { - "type": "integer", - "description": "Maximum number of CPUs that can be requested for any single job.", - "default": 16, - "fa_icon": "fas fa-microchip", - "hidden": true, - "help_text": "Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. `--max_cpus 1`" - }, - "max_memory": { - "type": "string", - "description": "Maximum amount of memory that can be requested for any single job.", - "default": "128.GB", - "fa_icon": "fas fa-memory", - "pattern": "^\\d+(\\.\\d+)?\\.?\\s*(K|M|G|T)?B$", - "hidden": true, - "help_text": "Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. `--max_memory '8.GB'`" - }, - "max_time": { - "type": "string", - "description": "Maximum amount of time that can be requested for any single job.", - "default": "240.h", - "fa_icon": "far fa-clock", - "pattern": "^(\\d+\\.?\\s*(s|m|h|day)\\s*)+$", - "hidden": true, - "help_text": "Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. `--max_time '2.h'`" - } - } - }, - "generic_options": { - "title": "Generic options", - "type": "object", - "fa_icon": "fas fa-file-import", - "description": "Less common options for the pipeline, typically set in a config file.", - "help_text": "These options are common to all nf-core pipelines and allow you to customise some of the core preferences for how the pipeline runs.\n\nTypically these options would be set in a Nextflow config file loaded for all pipeline runs, such as `~/.nextflow/config`.", - "properties": { - "help": { - "type": "boolean", - "description": "Display help text.", - "fa_icon": "fas fa-question-circle", - "hidden": true - }, - "publish_dir_mode": { - "type": "string", - "default": "copy", - "description": "Method used to save pipeline results to output directory.", - "help_text": "The Nextflow `publishDir` option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See [Nextflow docs](https://www.nextflow.io/docs/latest/process.html#publishdir) for details.", - "fa_icon": "fas fa-copy", - "enum": ["symlink", "rellink", "link", "copy", "copyNoFollow", "move"], - "hidden": true - }, - "email_on_fail": { - "type": "string", - "description": "Email address for completion summary, only when pipeline fails.", - "fa_icon": "fas fa-exclamation-triangle", - "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$", - "help_text": "An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully.", - "hidden": true - }, - "plaintext_email": { - "type": "boolean", - "description": "Send plain-text email instead of HTML.", - "fa_icon": "fas fa-remove-format", - "hidden": true - }, - "max_multiqc_email_size": { - "type": "string", - "description": "File size limit when attaching MultiQC reports to summary emails.", - "pattern": "^\\d+(\\.\\d+)?\\.?\\s*(K|M|G|T)?B$", - "default": "25.MB", - "fa_icon": "fas fa-file-upload", - "hidden": true - }, - "monochrome_logs": { - "type": "boolean", - "description": "Do not use coloured log outputs.", - "fa_icon": "fas fa-palette", - "hidden": true - }, - "multiqc_config": { - "type": "string", - "description": "Custom config file to supply to MultiQC.", - "fa_icon": "fas fa-cog", - "hidden": true - }, - "tracedir": { - "type": "string", - "description": "Directory to keep pipeline Nextflow logs and reports.", - "default": "${params.outdir}/pipeline_info", - "fa_icon": "fas fa-cogs", - "hidden": true - }, - "validate_params": { - "type": "boolean", - "description": "Boolean whether to validate parameters against the schema at runtime", - "default": true, - "fa_icon": "fas fa-check-square", - "hidden": true - }, - "show_hidden_params": { - "type": "boolean", - "fa_icon": "far fa-eye-slash", - "description": "Show all params when using `--help`", - "hidden": true, - "help_text": "By default, parameters set as _hidden_ in the schema are not shown on the command line when a user runs with `--help`. Specifying this option will tell the pipeline to show all parameters." - }, - "enable_conda": { - "type": "boolean", - "description": "Run this workflow with Conda. You can also use '-profile conda' instead of providing this parameter.", - "hidden": true, - "fa_icon": "fas fa-bacon" - } - } - } - }, - "allOf": [ - { - "$ref": "#/definitions/input_output_options" - }, - { - "$ref": "#/definitions/reference_genome_options" - }, - { - "$ref": "#/definitions/institutional_config_options" - }, - { - "$ref": "#/definitions/max_job_request_options" - }, - { - "$ref": "#/definitions/generic_options" - } - ] + ] } From 1405d48ca0fe0071efc8e025dba99c5a2971ebdc Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 10:45:20 +0100 Subject: [PATCH 080/110] Update custom/dumpsoftwareversions module --- nf_core/pipeline-template/modules.json | 2 +- .../nf-core/modules/custom/dumpsoftwareversions/main.nf | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/modules.json b/nf_core/pipeline-template/modules.json index 29a763e917..8a6a36ec3b 100644 --- a/nf_core/pipeline-template/modules.json +++ b/nf_core/pipeline-template/modules.json @@ -4,7 +4,7 @@ "repos": { "nf-core/modules": { "custom/dumpsoftwareversions": { - "git_sha": "20d8250d9f39ddb05dfb437603aaf99b5c0b2b41" + "git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d" }, "fastqc": { "git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d" diff --git a/nf_core/pipeline-template/modules/nf-core/modules/custom/dumpsoftwareversions/main.nf b/nf_core/pipeline-template/modules/nf-core/modules/custom/dumpsoftwareversions/main.nf index 934bb46723..327d510056 100644 --- a/nf_core/pipeline-template/modules/nf-core/modules/custom/dumpsoftwareversions/main.nf +++ b/nf_core/pipeline-template/modules/nf-core/modules/custom/dumpsoftwareversions/main.nf @@ -15,6 +15,9 @@ process CUSTOM_DUMPSOFTWAREVERSIONS { path "software_versions_mqc.yml", emit: mqc_yml path "versions.yml" , emit: versions + when: + task.ext.when == null || task.ext.when + script: def args = task.ext.args ?: '' template 'dumpsoftwareversions.py' From 3d07d698dc74574d23be0b31e188c5f410b2c2d9 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 10:52:11 +0100 Subject: [PATCH 081/110] Fix new MultiQC linting for report comment --- nf_core/lint/multiqc_config.py | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/nf_core/lint/multiqc_config.py b/nf_core/lint/multiqc_config.py index aa9ce5d33d..337d99edcd 100644 --- a/nf_core/lint/multiqc_config.py +++ b/nf_core/lint/multiqc_config.py @@ -70,14 +70,12 @@ def multiqc_config(self): assert "report_comment" in mqc_yml assert ( mqc_yml["report_comment"].strip() - == f'This report has been generated by the nf-core/{self.pipeline_name} ' - "analysis pipeline. For information about how to interpret these results, please see the " - 'documentation.' + == f'This report has been generated by the nf-core/{self.pipeline_name} analysis pipeline. For information about how to interpret these results, please see the documentation.' ) except (AssertionError, KeyError, TypeError): - warned.append("'assets/multiqc_config.yaml' does not contain a matching report_comment.") + failed.append("'assets/multiqc_config.yaml' does not contain a matching 'report_comment'.") else: - passed.append("'assets/multiqc_config.yaml' contains a matching report_comment.") + passed.append("'assets/multiqc_config.yaml' contains a matching 'report_comment'.") # Check that export_plots is activated try: @@ -88,4 +86,4 @@ def multiqc_config(self): else: passed.append("'assets/multiqc_config.yaml' contains 'export_plots: true'.") - return {"passed": passed, "warned": warned, "failed": failed} + return {"passed": passed, "failed": failed} From 06275d1c9c42c63a9137a257bfc3887980216286 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 11:08:26 +0100 Subject: [PATCH 082/110] Use .editorconfig for indentation config --- .editorconfig | 12 ++++++++++++ .prettierrc.yml | 8 -------- nf_core/pipeline-template/.prettierrc.yml | 8 -------- 3 files changed, 12 insertions(+), 16 deletions(-) create mode 100644 .editorconfig diff --git a/.editorconfig b/.editorconfig new file mode 100644 index 0000000000..c29b6c7eee --- /dev/null +++ b/.editorconfig @@ -0,0 +1,12 @@ +root = true + +[*] +charset = utf-8 +end_of_line = lf +insert_final_newline = true +trim_trailing_whitespace = true +indent_size = 4 +indent_style = space + +[*.{md,yml,yaml,html,css,scss,js}] +indent_size = 2 diff --git a/.prettierrc.yml b/.prettierrc.yml index 2866c05d40..c81f9a7660 100644 --- a/.prettierrc.yml +++ b/.prettierrc.yml @@ -1,9 +1 @@ printWidth: 120 -tabWidth: 2 - -# Override tabWidth for JSON - set to 2 spaces -overrides: - - files: - - "*.json" - options: - tabWidth: 4 diff --git a/nf_core/pipeline-template/.prettierrc.yml b/nf_core/pipeline-template/.prettierrc.yml index 2866c05d40..c81f9a7660 100644 --- a/nf_core/pipeline-template/.prettierrc.yml +++ b/nf_core/pipeline-template/.prettierrc.yml @@ -1,9 +1 @@ printWidth: 120 -tabWidth: 2 - -# Override tabWidth for JSON - set to 2 spaces -overrides: - - files: - - "*.json" - options: - tabWidth: 4 From a6dbdb1cb2ccdcd4d16784dc7774f1c8dd21ef74 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 11:43:46 +0100 Subject: [PATCH 083/110] Add a default for params.outdir on the test config --- nf_core/pipeline-template/conf/test.config | 2 ++ nf_core/pipeline-template/conf/test_full.config | 2 ++ 2 files changed, 4 insertions(+) diff --git a/nf_core/pipeline-template/conf/test.config b/nf_core/pipeline-template/conf/test.config index 1e2ea2e6bb..46c1ebe9c0 100644 --- a/nf_core/pipeline-template/conf/test.config +++ b/nf_core/pipeline-template/conf/test.config @@ -19,6 +19,8 @@ params { max_memory = '6.GB' max_time = '6.h' + outdir = "./results" + // Input data // TODO nf-core: Specify the paths to your test data on nf-core/test-datasets // TODO nf-core: Give any required params for the test so that command line flags are not needed diff --git a/nf_core/pipeline-template/conf/test_full.config b/nf_core/pipeline-template/conf/test_full.config index 87e4c96289..e5543221cc 100644 --- a/nf_core/pipeline-template/conf/test_full.config +++ b/nf_core/pipeline-template/conf/test_full.config @@ -14,6 +14,8 @@ params { config_profile_name = 'Full test profile' config_profile_description = 'Full test dataset to check pipeline function' + outdir = "./results" + // Input data for full size test // TODO nf-core: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA) // TODO nf-core: Give any required params for the test so that command line flags are not needed From 194f03f2496a590a642629d21b0aac96ac307da5 Mon Sep 17 00:00:00 2001 From: Edmund Miller Date: Fri, 18 Mar 2022 05:52:14 -0500 Subject: [PATCH 084/110] style: yaml => yml --- .../assets/{multiqc_config.yaml => multiqc_config.yml} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename nf_core/pipeline-template/assets/{multiqc_config.yaml => multiqc_config.yml} (100%) diff --git a/nf_core/pipeline-template/assets/multiqc_config.yaml b/nf_core/pipeline-template/assets/multiqc_config.yml similarity index 100% rename from nf_core/pipeline-template/assets/multiqc_config.yaml rename to nf_core/pipeline-template/assets/multiqc_config.yml From 136c34b4ed41725cc4659a26c5098e6940162d06 Mon Sep 17 00:00:00 2001 From: Edmund Miller Date: Fri, 18 Mar 2022 06:02:00 -0500 Subject: [PATCH 085/110] fix: Update linting for yaml => yml --- nf_core/lint/files_exist.py | 4 ++-- nf_core/lint/multiqc_config.py | 16 ++++++++-------- .../pipeline-template/.github/CONTRIBUTING.md | 2 +- nf_core/pipeline-template/workflows/pipeline.nf | 2 +- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/nf_core/lint/files_exist.py b/nf_core/lint/files_exist.py index f00c676196..f1c372672d 100644 --- a/nf_core/lint/files_exist.py +++ b/nf_core/lint/files_exist.py @@ -62,7 +62,7 @@ def files_exist(self): .. code-block:: bash main.nf - assets/multiqc_config.yaml + assets/multiqc_config.yml conf/base.config conf/igenomes.config .github/workflows/awstest.yml @@ -144,7 +144,7 @@ def files_exist(self): files_warn = [ ["main.nf"], - [os.path.join("assets", "multiqc_config.yaml")], + [os.path.join("assets", "multiqc_config.yml")], [os.path.join("conf", "base.config")], [os.path.join("conf", "igenomes.config")], [os.path.join(".github", "workflows", "awstest.yml")], diff --git a/nf_core/lint/multiqc_config.py b/nf_core/lint/multiqc_config.py index 337d99edcd..36c3647fd3 100644 --- a/nf_core/lint/multiqc_config.py +++ b/nf_core/lint/multiqc_config.py @@ -25,11 +25,11 @@ def multiqc_config(self): warned = [] failed = [] - fn = os.path.join(self.wf_path, "assets", "multiqc_config.yaml") + fn = os.path.join(self.wf_path, "assets", "multiqc_config.yml") # Return a failed status if we can't find the file if not os.path.isfile(fn): - return {"ignored": ["'assets/multiqc_config.yaml' not found"]} + return {"ignored": ["'assets/multiqc_config.yml' not found"]} try: with open(fn, "r") as fh: @@ -61,9 +61,9 @@ def multiqc_config(self): orders.values() ), f"Section software_versions should have the second lowest order" except (AssertionError, KeyError, TypeError) as e: - failed.append(f"'assets/multiqc_config.yaml' does not meet requirements: {e}") + failed.append(f"'assets/multiqc_config.yml' does not meet requirements: {e}") else: - passed.append("'assets/multiqc_config.yaml' follows the ordering scheme of the minimally required plugins.") + passed.append("'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.") # Check that the minimum plugins exist and are coming first in the summary try: @@ -73,17 +73,17 @@ def multiqc_config(self): == f'This report has been generated by the nf-core/{self.pipeline_name} analysis pipeline. For information about how to interpret these results, please see the documentation.' ) except (AssertionError, KeyError, TypeError): - failed.append("'assets/multiqc_config.yaml' does not contain a matching 'report_comment'.") + failed.append("'assets/multiqc_config.yml' does not contain a matching 'report_comment'.") else: - passed.append("'assets/multiqc_config.yaml' contains a matching 'report_comment'.") + passed.append("'assets/multiqc_config.yml' contains a matching 'report_comment'.") # Check that export_plots is activated try: assert "export_plots" in mqc_yml assert mqc_yml["export_plots"] == True except (AssertionError, KeyError, TypeError): - failed.append("'assets/multiqc_config.yaml' does not contain 'export_plots: true'.") + failed.append("'assets/multiqc_config.yml' does not contain 'export_plots: true'.") else: - passed.append("'assets/multiqc_config.yaml' contains 'export_plots: true'.") + passed.append("'assets/multiqc_config.yml' contains 'export_plots: true'.") return {"passed": passed, "failed": failed} diff --git a/nf_core/pipeline-template/.github/CONTRIBUTING.md b/nf_core/pipeline-template/.github/CONTRIBUTING.md index 2524037294..9005909712 100644 --- a/nf_core/pipeline-template/.github/CONTRIBUTING.md +++ b/nf_core/pipeline-template/.github/CONTRIBUTING.md @@ -73,7 +73,7 @@ If you wish to contribute a new step, please use the following coding standards: 6. Add sanity checks and validation for all relevant parameters. 7. Perform local tests to validate that the new code works as expected. 8. If applicable, add a new test command in `.github/workflow/ci.yml`. -9. Update MultiQC config `assets/multiqc_config.yaml` so relevant suffixes, file name clean up and module plots are in the appropriate order. If applicable, add a [MultiQC](https://https://multiqc.info/) module. +9. Update MultiQC config `assets/multiqc_config.yml` so relevant suffixes, file name clean up and module plots are in the appropriate order. If applicable, add a [MultiQC](https://https://multiqc.info/) module. 10. Add a description of the output files and if relevant any appropriate images from the MultiQC report to `docs/output.md`. ### Default values diff --git a/nf_core/pipeline-template/workflows/pipeline.nf b/nf_core/pipeline-template/workflows/pipeline.nf index 6ac463f8e8..97e80b7c3a 100644 --- a/nf_core/pipeline-template/workflows/pipeline.nf +++ b/nf_core/pipeline-template/workflows/pipeline.nf @@ -23,7 +23,7 @@ if (params.input) { ch_input = file(params.input) } else { exit 1, 'Input sample ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ */ -ch_multiqc_config = file("$projectDir/assets/multiqc_config.yaml", checkIfExists: true) +ch_multiqc_config = file("$projectDir/assets/multiqc_config.yml", checkIfExists: true) ch_multiqc_custom_config = params.multiqc_config ? Channel.fromPath(params.multiqc_config) : Channel.empty() /* From db3c429b10f88d893babf837857bdd4a57957fb5 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 13:47:59 +0100 Subject: [PATCH 086/110] Remove params.outdir from test configs again --- nf_core/pipeline-template/conf/test.config | 2 -- nf_core/pipeline-template/conf/test_full.config | 2 -- 2 files changed, 4 deletions(-) diff --git a/nf_core/pipeline-template/conf/test.config b/nf_core/pipeline-template/conf/test.config index 46c1ebe9c0..1e2ea2e6bb 100644 --- a/nf_core/pipeline-template/conf/test.config +++ b/nf_core/pipeline-template/conf/test.config @@ -19,8 +19,6 @@ params { max_memory = '6.GB' max_time = '6.h' - outdir = "./results" - // Input data // TODO nf-core: Specify the paths to your test data on nf-core/test-datasets // TODO nf-core: Give any required params for the test so that command line flags are not needed diff --git a/nf_core/pipeline-template/conf/test_full.config b/nf_core/pipeline-template/conf/test_full.config index e5543221cc..87e4c96289 100644 --- a/nf_core/pipeline-template/conf/test_full.config +++ b/nf_core/pipeline-template/conf/test_full.config @@ -14,8 +14,6 @@ params { config_profile_name = 'Full test profile' config_profile_description = 'Full test dataset to check pipeline function' - outdir = "./results" - // Input data for full size test // TODO nf-core: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA) // TODO nf-core: Give any required params for the test so that command line flags are not needed From 85820d009f0d8bf0bb498f25dbbf04968affe5f3 Mon Sep 17 00:00:00 2001 From: Sarah Guinchard Date: Fri, 18 Mar 2022 14:40:44 +0100 Subject: [PATCH 087/110] Added details about -resume --- nf_core/pipeline-template/docs/usage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/docs/usage.md b/nf_core/pipeline-template/docs/usage.md index 7169b46082..e5a795a943 100644 --- a/nf_core/pipeline-template/docs/usage.md +++ b/nf_core/pipeline-template/docs/usage.md @@ -124,7 +124,7 @@ If `-profile` is not specified, the pipeline will run locally and expect all sof ### `-resume` -Specify this when restarting a pipeline. Nextflow will used cached results from any pipeline steps where the inputs are the same, continuing from where it got to previously. +Specify this when restarting a pipeline. Nextflow will used cached results from any pipeline steps where the inputs are the same, continuing from where it got to previously. For input to be considered the same, not only the names must be the names must be identical but the files' contents as well. For more info about this parameter, see [this blog post](https://www.nextflow.io/blog/2019/demystifying-nextflow-resume.html). You can also supply a run name to resume a specific run: `-resume [run-name]`. Use the `nextflow log` command to show previous run names. From 362be3bdfb4cd43a72f7275cb324573b14ef50f7 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 15:03:34 +0100 Subject: [PATCH 088/110] Fix GitPod docker builds - Split two docker build commands into their own action steps - Bugfix: use --file instead of specfying the context incorrectly --- .github/workflows/push_dockerhub_dev.yml | 11 ++++++----- .github/workflows/push_dockerhub_release.yml | 11 ++++++----- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/.github/workflows/push_dockerhub_dev.yml b/.github/workflows/push_dockerhub_dev.yml index beab5cd536..5ba3e7fd8f 100644 --- a/.github/workflows/push_dockerhub_dev.yml +++ b/.github/workflows/push_dockerhub_dev.yml @@ -20,12 +20,13 @@ jobs: - name: Check out code uses: actions/checkout@v2 - - name: Build new docker image - run: | - docker build --no-cache . -t nfcore/tools:dev - docker build --no-cache nf_core/gitpod/gitpod.Dockerfile -t nfcore/gitpod:dev + - name: Build nfcore/tools:dev docker image + run: docker build --no-cache . -t nfcore/tools:dev + + - name: Build nfcore/gitpod:dev docker image + run: docker build --no-cache . --file nf_core/gitpod/gitpod.Dockerfile -t nfcore/gitpod:dev - - name: Push Docker image to DockerHub (dev) + - name: Push Docker images to DockerHub (dev) run: | echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin docker push nfcore/tools:dev diff --git a/.github/workflows/push_dockerhub_release.yml b/.github/workflows/push_dockerhub_release.yml index 11d3124767..c2dcf4b146 100644 --- a/.github/workflows/push_dockerhub_release.yml +++ b/.github/workflows/push_dockerhub_release.yml @@ -20,12 +20,13 @@ jobs: - name: Check out code uses: actions/checkout@v2 - - name: Build new docker image - run: | - docker build --no-cache . -t nfcore/tools:latest - docker build --no-cache nf_core/gitpod/gitpod.Dockerfile -t nfcore/gitpod:latest + - name: Build nfcore/tools:latest docker image + run: docker build --no-cache . -t nfcore/tools:latest + + - name: Build nfcore/gitpod:latest docker image + run: docker build --no-cache . --file nf_core/gitpod/gitpod.Dockerfile -t nfcore/gitpod:latest - - name: Push Docker image to DockerHub (release) + - name: Push Docker images to DockerHub (release) run: | echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin docker push nfcore/tools:latest From 07218a2ce9b4804387a72974e9bd9e31b08ff5c1 Mon Sep 17 00:00:00 2001 From: JoseEspinosa Date: Fri, 18 Mar 2022 15:05:27 +0100 Subject: [PATCH 089/110] Add versions.yml to the test.yml on the modules template --- CHANGELOG.md | 1 + nf_core/module-template/tests/test.yml | 2 ++ 2 files changed, 3 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 64daa7b208..c1ca2207c2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,7 @@ ### Template - Replace `markdownlint` and `yamllint` with [_Prettier_](https://prettier.io) for linting formatting / whitespace. +- Add md5sum check of `versions.yml` to `test.yml` on the modules template. ### General diff --git a/nf_core/module-template/tests/test.yml b/nf_core/module-template/tests/test.yml index 9b0c62356d..b0eb645846 100644 --- a/nf_core/module-template/tests/test.yml +++ b/nf_core/module-template/tests/test.yml @@ -10,3 +10,5 @@ files: - path: "output/{{ tool }}/test.bam" md5sum: e667c7caad0bc4b7ac383fd023c654fc + - path: output/{{ tool }}/versions.yml + md5sum: a01fe51bc4c6a3a6226fbf77b2c7cf3b From cb4f3c3f3215db6daa6fca6026ad2148149c7288 Mon Sep 17 00:00:00 2001 From: Rike Date: Fri, 18 Mar 2022 15:15:13 +0100 Subject: [PATCH 090/110] don't bother with this (prettier & editorconfig can't agree on a format for this one line)just make 2 sentences --- nf_core/pipeline-template/.github/CONTRIBUTING.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/nf_core/pipeline-template/.github/CONTRIBUTING.md b/nf_core/pipeline-template/.github/CONTRIBUTING.md index 9005909712..3a89788cba 100644 --- a/nf_core/pipeline-template/.github/CONTRIBUTING.md +++ b/nf_core/pipeline-template/.github/CONTRIBUTING.md @@ -15,8 +15,7 @@ Contributions to the code are even more welcome ;) If you'd like to write some code for {{ name }}, the standard workflow is as follows: -1. Check that there isn't already an issue about your idea in the [{{ name }} issues](https://github.com/{{ name }}/issues) to avoid duplicating work - - If there isn't one already, please create one so that others know you're working on this +1. Check that there isn't already an issue about your idea in the [{{ name }} issues](https://github.com/{{ name }}/issues) to avoid duplicating work. If there isn't one already, please create one so that others know you're working on this 2. [Fork](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) the [{{ name }} repository](https://github.com/{{ name }}) to your GitHub account 3. Make the necessary changes / additions within your forked repository following [Pipeline conventions](#pipeline-contribution-conventions) 4. Use `nf-core schema build` and add any new parameters to the pipeline JSON schema (requires [nf-core tools](https://github.com/nf-core/tools) >= 1.10). From 342997ce119b99561a4c18296ff5d30a30f7ba9d Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 15:19:57 +0100 Subject: [PATCH 091/110] Update nf_core/pipeline-template/docs/usage.md Co-authored-by: Gisela Gabernet --- nf_core/pipeline-template/docs/usage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/docs/usage.md b/nf_core/pipeline-template/docs/usage.md index e5a795a943..92a46d73bd 100644 --- a/nf_core/pipeline-template/docs/usage.md +++ b/nf_core/pipeline-template/docs/usage.md @@ -124,7 +124,7 @@ If `-profile` is not specified, the pipeline will run locally and expect all sof ### `-resume` -Specify this when restarting a pipeline. Nextflow will used cached results from any pipeline steps where the inputs are the same, continuing from where it got to previously. For input to be considered the same, not only the names must be the names must be identical but the files' contents as well. For more info about this parameter, see [this blog post](https://www.nextflow.io/blog/2019/demystifying-nextflow-resume.html). +Specify this when restarting a pipeline. Nextflow will use cached results from any pipeline steps where the inputs are the same, continuing from where it got to previously. For input to be considered the same, not only the names must be identical but the files' contents as well. For more info about this parameter, see [this blog post](https://www.nextflow.io/blog/2019/demystifying-nextflow-resume.html). You can also supply a run name to resume a specific run: `-resume [run-name]`. Use the `nextflow log` command to show previous run names. From 4d5079441eadcef6d07da5be8898db4819ace1e8 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 16:38:29 +0100 Subject: [PATCH 092/110] Fix gitpod file permissions --- nf_core/gitpod/gitpod.Dockerfile | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/nf_core/gitpod/gitpod.Dockerfile b/nf_core/gitpod/gitpod.Dockerfile index aa8bfe682d..7cf45f5e06 100644 --- a/nf_core/gitpod/gitpod.Dockerfile +++ b/nf_core/gitpod/gitpod.Dockerfile @@ -9,8 +9,14 @@ RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh & ENV PATH="/opt/conda/bin:$PATH" -RUN chown -R gitpod:gitpod /opt/conda +# Add the nf-core source files to the image +COPY . /usr/src/nf_core +WORKDIR /usr/src/nf_core + +# Change ownership for gitpod +RUN chown -R gitpod:gitpod /opt/conda /usr/src/nf_core +# Change user to gitpod USER gitpod # Install nextflow, nf-core, Mamba, and pytest-workflow @@ -19,20 +25,15 @@ RUN conda update -n base -c defaults conda && \ conda config --add channels bioconda && \ conda config --add channels conda-forge && \ conda install \ - openjdk=11.0.13 \ - nextflow=21.10.6 \ - pytest-workflow=1.6.0 \ - mamba=0.22.1 \ - pip=22.0.4 \ - black=22.1.0 \ - yamllint=1.26.3 \ - -n base && \ + openjdk=11.0.13 \ + nextflow=21.10.6 \ + pytest-workflow=1.6.0 \ + mamba=0.22.1 \ + pip=22.0.4 \ + black=22.1.0 \ + -n base && \ nextflow self-update && \ conda clean --all -f -y -# Add the nf-core source files to the image -COPY . /usr/src/nf_core -WORKDIR /usr/src/nf_core - # Install nf-core -RUN python -m pip install . \ No newline at end of file +RUN python -m pip install . From e3a43484b3dae5ac5acdc54a9b6a70938d26df52 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 16:43:50 +0100 Subject: [PATCH 093/110] Put indentation back --- nf_core/gitpod/gitpod.Dockerfile | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/nf_core/gitpod/gitpod.Dockerfile b/nf_core/gitpod/gitpod.Dockerfile index 7cf45f5e06..0388071d11 100644 --- a/nf_core/gitpod/gitpod.Dockerfile +++ b/nf_core/gitpod/gitpod.Dockerfile @@ -25,13 +25,13 @@ RUN conda update -n base -c defaults conda && \ conda config --add channels bioconda && \ conda config --add channels conda-forge && \ conda install \ - openjdk=11.0.13 \ - nextflow=21.10.6 \ - pytest-workflow=1.6.0 \ - mamba=0.22.1 \ - pip=22.0.4 \ - black=22.1.0 \ - -n base && \ + openjdk=11.0.13 \ + nextflow=21.10.6 \ + pytest-workflow=1.6.0 \ + mamba=0.22.1 \ + pip=22.0.4 \ + black=22.1.0 \ + -n base && \ nextflow self-update && \ conda clean --all -f -y From e4577badd08de2350287c08f05858b98463b677b Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 15:22:14 +0100 Subject: [PATCH 094/110] Add editorconfig linting to tools CI tests --- .github/workflows/create-lint-wf.yml | 9 ++++++++- .github/workflows/lint-code.yml | 13 +++++++++++++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/.github/workflows/create-lint-wf.yml b/.github/workflows/create-lint-wf.yml index d26333540b..647b140c15 100644 --- a/.github/workflows/create-lint-wf.yml +++ b/.github/workflows/create-lint-wf.yml @@ -47,12 +47,16 @@ jobs: wget -qO- get.nextflow.io | bash sudo mv nextflow /usr/local/bin/ - # Install the Prettier linting stuff + # Install the Prettier linting tools - uses: actions/setup-node@v2 - name: Install Prettier run: npm install -g prettier + # Install the editorconfig linting tools + - name: Install editorconfig-checker + run: npm install -g editorconfig-checker + # Build a pipeline from the template - name: nf-core create run: nf-core --log-file log.txt create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface" @@ -69,6 +73,9 @@ jobs: - name: Run Prettier --check run: prettier --check nf-core-testpipeline + - name: Run ECLint check + run: editorconfig-checker -exclude README.md $(git ls-files | grep -v test) + # Update modules to the latest version - name: nf-core modules update run: nf-core --log-file log.txt modules update --dir nf-core-testpipeline --all --no-preview diff --git a/.github/workflows/lint-code.yml b/.github/workflows/lint-code.yml index 3a40ba8f2b..d49ed8c912 100644 --- a/.github/workflows/lint-code.yml +++ b/.github/workflows/lint-code.yml @@ -2,6 +2,19 @@ name: Lint tools code formatting on: [push, pull_request] jobs: + EditorConfig: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + + - uses: actions/setup-node@v2 + + - name: Install editorconfig-checker + run: npm install -g editorconfig-checker + + - name: Run ECLint check + run: editorconfig-checker -exclude README.md $(git ls-files | grep -v test) + Prettier: runs-on: ubuntu-latest steps: From 3d2d23a8788daae4d1db7f18c5ad2b628f1a4c82 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 15:43:50 +0100 Subject: [PATCH 095/110] eclint - lint created pipeline - list relevant files only --- .github/workflows/create-lint-wf.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/create-lint-wf.yml b/.github/workflows/create-lint-wf.yml index 647b140c15..6804d18014 100644 --- a/.github/workflows/create-lint-wf.yml +++ b/.github/workflows/create-lint-wf.yml @@ -74,7 +74,7 @@ jobs: run: prettier --check nf-core-testpipeline - name: Run ECLint check - run: editorconfig-checker -exclude README.md $(git ls-files | grep -v test) + run: editorconfig-checker -exclude README.md $(find nf-core-testpipeline/.* -type f | grep -v .git) # Update modules to the latest version - name: nf-core modules update From 43b2747dd79a3742826f35987e968525e5944f0b Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 15:55:25 +0100 Subject: [PATCH 096/110] Run editorconfig check only on files not covered by another linter --- .github/workflows/create-lint-wf.yml | 2 +- .github/workflows/lint-code.yml | 3 ++- Dockerfile | 20 ++++++++++---------- MANIFEST.in | 2 +- docs/api/Makefile | 4 ++-- nf_core/gitpod/gitpod.Dockerfile | 18 +++++++++--------- pyproject.toml | 10 +++++----- requirements-dev.txt | 2 +- 8 files changed, 31 insertions(+), 30 deletions(-) diff --git a/.github/workflows/create-lint-wf.yml b/.github/workflows/create-lint-wf.yml index 6804d18014..d916112e6a 100644 --- a/.github/workflows/create-lint-wf.yml +++ b/.github/workflows/create-lint-wf.yml @@ -74,7 +74,7 @@ jobs: run: prettier --check nf-core-testpipeline - name: Run ECLint check - run: editorconfig-checker -exclude README.md $(find nf-core-testpipeline/.* -type f | grep -v .git) + run: editorconfig-checker -exclude README.md $(find nf-core-testpipeline/.* -type f grep -v '.git\|.py\|md\|json\|yml\|yaml\|html\|css') # Update modules to the latest version - name: nf-core modules update diff --git a/.github/workflows/lint-code.yml b/.github/workflows/lint-code.yml index d49ed8c912..5609453acf 100644 --- a/.github/workflows/lint-code.yml +++ b/.github/workflows/lint-code.yml @@ -12,8 +12,9 @@ jobs: - name: Install editorconfig-checker run: npm install -g editorconfig-checker + # Run editor config check only on files not covered by a linter - name: Run ECLint check - run: editorconfig-checker -exclude README.md $(git ls-files | grep -v test) + run: editorconfig-checker -exclude README.md $(git ls-files | grep -v 'test\|.py\|md\|json\|yml\|yaml\|html\|css') Prettier: runs-on: ubuntu-latest diff --git a/Dockerfile b/Dockerfile index 8343e06ca1..54408b372f 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,6 +1,6 @@ -FROM python:3.8.9-slim +FROM python:3.8.9-slim LABEL authors="phil.ewels@scilifelab.se,erik.danielsson@scilifelab.se" \ - description="Docker image containing requirements for the nfcore tools" + description="Docker image containing requirements for the nfcore tools" # Do not pick up python packages from $HOME ENV PYTHONNUSERSITE=1 @@ -14,23 +14,23 @@ RUN python -m pip install -r requirements.txt # Install Nextflow dependencies RUN apt-get update \ - && apt-get upgrade -y \ - && apt-get install -y git \ - && apt-get install -y curl + && apt-get upgrade -y \ + && apt-get install -y git \ + && apt-get install -y curl -# Create man dir required for Java installation +# Create man dir required for Java installation # and install Java RUN mkdir -p /usr/share/man/man1 \ - && apt-get install -y openjdk-11-jre \ - && apt-get clean -y && rm -rf /var/lib/apt/lists/* + && apt-get install -y openjdk-11-jre \ + && apt-get clean -y && rm -rf /var/lib/apt/lists/* # Setup ARG for NXF_VER ENV ARG NXF_VER="" ENV NXF_VER ${NXF_VER} # Install Nextflow RUN curl -s https://get.nextflow.io | bash \ - && mv nextflow /usr/local/bin \ - && chmod a+rx /usr/local/bin/nextflow + && mv nextflow /usr/local/bin \ + && chmod a+rx /usr/local/bin/nextflow # Add the nf-core source files to the image COPY . /usr/src/nf_core WORKDIR /usr/src/nf_core diff --git a/MANIFEST.in b/MANIFEST.in index bb3c5d6dac..56005ebc33 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -2,4 +2,4 @@ include LICENSE include README.md graft nf_core/module-template graft nf_core/pipeline-template -include requirements.txt \ No newline at end of file +include requirements.txt diff --git a/docs/api/Makefile b/docs/api/Makefile index f961e4ded1..69f3987065 100644 --- a/docs/api/Makefile +++ b/docs/api/Makefile @@ -9,11 +9,11 @@ BUILDDIR = _build # Put it first so that "make" without argument is like "make help". help: - @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) .PHONY: help Makefile # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile - @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) \ No newline at end of file + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/nf_core/gitpod/gitpod.Dockerfile b/nf_core/gitpod/gitpod.Dockerfile index aa8bfe682d..b097b29cd2 100644 --- a/nf_core/gitpod/gitpod.Dockerfile +++ b/nf_core/gitpod/gitpod.Dockerfile @@ -19,14 +19,14 @@ RUN conda update -n base -c defaults conda && \ conda config --add channels bioconda && \ conda config --add channels conda-forge && \ conda install \ - openjdk=11.0.13 \ - nextflow=21.10.6 \ - pytest-workflow=1.6.0 \ - mamba=0.22.1 \ - pip=22.0.4 \ - black=22.1.0 \ - yamllint=1.26.3 \ - -n base && \ + openjdk=11.0.13 \ + nextflow=21.10.6 \ + pytest-workflow=1.6.0 \ + mamba=0.22.1 \ + pip=22.0.4 \ + black=22.1.0 \ + yamllint=1.26.3 \ + -n base && \ nextflow self-update && \ conda clean --all -f -y @@ -35,4 +35,4 @@ COPY . /usr/src/nf_core WORKDIR /usr/src/nf_core # Install nf-core -RUN python -m pip install . \ No newline at end of file +RUN python -m pip install . diff --git a/pyproject.toml b/pyproject.toml index 1bd66769db..f05ed68402 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,17 +1,17 @@ [build-system] -build-backend = 'setuptools.build_meta' +build-backend = "setuptools.build_meta" requires = [ - 'setuptools>=40.6.0', - 'wheel' + "setuptools>=40.6.0", + "wheel" ] [tool.black] line-length = 120 -target_version = ['py36','py37','py38'] +target_version = ["py36","py37","py38"] [tool.pytest.ini_options] markers = [ "datafiles: load datafiles" ] testpaths = ["tests"] -norecursedirs = [ '.*', 'build', 'dist', '*.egg', 'data', '__pycache__', '.github', 'nf_core', 'docs'] \ No newline at end of file +norecursedirs = [ ".*", "build", "dist", "*.egg", "data", "__pycache__", ".github", "nf_core", "docs"] diff --git a/requirements-dev.txt b/requirements-dev.txt index 1ad518c05c..081e32aea7 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -4,4 +4,4 @@ pytest-cov mock black Sphinx -sphinx_rtd_theme \ No newline at end of file +sphinx_rtd_theme From df743738252efcc8e8e43b36ef24c4f846c47de3 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 15:58:44 +0100 Subject: [PATCH 097/110] Fix grep command --- .github/workflows/create-lint-wf.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/create-lint-wf.yml b/.github/workflows/create-lint-wf.yml index d916112e6a..4ac8edc1b1 100644 --- a/.github/workflows/create-lint-wf.yml +++ b/.github/workflows/create-lint-wf.yml @@ -74,7 +74,7 @@ jobs: run: prettier --check nf-core-testpipeline - name: Run ECLint check - run: editorconfig-checker -exclude README.md $(find nf-core-testpipeline/.* -type f grep -v '.git\|.py\|md\|json\|yml\|yaml\|html\|css') + run: editorconfig-checker -exclude README.md $(find nf-core-testpipeline/.* -type f | grep -v '.git\|.py\|md\|json\|yml\|yaml\|html\|css\|work\|.nextflow') # Update modules to the latest version - name: nf-core modules update From 653633302300ba80d6748718dc4ed639a3cc7605 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 16:03:33 +0100 Subject: [PATCH 098/110] Exclude more files from the eclint test on the resolved pipeline --- .github/workflows/create-lint-wf.yml | 2 +- nf_core/module-template/tests/main.nf | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/create-lint-wf.yml b/.github/workflows/create-lint-wf.yml index 4ac8edc1b1..f996adf381 100644 --- a/.github/workflows/create-lint-wf.yml +++ b/.github/workflows/create-lint-wf.yml @@ -74,7 +74,7 @@ jobs: run: prettier --check nf-core-testpipeline - name: Run ECLint check - run: editorconfig-checker -exclude README.md $(find nf-core-testpipeline/.* -type f | grep -v '.git\|.py\|md\|json\|yml\|yaml\|html\|css\|work\|.nextflow') + run: editorconfig-checker -exclude README.md $(find nf-core-testpipeline/.* -type f | grep -v '.git\|.py\|md\|json\|yml\|yaml\|html\|css\|work\|.nextflow\|build\|nf_core.egg-info\|log.txt') # Update modules to the latest version - name: nf-core modules update diff --git a/nf_core/module-template/tests/main.nf b/nf_core/module-template/tests/main.nf index b172736c3a..a7be132962 100644 --- a/nf_core/module-template/tests/main.nf +++ b/nf_core/module-template/tests/main.nf @@ -6,9 +6,9 @@ include { {{ tool_name_underscore|upper }} } from '../../../{{ "../" if subtool workflow test_{{ tool_name_underscore }} { {% if has_meta %} - input = [ + input = [ [ id:'test', single_end:false ], // meta map - file(params.test_data['sarscov2']['illumina']['test_paired_end_bam'], checkIfExists: true) + file(params.test_data['sarscov2']['illumina']['test_paired_end_bam'], checkIfExists: true) ] {%- else %} input = file(params.test_data['sarscov2']['illumina']['test_single_end_bam'], checkIfExists: true) From ab425c6df41264c77e537015d1e62022fd90407f Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 16:34:10 +0100 Subject: [PATCH 099/110] Reset Sphinx Makefile and don't run eclint on it --- .github/workflows/lint-code.yml | 2 +- docs/api/Makefile | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/lint-code.yml b/.github/workflows/lint-code.yml index 5609453acf..2f6f046cbf 100644 --- a/.github/workflows/lint-code.yml +++ b/.github/workflows/lint-code.yml @@ -14,7 +14,7 @@ jobs: # Run editor config check only on files not covered by a linter - name: Run ECLint check - run: editorconfig-checker -exclude README.md $(git ls-files | grep -v 'test\|.py\|md\|json\|yml\|yaml\|html\|css') + run: editorconfig-checker -exclude README.md $(git ls-files | grep -v 'test\|.py\|md\|json\|yml\|yaml\|html\|css\|Makefile') Prettier: runs-on: ubuntu-latest diff --git a/docs/api/Makefile b/docs/api/Makefile index 69f3987065..f961e4ded1 100644 --- a/docs/api/Makefile +++ b/docs/api/Makefile @@ -9,11 +9,11 @@ BUILDDIR = _build # Put it first so that "make" without argument is like "make help". help: - @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) .PHONY: help Makefile # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile - @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) \ No newline at end of file From 64bb9e433d0df0522a9f49a31433bccb28982b82 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 16:40:47 +0100 Subject: [PATCH 100/110] Ignore makefile in the other CI test as well --- .github/workflows/create-lint-wf.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/create-lint-wf.yml b/.github/workflows/create-lint-wf.yml index f996adf381..e87eca38a8 100644 --- a/.github/workflows/create-lint-wf.yml +++ b/.github/workflows/create-lint-wf.yml @@ -74,7 +74,7 @@ jobs: run: prettier --check nf-core-testpipeline - name: Run ECLint check - run: editorconfig-checker -exclude README.md $(find nf-core-testpipeline/.* -type f | grep -v '.git\|.py\|md\|json\|yml\|yaml\|html\|css\|work\|.nextflow\|build\|nf_core.egg-info\|log.txt') + run: editorconfig-checker -exclude README.md $(find nf-core-testpipeline/.* -type f | grep -v '.git\|.py\|md\|json\|yml\|yaml\|html\|css\|work\|.nextflow\|build\|nf_core.egg-info\|log.txt\|Makefile') # Update modules to the latest version - name: nf-core modules update From 9e078a2e0b8882117353428e8f6a7a44233bab50 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 18 Mar 2022 16:53:38 +0100 Subject: [PATCH 101/110] Update editorconfig-checker command in the pipeline CI workflow as well --- nf_core/pipeline-template/.github/workflows/linting.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/.github/workflows/linting.yml b/nf_core/pipeline-template/.github/workflows/linting.yml index 48d643f657..ca13c70e21 100644 --- a/nf_core/pipeline-template/.github/workflows/linting.yml +++ b/nf_core/pipeline-template/.github/workflows/linting.yml @@ -20,7 +20,7 @@ jobs: run: npm install -g editorconfig-checker - name: Run ECLint check - run: editorconfig-checker -exclude README.md $(git ls-files | grep -v test) + run: editorconfig-checker -exclude README.md $(find .* -type f | grep -v '.git\|.py\|.md\|json\|yml\|yaml\|html\|css\|work\|.nextflow\|build\|nf_core.egg-info\|log.txt\|Makefile') Prettier: runs-on: ubuntu-latest From 65c26e6d7b3484402895c64e76d9b0ab6c945ef0 Mon Sep 17 00:00:00 2001 From: Edmund Miller Date: Fri, 18 Mar 2022 11:45:01 -0500 Subject: [PATCH 102/110] test: Ignore pipeline name line When the template is generated the description and name take up more than 120 characters, so then prettier re-formats the email_template and breaks it up into 3 separate lines because it's longer the 120. We can't just break it up into 3 lines though because if there's a short name and description it'll reformat it into one line. Co-authored-by: Rike --- nf_core/pipeline-template/assets/email_template.html | 1 + 1 file changed, 1 insertion(+) diff --git a/nf_core/pipeline-template/assets/email_template.html b/nf_core/pipeline-template/assets/email_template.html index 069da09361..24a3ad9d0c 100644 --- a/nf_core/pipeline-template/assets/email_template.html +++ b/nf_core/pipeline-template/assets/email_template.html @@ -4,6 +4,7 @@ + {{ name }} Pipeline Report From 382d383fb88cda96ce3c08ff0905497ae72e0e84 Mon Sep 17 00:00:00 2001 From: Mei Wu <25568561+projectoriented@users.noreply.github.com> Date: Mon, 21 Mar 2022 15:57:34 +0100 Subject: [PATCH 103/110] change to params.outdir --- nf_core/pipeline-template/docs/usage.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/nf_core/pipeline-template/docs/usage.md b/nf_core/pipeline-template/docs/usage.md index 8d84911d67..cfc859e261 100644 --- a/nf_core/pipeline-template/docs/usage.md +++ b/nf_core/pipeline-template/docs/usage.md @@ -65,9 +65,9 @@ This will launch the pipeline with the `docker` configuration profile. See below Note that the pipeline will create the following files in your working directory: ```console -work # Directory containing the nextflow working files -results # Finished results (configurable, see below) -.nextflow_log # Log file from Nextflow +work # Directory containing the nextflow working files +params.outdir # Finished results (configurable, see below) +.nextflow_log # Log file from Nextflow # Other nextflow hidden files, eg. history of pipeline runs and old logs. ``` From 8087928891161090e5cac66d3f08df3962a49a63 Mon Sep 17 00:00:00 2001 From: Mei Wu <25568561+projectoriented@users.noreply.github.com> Date: Mon, 21 Mar 2022 21:08:38 +0100 Subject: [PATCH 104/110] applying suggestions --- nf_core/pipeline-template/docs/usage.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/nf_core/pipeline-template/docs/usage.md b/nf_core/pipeline-template/docs/usage.md index cfc859e261..767a4fa530 100644 --- a/nf_core/pipeline-template/docs/usage.md +++ b/nf_core/pipeline-template/docs/usage.md @@ -65,9 +65,9 @@ This will launch the pipeline with the `docker` configuration profile. See below Note that the pipeline will create the following files in your working directory: ```console -work # Directory containing the nextflow working files -params.outdir # Finished results (configurable, see below) -.nextflow_log # Log file from Nextflow +work # Directory containing the nextflow working files + # Finished results in specified location, --outdir (configurable, see below) +.nextflow_log # Log file from Nextflow # Other nextflow hidden files, eg. history of pipeline runs and old logs. ``` From 7e50282cb0eb7a3f3dc27ff3f3e829205cf70243 Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Tue, 22 Mar 2022 08:38:26 +0100 Subject: [PATCH 105/110] Update nf_core/pipeline-template/docs/usage.md --- nf_core/pipeline-template/docs/usage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/docs/usage.md b/nf_core/pipeline-template/docs/usage.md index 767a4fa530..1a4adc1e9e 100644 --- a/nf_core/pipeline-template/docs/usage.md +++ b/nf_core/pipeline-template/docs/usage.md @@ -66,7 +66,7 @@ Note that the pipeline will create the following files in your working directory ```console work # Directory containing the nextflow working files - # Finished results in specified location, --outdir (configurable, see below) + # Finished results in specified location (defined with --outdir) .nextflow_log # Log file from Nextflow # Other nextflow hidden files, eg. history of pipeline runs and old logs. ``` From 954c5a74b6bc73a0447e65ed16c52b5e3754cc67 Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Tue, 22 Mar 2022 08:39:58 +0100 Subject: [PATCH 106/110] Lint --- nf_core/pipeline-template/docs/usage.md | 1 + 1 file changed, 1 insertion(+) diff --git a/nf_core/pipeline-template/docs/usage.md b/nf_core/pipeline-template/docs/usage.md index 1a4adc1e9e..c89bd90f0f 100644 --- a/nf_core/pipeline-template/docs/usage.md +++ b/nf_core/pipeline-template/docs/usage.md @@ -67,6 +67,7 @@ Note that the pipeline will create the following files in your working directory ```console work # Directory containing the nextflow working files # Finished results in specified location (defined with --outdir) + .nextflow_log # Log file from Nextflow # Other nextflow hidden files, eg. history of pipeline runs and old logs. ``` From 0f4ad5c198acf9fa20867ed02b13aaaf602e830b Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Tue, 22 Mar 2022 08:54:38 +0100 Subject: [PATCH 107/110] =?UTF-8?q?Linting=20again,=20added=20extra=20new?= =?UTF-8?q?=20line=20=F0=9F=A4=A6=E2=80=8D=E2=99=80?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- nf_core/pipeline-template/docs/usage.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/nf_core/pipeline-template/docs/usage.md b/nf_core/pipeline-template/docs/usage.md index c89bd90f0f..b074617e2e 100644 --- a/nf_core/pipeline-template/docs/usage.md +++ b/nf_core/pipeline-template/docs/usage.md @@ -66,8 +66,7 @@ Note that the pipeline will create the following files in your working directory ```console work # Directory containing the nextflow working files - # Finished results in specified location (defined with --outdir) - + # Finished results in specified location (defined with --outdir) .nextflow_log # Log file from Nextflow # Other nextflow hidden files, eg. history of pipeline runs and old logs. ``` From 8c33481238b1a30270bc8b10f2f81eee49156e6f Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 23 Mar 2022 14:03:34 +0100 Subject: [PATCH 108/110] Changelog, bump to 2.3.1 --- CHANGELOG.md | 12 ++++++++++-- setup.py | 2 +- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c1ca2207c2..bd9bc89204 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,17 +1,25 @@ # nf-core/tools: Changelog -## v2.4dev +## [v2.3.1 - Mercury Vulture Formatting](https://github.com/nf-core/tools/releases/tag/2.3.1) - [2022-03-23] ### Template -- Replace `markdownlint` and `yamllint` with [_Prettier_](https://prettier.io) for linting formatting / whitespace. +- Replace `markdownlint` and `yamllint` with [_Prettier_](https://prettier.io) for linting formatting / whitespace ([#1470](https://github.com/nf-core/tools/pull/1470)) +- Add CI test using `editorconfig-checker` for other file types to look for standardised indentation and formatting ([#1476](https://github.com/nf-core/tools/pull/1476)) - Add md5sum check of `versions.yml` to `test.yml` on the modules template. +- Update bundled module wrappers to latest versions ([#1462](https://github.com/nf-core/tools/pull/1462)) +- Renamed `assets/multiqc_config.yaml` to `assets/multiqc_config.yml` (`yml` not `yaml`) ([#1471](https://github.com/nf-core/tools/pull/1471)) ### General - Convert nf-core/tools API / lint test documentation to MyST ([#1245](https://github.com/nf-core/tools/pull/1245)) - Build documentation for the `nf-core modules lint` tests ([#1250](https://github.com/nf-core/tools/pull/1250)) +- Fix some colours in the nf-core/tools API docs ([#1467](https://github.com/nf-core/tools/pull/1467)) - Install tools inside GitPod Docker using the repo itself and not from Conda. +- Rewrite GitHub Actions workflow for publishing the GitPod Docker image. +- Improve config for PyTest so that you can run `pytest` instead of `pytest tests/` ([#1461](https://github.com/nf-core/tools/pull/1461)) +- New pipeline lint test `multiqc_config` that checks YAML structure instead of basic file contents ([#1461](https://github.com/nf-core/tools/pull/1461)) +- Updates to the GitPod docker image to install the latest version of nf-core/tools ### Modules diff --git a/setup.py b/setup.py index 5fda0ae5be..9e407d082f 100644 --- a/setup.py +++ b/setup.py @@ -2,7 +2,7 @@ from setuptools import setup, find_packages -version = "2.4dev" +version = "2.3.1" with open("README.md") as f: readme = f.read() From 41825073d988a29bc320f64ab954f04d731937a7 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 23 Mar 2022 14:18:01 +0100 Subject: [PATCH 109/110] Add Prettier instructions to the changelog --- CHANGELOG.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index bd9bc89204..ea73fae36f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,29 @@ ## [v2.3.1 - Mercury Vulture Formatting](https://github.com/nf-core/tools/releases/tag/2.3.1) - [2022-03-23] +This patch release is primarily to address problems that we had in the v2.3 release with code linting. +Instead of resolving those specific issues, we chose to replace the linting tools (`markdownlint`, `yamllint`) with a new tool: [_Prettier_](https://prettier.io) + +This is a fairly major change and affects a lot of files. However, it will hopefully simplify future usage. +Prettier can auto-format many different file formats (for pipelines the most relevant are markdown and YAML) and is extensible with plugins (Nextflow, anyone?). +It tends to be a bit less strict than `markdownlint` and `yamllint` and importantly _can fix files for you_ rather than just complaining. + +The sync PR may be a little big because of many major changes (whitespace, quotation mark styles etc). +To help with the merge, _**we highly recommend running Prettier on your pipeline's codebase before attempting the template merge**_. +If you take this approach, please copy `.editorconfig` and `.prettierrc.yml` from the template to your pipeline root first, +as they configure the behaviour of Prettier. + +To run Prettier, go to the base of the repository where `.editorconfig` and `.prettierrc.yml` are located. +Make sure your `git status` is clean so that the changes don't affect anything you're working on and run: + +```bash +prettier --write . +``` + +This runs Prettier and tells it to fix any issues it finds in place. + +Please note that there are many excellent integrations for Prettier available, for example VSCode can be set up to automatically format files on save. + ### Template - Replace `markdownlint` and `yamllint` with [_Prettier_](https://prettier.io) for linting formatting / whitespace ([#1470](https://github.com/nf-core/tools/pull/1470)) From 7d8ff656d0a703b900f4b1aacec8220b7dfd4326 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 23 Mar 2022 14:25:44 +0100 Subject: [PATCH 110/110] Update CHANGELOG.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Matthias Hörtenhuber --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ea73fae36f..518fa1bdd6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ This patch release is primarily to address problems that we had in the v2.3 rele Instead of resolving those specific issues, we chose to replace the linting tools (`markdownlint`, `yamllint`) with a new tool: [_Prettier_](https://prettier.io) This is a fairly major change and affects a lot of files. However, it will hopefully simplify future usage. -Prettier can auto-format many different file formats (for pipelines the most relevant are markdown and YAML) and is extensible with plugins (Nextflow, anyone?). +Prettier can auto-format many different file formats (for pipelines the most relevant are markdown and YAML) and is extensible with plugins ([Nextflow](https://github.com/nf-core/prettier-plugin-nextflow), anyone?). It tends to be a bit less strict than `markdownlint` and `yamllint` and importantly _can fix files for you_ rather than just complaining. The sync PR may be a little big because of many major changes (whitespace, quotation mark styles etc).