Finish porting the pipeline to updated Nextflow DSL2 syntax #58

drpatelh · 2021-11-27T22:09:28Z

No description provided.

github-actions · 2021-11-28T14:20:11Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 0c5a67b

+| ✅ 116 tests passed       |+
#| ❔  16 tests were ignored |#
!| ❗   5 tests had warnings |!

❗ Test warnings:

files_exist - File not found: assets/multiqc_config.yaml
files_exist - File not found: conf/igenomes.config
files_exist - File not found: lib/WorkflowFetchngs.groovy
nextflow_config - Config manifest.version should end in dev: '1.5'
readme - README did not have a Nextflow minimum version mentioned in Quick Start section.

❔ Tests ignored:

files_exist - File is ignored: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File is ignored: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File is ignored: bin/scrape_software_versions.py
files_exist - File is ignored: modules/local/get_software_versions.nf
files_unchanged - File ignored due to lint config: .gitattributes
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File does not exist: .github/ISSUE_TEMPLATE/bug_report.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/config.yml
files_unchanged - File does not exist: .github/ISSUE_TEMPLATE/feature_request.md
files_unchanged - File ignored due to lint config: .github/workflows/linting_comment.yml
files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
files_unchanged - File does not exist: bin/scrape_software_versions.py
files_unchanged - File ignored due to lint config: lib/NfcoreSchema.groovy
files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
files_unchanged - File does not exist: assets/multiqc_config.yaml
actions_ci - actions_ci

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .markdownlint.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-fetchngs_logo.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-fetchngs_logo.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreSchema.groovy
files_exist - File found: lib/NfcoreTemplate.groovy
files_exist - File found: lib/Utils.groovy
files_exist - File found: lib/WorkflowMain.groovy
files_exist - File found: main.nf
files_exist - File found: conf/base.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.show_hidden_params
nextflow_config - Config variable found: params.schema_ignore_params
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.version
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .svg
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
files_unchanged - .markdownlint.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/nf-core-fetchngs_logo.png matches the template
files_unchanged - docs/images/nf-core-fetchngs_logo.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
files_unchanged - .gitignore matches the template
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 21.10.3, Config: 21.10.3
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (70 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_params - Schema matched params returned from nextflow config
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json

Run details

nf-core/tools version 2.1
Run at 2021-11-29 10:55:55

JoseEspinosa

LGTM, just added a few suggestions. After solving them feel free to merge! :rocket

JoseEspinosa · 2021-11-29T08:57:22Z

.github/CONTRIBUTING.md

+3. Define the output channel if needed (see below).
+4. Add any new flags/options to `nextflow.config` with a default (see below).
+5. Add any new flags/options to `nextflow_schema.json` with help text (with `nf-core schema build`).
+6. Add any new flags/options to the help message (for integer/text parameters, print to help the corresponding `nextflow.config` parameter).


Maybe add here add that should be added to the nextflow_schema.json for consistency with the two previous points?

I didn't understand?

Ah, I see. Think we can remove it altogether to be honest because it is already mentioned in point 5.

Fixed in eab36c9

JoseEspinosa · 2021-11-29T08:59:25Z

.github/CONTRIBUTING.md

+6. Add any new flags/options to the help message (for integer/text parameters, print to help the corresponding `nextflow.config` parameter).
+7. Add sanity checks for all relevant parameters.
+8. Add any new software to the `scrape_software_versions.py` script in `bin/` and the version command to the `scrape_software_versions` process in `main.nf`.
+9. Do local tests that the new code works properly and as expected.


Like more the previous version:

Suggested change

9. Do local tests that the new code works properly and as expected.

9. Perform local tests to validate that the new code works properly and as expected.

I got this from the pipeline template so will need to be updated there too. Will fix here for now 👍🏽

Fixed in eab36c9

JoseEspinosa · 2021-11-29T09:10:20Z

.github/CONTRIBUTING.md

+### Software version reporting
+
+If you add a new tool to the pipeline, please ensure you add the information of the tool to the `get_software_version` process.
+
+Add to the script block of the process, something like the following:
+
+```bash
+<YOUR_TOOL> --version &> v_<YOUR_TOOL>.txt 2>&1 || true
+```
+
+or
+
+```bash
+<YOUR_TOOL> --help | head -n 1 &> v_<YOUR_TOOL>.txt 2>&1 || true
+```
+
+You then need to edit the script `bin/scrape_software_versions.py` to:
+
+1. Add a Python regex for your tool's `--version` output (as in stored in the `v_<YOUR_TOOL>.txt` file), to ensure the version is reported as a `v` and the version number e.g. `v2.1.1`
+2. Add a HTML entry to the `OrderedDict` for formatting in MultiQC.
+


This section is not anymore in the template should we add a software version reporting section in the template?

Suggested change

### Software version reporting

If you add a new tool to the pipeline, please ensure you add the information of the tool to the `get_software_version` process.

Add to the script block of the process, something like the following:

```bash

<YOUR_TOOL> --version &> v_<YOUR_TOOL>.txt 2>&1 || true

```

or

```bash

<YOUR_TOOL> --help | head -n 1 &> v_<YOUR_TOOL>.txt 2>&1 || true

```

You then need to edit the script `bin/scrape_software_versions.py` to:

1. Add a Python regex for your tool's `--version` output (as in stored in the `v_<YOUR_TOOL>.txt` file), to ensure the version is reported as a `v` and the version number e.g. `v2.1.1`

2. Add a HTML entry to the `OrderedDict` for formatting in MultiQC.

Fixed in eab36c9

CHANGELOG.md

modules/local/sratools_fasterqdump.nf

modules/local/sratools_prefetch.nf

drpatelh added 14 commits November 27, 2021 09:46

Install updated syntax module from nf-core/modules

08d7c1e

Change version command to use task.process

1f307ce

Install nf-core modules

ab3bf4e

Add executor local for native groovy processes

b1ced19

Bump NF version from 21.04.0 to 21.10.3

4fcd56b

Move sratools modules to local due to customisation

2c28be2

Update CHANGELOG

25640bd

Remove params.publish_dir_mode

6e3259a

Remove params.publish_dir_mode

12fcb97

Fix tyop in versions syntax

194fb72

Rename dumpsoftwareversions module

4ee5489

Bump version to 1.5

27f7827

Upgrade CI tests to those on dev

c12cee0

Fix linting

e141707

drpatelh added 2 commits November 28, 2021 20:06

Lint and fix repo relative to latest dev

b21a617

Fix nf-core lint

881e1f8

drpatelh requested review from JoseEspinosa and Midnighter November 29, 2021 09:06

JoseEspinosa approved these changes Nov 29, 2021

View reviewed changes

drpatelh added 2 commits November 29, 2021 10:51

Address review comments

eab36c9

Fix nf-core lint

0c5a67b

drpatelh merged commit d9f11ac into nf-core:dev Nov 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Finish porting the pipeline to updated Nextflow DSL2 syntax #58

Finish porting the pipeline to updated Nextflow DSL2 syntax #58

drpatelh commented Nov 27, 2021

github-actions bot commented Nov 28, 2021 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

JoseEspinosa left a comment

JoseEspinosa Nov 29, 2021

drpatelh Nov 29, 2021

drpatelh Nov 29, 2021

drpatelh Nov 29, 2021

JoseEspinosa Nov 29, 2021

drpatelh Nov 29, 2021

drpatelh Nov 29, 2021

JoseEspinosa Nov 29, 2021

drpatelh Nov 29, 2021

	9. Do local tests that the new code works properly and as expected.
	9. Perform local tests to validate that the new code works properly and as expected.

Finish porting the pipeline to updated Nextflow DSL2 syntax #58

Finish porting the pipeline to updated Nextflow DSL2 syntax #58

Conversation

drpatelh commented Nov 27, 2021

github-actions bot commented Nov 28, 2021 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

JoseEspinosa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Nov 28, 2021 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️