Versions poc #150

edmundmiller · 2024-07-23T18:44:38Z

@FriederikeHanssen mentioned that @bentsherman mentioned that we could use mix on topic channels.

I think this is the best way to handle this and modules can be moved over piecemeal, or automated, but at least maintaining support for both options for a bit.

github-actions · 2024-07-23T18:46:42Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 066e0e5

+| ✅ 196 tests passed       |+
#| ❔   4 tests were ignored |#
!| ❗   2 tests had warnings |!

❗ Test warnings:

pipeline_todos - TODO string in README.md: Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline

❔ Tests ignored:

files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/email_template.html
actions_ci - actions_ci

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-nascent_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-nascent_logo_light.png
files_exist - File found: docs/images/nf-core-nascent_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-nascent_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowNascent.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 2.3.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.aligner= bwa
nextflow_config - Config default value correct: params.igenomes_base= s3://ngi-igenomes/igenomes/
nextflow_config - Config default value correct: params.human_pangenomics_base= https://s3-us-west-2.amazonaws.com/human-pangenomics
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.max_cpus= 16
nextflow_config - Config default value correct: params.max_memory= 128.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-nascent_logo_light.png matches the template
files_unchanged - docs/images/nf-core-nascent_logo_light.png matches the template
files_unchanged - docs/images/nf-core-nascent_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 24.04.3, Config: 24.04.3
readme - README Zenodo placeholder was replaced with DOI.
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (452 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains a matching 'report_comment'.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
modules_config - conf/modules.config found and not ignored.
modules_config - FASTQC found in conf/modules.config and Nextflow scripts.
modules_config - MULTIQC found in conf/modules.config and Nextflow scripts.
modules_config - FASTP found in conf/modules.config and Nextflow scripts.
modules_config - GFFREAD found in conf/modules.config and Nextflow scripts.
modules_config - BWA_MEM found in conf/modules.config and Nextflow scripts.
modules_config - DRAGMAP_HASHTABLE found in conf/modules.config and Nextflow scripts.
modules_config - STAR_ALIGN found in conf/modules.config and Nextflow scripts.
modules_config - SAMTOOLS_SORT found in conf/modules.config and Nextflow scripts.
modules_config - PRESEQ_CCURVE found in conf/modules.config and Nextflow scripts.
modules_config - PRESEQ_LCEXTRAP found in conf/modules.config and Nextflow scripts.
modules_config - BEDTOOLS_GENOMECOV_PLUS found in conf/modules.config and Nextflow scripts.
modules_config - BEDTOOLS_GENOMECOV_MINUS found in conf/modules.config and Nextflow scripts.
modules_config - DEEPTOOLS_BAMCOVERAGE_PLUS found in conf/modules.config and Nextflow scripts.
modules_config - DEEPTOOLS_BAMCOVERAGE_MINUS found in conf/modules.config and Nextflow scripts.
modules_config - DREG_PREP found in conf/modules.config and Nextflow scripts.
modules_config - SUBREAD_FEATURECOUNTS_GENE found in conf/modules.config and Nextflow scripts.
modules_config - SUBREAD_FEATURECOUNTS_PREDICTED found in conf/modules.config and Nextflow scripts.
modules_config - HOMER_FINDPEAKS found in conf/modules.config and Nextflow scripts.
modules_config - HOMER_MAKETAGDIRECTORY found in conf/modules.config and Nextflow scripts.
modules_config - HOMER_MAKEUCSCFILE found in conf/modules.config and Nextflow scripts.
modules_config - PINTS_CALLER found in conf/modules.config and Nextflow scripts.
modules_config - BEDTOOLS_SORT found in conf/modules.config and Nextflow scripts.
modules_config - BEDTOOLS_MERGE found in conf/modules.config and Nextflow scripts.
modules_config - BEDTOOLS_INTERSECT_FILTER found in conf/modules.config and Nextflow scripts.
modules_config - BEDTOOLS_INTERSECT found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 2.14.1

Run details

nf-core/tools version 2.14.1
Run at 2024-07-25 19:43:02

edmundmiller · 2024-07-23T19:39:18Z

Apparently Nascent's MultiQC version reporting is broken 😞

But here's the versions file:

BBMAP_PILEUP:
  bbmap: 39.01
  samtools: 1.16.1
  pigz: 2.6
BEDTOOLS_GENOMECOV_MINUS:
  bedtools: 2.31.1
BEDTOOLS_GENOMECOV_PLUS:
  bedtools: 2.31.1
BEDTOOLS_INTERSECT:
  bedtools: 2.31.1
BEDTOOLS_INTERSECT_FILTER:
  bedtools: 2.31.1
BWA_INDEX:
  bwa: 0.7.18-r1243-dirty
BWA_MEM:
  bwa: 0.7.18-r1243-dirty
  samtools: 1.2
DEEPTOOLS_BAMCOVERAGE_MINUS:
  deeptools: 3.5.1
DEEPTOOLS_BAMCOVERAGE_PLUS:
  deeptools: 3.5.1
FASTP:
  fastp: 0.23.4
FASTQC:
  fastqc: 0.12.1
HOMER_MAKETAGDIRECTORY:
  homer: 4.11
  samtools: 1.11
PINTS_CALLER:
    PINTS: 1.1.8
PINTS_CALLER:
    Python: 3.10.6
PRESEQ_CCURVE:
  preseq: 3.1.1
PRESEQ_LCEXTRAP:
  preseq: 3.1.1
RSEQC_INFEREXPERIMENT:
  rseqc: 5.0.2
RSEQC_READDISTRIBUTION:
  rseqc: 5.0.2
RSEQC_READDUPLICATION:
  rseqc: 5.0.2
SUBREAD_FEATURECOUNTS_GENE:
  subread: 2.0.1
Workflow:
    nf-core/nascent: v2.3.0dev
    Nextflow: 24.04.3

Co-authored-by: Ben Sherman <ben.sherman@seqera.io>

edmundmiller · 2024-09-27T13:50:01Z

Going to close this to snapshot the minimal changes and move to adopting it everywhere.

edmundmiller self-assigned this Jul 23, 2024

edmundmiller requested review from ewels, mashehu, bentsherman and FriederikeHanssen July 23, 2024 19:36

edmundmiller requested review from maxulysse and mirpedrol July 23, 2024 20:12

edmundmiller force-pushed the versions_poc branch 3 times, most recently from 5ed4fa2 to 4ef3a40 Compare July 24, 2024 15:51

edmundmiller and others added 11 commits July 24, 2024 16:57

test: Add a test for versions in the pipeline

e09bef0

refactor: Use an eval

6d1b2e2

duh it's a topic channel

02fe557

Try to view the topic

11314f1

fix: Add nextflow.preview.topic

37a29e9

Co-authored-by: Ben Sherman <ben.sherman@seqera.io>

fix: Add a $task.process

2fef8bb

fix: Add a val for the tool name

2514664

feat: Add topicVersionToYAML function

df55d42

feat: Try groupTuple

f4a5700

fix: Get the formatting just right on the versions

df910cd

chore: Clean up debug statements

c322679

edmundmiller force-pushed the versions_poc branch from bd798b6 to c322679 Compare July 24, 2024 22:04

build: Bump minimum version to 24.04.3 for "topic" channels

066e0e5

edmundmiller marked this pull request as draft August 1, 2024 18:17

edmundmiller mentioned this pull request Aug 1, 2024

Support version topic channels nf-core/tools#3093

Open

4 tasks

edmundmiller closed this Sep 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Versions poc #150

Versions poc #150

edmundmiller commented Jul 23, 2024 •

edited

Loading

github-actions bot commented Jul 23, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

edmundmiller commented Jul 23, 2024

edmundmiller commented Sep 27, 2024

Versions poc #150

Versions poc #150

Conversation

edmundmiller commented Jul 23, 2024 • edited Loading

github-actions bot commented Jul 23, 2024 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

edmundmiller commented Jul 23, 2024

edmundmiller commented Sep 27, 2024

edmundmiller commented Jul 23, 2024 •

edited

Loading

github-actions bot commented Jul 23, 2024 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️