Adding our nf-test tests to the GitHub Actions CI #66

RobJY · 2024-10-29T15:00:12Z

PR checklist

This is still a work in progress, but I think it's getting close.
To do Items:

CI is failing for Singularity tests. Looks like a problem running mesmer. I've updated it, but that hasn't helped.
- Docker tests pass
failing lint, but much of that is because of the updated template
I'm forcing more CI tests than it would normally run, so that code needs to be removed before deployment
Check that CI runs nf-tests for pull-request
- They work for push and workflow_dispatch

github-actions · 2024-10-29T15:09:10Z

`nf-core pipelines lint` overall result: Failed ❌

Posted for pipeline commit 271f172

+| ✅ 167 tests passed       |+
#| ❔   1 tests were ignored |#
!| ❗  26 tests had warnings |!
-| ❌  13 tests failed       |-

❌ Test failures:

nextflow_config - Config variable (incorrectly) found: params.max_cpus
nextflow_config - Config variable (incorrectly) found: params.max_memory
nextflow_config - Config variable (incorrectly) found: params.max_time
nextflow_config - Old lines for loading custom profiles found. File should contain: ```groovy
// Load nf-core custom profiles from different Institutions
includeConfig !System.getenv('NXF_OFFLINE') && params.custom_config_base ? "${params.custom_config_base}/nfcore_custom.config" : "/dev/null"
files_unchanged - .github/CONTRIBUTING.md does not match the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md does not match the template
files_unchanged - .github/workflows/linting_comment.yml does not match the template
files_unchanged - .github/workflows/linting.yml does not match the template
files_unchanged - .gitignore does not match the template
actions_ci - Minimum pipeline NF version '23.04.0' is not tested in '.github/workflows/ci.yml'
actions_awsfulltest - .github/workflows/awsfulltest.yml is not triggered correctly
schema_params - Param modules_testdata_base_path from nextflow config not found in nextflow_schema.json
multiqc_config - assets/multiqc_config.yml does not contain a matching 'report_comment'.
The expected comment is:
This report has been generated by the <a href="https://github.com/nf-core/mcmicro/tree/dev" target="_blank">nf-core/mcmicro</a> analysis pipeline. For information about how to interpret these results, please see the <a href="https://nf-co.re/mcmicro/dev/docs/output" target="_blank">documentation</a>.
The current comment is:
This report has been generated by the <a href="https://github.com/nf-core/mcmicro/releases/tag/2.0.0" target="_blank">nf-core/mcmicro</a> analysis pipeline. For information about how to interpret these results, please see the <a href="https://nf-co.re/mcmicro/2.0.0/docs/output" target="_blank">documentation</a>.

❗ Test warnings:

files_exist - File not found: conf/igenomes_ignored.config
nextflow_config - nf-validation has been detected in the pipeline. Please migrate to nf-schema: https://nextflow-io.github.io/nf-schema/latest/migration_guide/
readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
pipeline_todos - TODO string in README.md: TODO nf-core:
pipeline_todos - TODO string in README.md: Include a figure that guides the user through the major workflow steps. Many nf-core
pipeline_todos - TODO string in README.md: Fill in short bullet-pointed list of the default steps in the pipeline
pipeline_todos - TODO string in README.md: Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.
pipeline_todos - TODO string in README.md: update the following command to include all required parameters for a minimal example
pipeline_todos - TODO string in README.md: If applicable, make list of people who have also contributed
pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
pipeline_todos - TODO string in README.md: Add bibliography of tools and data used in your pipeline
pipeline_todos - TODO string in output.md: Write this documentation describing your workflow's output
pipeline_todos - TODO string in usage.md: Add documentation about anything specific to running your pipeline. For general topics, please point to (and add to) the main nf-core website.
pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
pipeline_todos - TODO string in test.config: Specify the paths to your test data on nf-core/test-datasets
pipeline_todos - TODO string in test.config: Give any required params for the test so that command line flags are not needed
pipeline_todos - TODO string in base.config: Check the defaults for all processes
pipeline_todos - TODO string in base.config: Customise requirements for specific processes.
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
schema_lint - Parameter input not found in schema
nfcore_yml - nf-core version in .nf-core.yml is not set to the latest version. Should be 3.0.2 but was 2.14.1

❔ Tests ignored:

nextflow_config - Config variable ignored: params.input

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-mcmicro_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-mcmicro_logo_light.png
files_exist - File found: docs/images/nf-core-mcmicro_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-mcmicro_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowMcmicro.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-validation plugin
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 2.0.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.max_cpus= 16
nextflow_config - Config default value correct: params.max_memory= 128.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-mcmicro_logo_light.png matches the template
files_unchanged - docs/images/nf-core-mcmicro_logo_light.png matches the template
files_unchanged - docs/images/nf-core-mcmicro_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .prettierignore matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
plugin_includes - No wrong validation plugin imports have been found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (0 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
modules_config - conf/modules.config found and not ignored.
modules_config - MULTIQC found in conf/modules.config and Nextflow scripts.
modules_config - BASICPY found in conf/modules.config and Nextflow scripts.
modules_config - ASHLAR found in conf/modules.config and Nextflow scripts.
modules_config - DEEPCELL_MESMER found in conf/modules.config and Nextflow scripts.
modules_config - CELLPOSE found in conf/modules.config and Nextflow scripts.
modules_config - MCQUANT found in conf/modules.config and Nextflow scripts.
modules_config - COREOGRAPH found in conf/modules.config and Nextflow scripts.
modules_config - SCIMAP_MCMICRO found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline

Run details

nf-core/tools version 3.0.2
Run at 2024-11-04 15:31:06

RobJY · 2024-10-29T15:57:55Z

Ah, the paths I use to force more CI tests probably don't exist when running in the PR, so I've commented that out.

RobJY · 2024-10-29T20:12:54Z

Looks like one of the things we're getting hung up on is a missing file for a module test for mesmer:

Test Process DEEPCELL_MESMER

  Test [4cdeb94e] 'mesmer - tif' 
    > Nextflow 24.10.0 is available - Please consider updating your version to it
    > N E X T F L O W  ~  version 24.04.2
    > Launching `/home/runner/work/mcmicro/mcmicro/.nf-test-4cdeb94e8ab416d5ddd3cf7f197d7523.nf` [shrivelled_curie] DSL2 - revision: fe2ec8638a
    > Downloading plugin nf-validation@1.1.3
    > ERROR ~ No such file or directory: https://raw.githubusercontent.com/nf-core/test-datasets/imaging/segmentation/cycif_tonsil_registered.ome.tif
    > 
    >  -- Check script '/home/runner/work/mcmicro/mcmicro/.nf-test-4cdeb94e8ab416d5ddd3cf7f197d7523.nf' at line: 34 or see '/home/runner/work/mcmicro/mcmicro/.nf-test/tests/4cdeb94e8ab416d5ddd3cf7f197d7523/meta/nextflow.log' file for more details
Assertion failed:

Looking into how to fix this.

RobJY · 2024-10-30T19:18:09Z

Easy fix. It just needed a small path change.

mashehu · 2024-11-02T12:24:26Z

@nf-core-bot fix linting

kbestak · 2024-11-04T11:30:37Z

Hi @RobJY,
some of the changes made here are also done in the template update. It would be better practice under nf-core to first merge the template update. This is important as CI is affected by the nf-core/tools version. Could you, or Jeremy please first review #65 the PR here, and then check for conflicts here?

.github/workflows/ci.yml

RobJY · 2024-11-05T20:02:47Z

Looks like everything is passing except this lint error:

files_unchanged: docs/images/nf-core-mcmicro_logo_dark.png does not match the template

Not sure why a file we added is setting off the template "files_unchanged" error.
It's passing on my local system.
Looking into it. Please let me know if you have any suggestions. Thanks!
I'll try the bot fix above again.

RobJY · 2024-11-05T20:02:59Z

@nf-core-bot fix linting

RobJY · 2024-11-05T20:30:01Z

Hmm, no, bot didn't fix it.

RobJY · 2024-11-05T21:28:40Z

Ok, that worked, but it seems like more of a workaround than a permanent fix. I just added docs/images/nf-core-mcmicro_logo_dark.png that lint was complaining about to the files_unchanged section of .nf-core.yml.
It doesn't seem like lint should be flagging this file as files_unchanged.
Still looking for a better solution.

kbestak

Overall looks good to me! Great job sorting out the linting issues. I just left a couple of questions that are more of a technicality for clarification.

kbestak · 2024-11-05T21:23:44Z

modules/nf-core/basicpy/basicpy.diff

Is there a specific reason the module was changed? I have an open PR adding nf-test to basicpy nf-core/modules#6929 if there are some changes to be made to make the module more standardized feel free to add to that PR (it shouldn't be an issue since it's only in MCMICRO currently).

conf/ci.config

.github/workflows/ci.yml

nextflow.config

kbestak · 2024-11-05T21:34:06Z

Ok, that worked, but it seems like more of a workaround than a permanent fix. I just added docs/images/nf-core-mcmicro_logo_dark.png that lint was complaining about to the files_unchanged section of .nf-core.yml. It doesn't seem like lint should be flagging this file as files_unchanged. Still looking for a better solution.

I don't think this is an issue. Especially if the logo is updated down the line, this would be added to files_unchanged anyway.

Co-authored-by: Krešimir Beštak <86408271+kbestak@users.noreply.github.com>

kbestak

LGTM 🚀

jmuhlich · 2024-11-25T20:00:02Z

The logo is a known problem. There's a really subtle font rendering difference in the CI environment compared to our local environments. We had it set up previously where the file in git was the version the CI check would approve, and only local lint tests would fail that check (and we know to ignore that).

rebased on dev

jmuhlich

Looks good overall. My comments are mostly about removing commented-out stuff, but also I think there's a better fix for the lint error with the dark logo png.

.github/workflows/ci.yml

.nf-core.yml

docs/images/nf-core-mcmicro_logo_dark.png

RobJY added 5 commits October 28, 2024 14:33

added nf-tests to ci and updated basipy and ashlar

092b889

forcing more nf-tests in ci

a0a98cd

update mesmer and added test param to not check conda channels

3b8797b

removed ineffective test param

65c6b2a

schema fix

1d2885b

removing forced tests for testing

a65f5aa

RobJY added 5 commits October 29, 2024 12:01

ran prettier

c31dced

added required final newline to ci.yml

d6ba5bd

testing detect-nf-test-changes fix

7cc158f

testing another possible fix for detect changes

802a76d

ci fix: added modules_testdata_base_path

40dc458

fixed mesmer test file url

89ed2b3

RobJY added 2 commits October 30, 2024 15:51

ran prettier again

d079b34

removed documentation branch from ci

6e1b4d1

RobJY requested a review from jmuhlich October 31, 2024 18:04

kbestak reviewed Nov 4, 2024

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

kbestak mentioned this pull request Nov 4, 2024

Documentation update and added a full test #67

Merged

11 tasks

RobJY added 5 commits November 4, 2024 10:29

added dev branch back to ci

271f172

pulled in template changes from dev

7157cbe

ran lint with fix option

c2179d3

updated schema with missing modules_testsdata_base_path

c37bd8f

fixed editorconfig-checker error

afe7aa7

possible fix for last lint error

502e25b

kbestak reviewed Nov 5, 2024

View reviewed changes

Update .github/workflows/ci.yml

8c8fcdb

Co-authored-by: Krešimir Beštak <86408271+kbestak@users.noreply.github.com>

kbestak approved these changes Nov 8, 2024

View reviewed changes

RobJY added 6 commits November 25, 2024 17:05

added nf-tests to ci and updated basipy and ashlar

acba646

rebased on dev

schema fix

4d525a4

rebased on dev

bug fix: duplicated variable during rebase conflict resolution

6a05d51

bug fix: another duplication

3eb04fe

bug fix: corrected schema version

14d06da

removed extra branches from ci push

2de5739

jmuhlich requested changes Dec 2, 2024

View reviewed changes

RobJY added 2 commits December 2, 2024 14:27

changes requested in PR

288336c

one more suggested change from PR

315ec7e

jmuhlich mentioned this pull request Dec 2, 2024

added background subtraction #48

Merged

11 tasks

jmuhlich approved these changes Dec 2, 2024

View reviewed changes

jmuhlich merged commit a02de4b into nf-core:dev Dec 2, 2024
12 checks passed

jmuhlich mentioned this pull request Jan 15, 2025

Template update for nf-core/tools version 3.1.1 #73

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding our nf-test tests to the GitHub Actions CI #66

Adding our nf-test tests to the GitHub Actions CI #66

RobJY commented Oct 29, 2024 •

edited

Loading

github-actions bot commented Oct 29, 2024 •

edited

Loading

❌ Test failures:

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

RobJY commented Oct 29, 2024

RobJY commented Oct 29, 2024 •

edited

Loading

RobJY commented Oct 30, 2024

mashehu commented Nov 2, 2024

kbestak commented Nov 4, 2024 •

edited

Loading

RobJY commented Nov 5, 2024

RobJY commented Nov 5, 2024

RobJY commented Nov 5, 2024

RobJY commented Nov 5, 2024

kbestak left a comment

kbestak Nov 5, 2024

kbestak commented Nov 5, 2024

kbestak left a comment

jmuhlich commented Nov 25, 2024

jmuhlich left a comment

Adding our nf-test tests to the GitHub Actions CI #66

Adding our nf-test tests to the GitHub Actions CI #66

Conversation

RobJY commented Oct 29, 2024 • edited Loading

PR checklist

github-actions bot commented Oct 29, 2024 • edited Loading

nf-core pipelines lint overall result: Failed ❌

❌ Test failures:

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

RobJY commented Oct 29, 2024

RobJY commented Oct 29, 2024 • edited Loading

RobJY commented Oct 30, 2024

mashehu commented Nov 2, 2024

kbestak commented Nov 4, 2024 • edited Loading

RobJY commented Nov 5, 2024

RobJY commented Nov 5, 2024

RobJY commented Nov 5, 2024

RobJY commented Nov 5, 2024

kbestak left a comment

Choose a reason for hiding this comment

kbestak Nov 5, 2024

Choose a reason for hiding this comment

kbestak commented Nov 5, 2024

kbestak left a comment

Choose a reason for hiding this comment

jmuhlich commented Nov 25, 2024

jmuhlich left a comment

Choose a reason for hiding this comment

RobJY commented Oct 29, 2024 •

edited

Loading

github-actions bot commented Oct 29, 2024 •

edited

Loading

`nf-core pipelines lint` overall result: Failed ❌

RobJY commented Oct 29, 2024 •

edited

Loading

kbestak commented Nov 4, 2024 •

edited

Loading