nf-schema support #3116

nvnieuwk · 2024-08-14T14:02:53Z

PR features:

Minimal support for nf-schema (includes validation of draft 2020-12, definition notation checks...)
Update template schema
Check the pinning of the nf-schema version
Check for draft 2020-12 usage when nf-schema is used
A linting warning that advises the user to migrate to nf-schema
Check the plugin function imports are using nf-schema when nf-schema is specified in the config
we should add a "nf-validation" field in the templatefeatures.yml allowing one to opt-out of using nf-validation.
support nested parameters (Not doing this since this will no longer be supported in the future)

Features that are nice but not urgent (so probably not for this PR)

Check for the usage of the new config options instead of the parameters

nf_core/pipelines/lint/nextflow_config.py

…to feat/linting-nf-schema

nf_core/pipeline-template/nextflow.config

mirpedrol · 2024-08-20T10:02:22Z

nf_core/pipeline-template/subworkflows/nf-core/utils_nfcore_pipeline/main.nf

Just a reminder that this should be added to the nf-core repo, maybe we can create a new subworkflow in case someone wants to continue using nf-validation

Yep I made a PR to the modules repo with these changes :p these changes are not definitive yet

nf-core/modules#6217

mirpedrol · 2024-08-20T10:18:19Z

nf_core/pipeline-template/nextflow.config

+        beforeText = """
+-${colors.dim}----------------------------------------------------${colors.reset}-
+                                        ${colors.green},--.${colors.black}/${colors.green},-.${colors.reset}
+${colors.blue}        ___     __   __   __   ___     ${colors.green}/,-._.--~\'${colors.reset}
+${colors.blue}  |\\ | |__  __ /  ` /  \\ |__) |__         ${colors.yellow}}  {${colors.reset}
+${colors.blue}  | \\| |       \\__, \\__/ |  \\ |___     ${colors.green}\\`-._,-`-,${colors.reset}
+                                        ${colors.green}`._,._,\'${colors.reset}
+${colors.purple}  ${manifest.name} ${manifest.version}${colors.reset}
+-${colors.dim}----------------------------------------------------${colors.reset}-
+"""
+        afterText = """${manifest.doi ? "* The pipeline\n" : ""}${manifest.doi.tokenize(",").collect { "  https://doi.org/${it.trim().replace('https://doi.org/','')}"}.join("\n")}${manifest.doi ? "\n" : ""}
+* The nf-core framework 
+  https://doi.org/10.1038/s41587-020-0439-x
+
+* Software dependencies
+  https://github.com/${manifest.name}/blob/master/CITATIONS.md
+"""


should we lint for this in nf-core pipelines?

Good idea, but I want to implement this in a better way as it's soooo ugly right now, I'm just not sure how.

I'll also add a chrome color filter to nf-schema if validation.monochromeLogs is set to true. This should simplify the configs a lot already (We can hardcode the monochrome values in the message together with the pipeline name and version, same thing for the citations) I'd love to have these values as plain strings without having to use any external variables

I think the current way will be the best I can do. This will also stay supported in Nextflow in the future

mirpedrol

Nice! Only some last comments 😄

nf_core/pipeline-template/nextflow.config

nf_core/pipeline-template/nextflow_schema.json

nf_core/pipeline-template/subworkflows/local/utils_nfcore_pipeline_pipeline/main.nf

nf_core/pipeline-template/subworkflows/nf-core/utils_nextflow_pipeline/main.nf

nf_core/pipelines/create/templatefeatures.yml

nf_core/pipelines/lint/files_exist.py

tests/pipelines/lint/test_files_exist.py

tests/pipelines/lint/test_plugin_includes.py

Co-authored-by: Júlia Mir Pedrol <mirp.julia@gmail.com>

…line_pipeline/main.nf Co-authored-by: Júlia Mir Pedrol <mirp.julia@gmail.com>

Co-authored-by: Júlia Mir Pedrol <mirp.julia@gmail.com>

nf_core/pipeline-template/nextflow.config

nf_core/pipeline-template/subworkflows/local/utils_nfcore_pipeline_pipeline/main.nf

nf_core/pipelines/create/templatefeatures.yml

nf_core/pipelines/lint/nextflow_config.py

nf_core/pipelines/lint/plugin_includes.py

mashehu · 2024-08-27T12:06:39Z

nf_core/pipelines/lint/schema_description.py

@@ -36,8 +36,9 @@ def schema_description(self):
                warned.append(f"Ungrouped param in schema: `{up}`")

    # Iterate over groups and add warning for parameters without a description
-    for group_key in self.schema_obj.schema["definitions"].keys():
-        group = self.schema_obj.schema["definitions"][group_key]
+    defs_notation = self.schema_obj.defs_notation


don't see that this can be any other value than $defs in the code

This can still be definitions if the pipeline still uses nf-validation

I don't see that value written to in places different to the initialization and here

It's set everytime the schema of the PipelineSchema class is updated. This option is used heavily in schema.py and a bit in schema_description.py (or do you mean something else?)

So it depends on which schema has been used

I am talking about self.schema_obj.defs_notation. where is that value set to something different than $defs?

here:

tools/nf_core/pipelines/schema.py

Line 107 in 3600db7

self.defs_notation = "definitions"

basically if nf-validation is set in the config, then use definitions else use $defs

Co-authored-by: Matthias Hörtenhuber <mashehu@users.noreply.github.com>

…to feat/linting-nf-schema

nvnieuwk · 2024-08-27T12:59:31Z

Not sure what I can do about the snapshotting test 😁

mirpedrol · 2024-08-27T13:23:15Z

Not sure what I can do about the snapshotting test 😁

You can update them by running pytest tests/pipelines/test_create_app.py --snapshot-update, only if the update makes sense 😄 which in this case does, I'm not sure why updating the report is failing, but if it were working you could see a report showing the changes like this:

nvnieuwk · 2024-08-27T13:32:56Z

Thanks that did it! 🥳

nvnieuwk added 5 commits August 14, 2024 15:07

add schema checking for draft 2020-12

a754fe0

update validate schema

464f16e

update more code to support draft 2020-12

6658449

fix draft 7 schema using http

6ecde20

ruff

36fad24

nvnieuwk changed the title ~~Feat/linting nf-schema~~ nf-schema support Aug 14, 2024

nvnieuwk added 5 commits August 14, 2024 16:33

fix tests + add draft 2020-12 as default draft

6777694

fix typo

0dfd15d

ruff

7f29f37

add checks for nf-validation and nf-schema plugin config

730c677

linting fully works now!

7fbbff7

mashehu reviewed Aug 15, 2024

View reviewed changes

nf_core/pipelines/lint/nextflow_config.py Outdated Show resolved Hide resolved

nvnieuwk and others added 4 commits August 19, 2024 14:54

Merge branch 'dev' into feat/linting-nf-schema

59e2eb0

warning for nf-validation

0838d6b

Merge branch 'feat/linting-nf-schema' of github.com:nvnieuwk/tools in…

ba329fa

…to feat/linting-nf-schema

fetch plugin on schema filename setting

8559c2d

nvnieuwk mentioned this pull request Aug 19, 2024

Update docs with latest changes nextflow-io/nf-schema#42

Merged

nvnieuwk added 3 commits August 19, 2024 17:00

add a check for defs usage instead of $defs

0296f7d

change all definitions to self.defs_notation

0b8d04f

migrate template part 1

5443813

mirpedrol reviewed Aug 20, 2024

View reviewed changes

nvnieuwk added 9 commits August 20, 2024 15:24

some small updates for the new release

5519c83

some more small updates to the template

171640b

update handling of ignored parameters

249b3da

update validation plugin fetching

af34d07

update plugin linting

e6381b7

ignore genomes for now

e66a732

improve config linting for validation even more

b6ae8d0

check plugin includes in pipeline

1be1737

make nf-schema optional in the template

2f1a442

nvnieuwk and others added 4 commits August 27, 2024 10:41

fix lint test

9fd9869

update textual snapshots

6cccc1e

skip linting before and after Text params for custom pipelines

2ccbdd9

typo

ed01eba

mirpedrol reviewed Aug 27, 2024

View reviewed changes

nvnieuwk and others added 8 commits August 27, 2024 12:38

Update nf_core/pipeline-template/nextflow.config

1b597f1

Co-authored-by: Júlia Mir Pedrol <mirp.julia@gmail.com>

Update nf_core/pipeline-template/subworkflows/local/utils_nfcore_pipe…

93cc41b

…line_pipeline/main.nf Co-authored-by: Júlia Mir Pedrol <mirp.julia@gmail.com>

Update nf_core/pipeline-template/subworkflows/local/utils_nfcore_pipe…

6e358f3

…line_pipeline/main.nf Co-authored-by: Júlia Mir Pedrol <mirp.julia@gmail.com>

Update nf_core/pipeline-template/subworkflows/local/utils_nfcore_pipe…

dbe9318

…line_pipeline/main.nf Co-authored-by: Júlia Mir Pedrol <mirp.julia@gmail.com>

Update nf_core/pipelines/create/templatefeatures.yml

4025b54

Co-authored-by: Júlia Mir Pedrol <mirp.julia@gmail.com>

Update tests/pipelines/lint/test_plugin_includes.py

679de82

Co-authored-by: Júlia Mir Pedrol <mirp.julia@gmail.com>

review suggestions

3ac1bd8

use latest version of utils_nextflow_pipeline

d4266c0

mashehu reviewed Aug 27, 2024

View reviewed changes

nf_core/pipeline-template/nextflow.config Outdated Show resolved Hide resolved

make help params snake_case

bc20099

mashehu approved these changes Aug 27, 2024

View reviewed changes

nvnieuwk and others added 7 commits August 27, 2024 14:33

Update nf_core/pipelines/create/templatefeatures.yml

829d594

Co-authored-by: Matthias Hörtenhuber <mashehu@users.noreply.github.com>

Update nf_core/pipelines/lint/plugin_includes.py

1ff7cbc

Co-authored-by: Matthias Hörtenhuber <mashehu@users.noreply.github.com>

apply review suggestions

aed9460

Merge branch 'feat/linting-nf-schema' of github.com:nvnieuwk/tools in…

7612e66

…to feat/linting-nf-schema

update utils_nextflow_pipeline commit messages

4407901

update ignoring of help params

330e43c

update file exists docstring

3600db7

update snapshot

e816faf

nvnieuwk merged commit 7d23535 into nf-core:dev Aug 27, 2024
82 checks passed

nvnieuwk deleted the feat/linting-nf-schema branch August 28, 2024 07:41

mirpedrol mentioned this pull request Sep 26, 2024

Update linting to support to changes introduced in nf-schema #2932

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nf-schema support #3116

nf-schema support #3116

nvnieuwk commented Aug 14, 2024 •

edited

Loading

mirpedrol Aug 20, 2024

nvnieuwk Aug 20, 2024

nvnieuwk Aug 20, 2024

mirpedrol Aug 20, 2024

nvnieuwk Aug 20, 2024

nvnieuwk Aug 21, 2024

mirpedrol left a comment

mashehu Aug 27, 2024

nvnieuwk Aug 27, 2024

mashehu Aug 27, 2024

nvnieuwk Aug 27, 2024

nvnieuwk Aug 27, 2024

mashehu Aug 27, 2024

nvnieuwk Aug 27, 2024

nvnieuwk commented Aug 27, 2024

mirpedrol commented Aug 27, 2024

nvnieuwk commented Aug 27, 2024

nf-schema support #3116

nf-schema support #3116

Conversation

nvnieuwk commented Aug 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mirpedrol left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nvnieuwk commented Aug 27, 2024

mirpedrol commented Aug 27, 2024

nvnieuwk commented Aug 27, 2024

nvnieuwk commented Aug 14, 2024 •

edited

Loading