Add an option to make the new workflow name checks optional #2117

nvnieuwk · 2022-12-09T13:35:12Z

Description of feature

When running nf-core lint on my pipeline it immediately fails with following error:

$ nf-core lint

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 2.7.1 - https://nf-co.re


INFO     Testing pipeline: .                                                                                                                                         
ERROR    Invalid workflow name: must be lowercase without punctuation.

The pipeline name is something we agreed on for all our pipeline so I can't simply change it. So an option to disable this check would be helpful. Thanks in advance!!

The text was updated successfully, but these errors were encountered:

awgymer · 2022-12-09T16:19:33Z

I thought we added code to skip this:

tools/nf_core/create.py

Lines 170 to 180 in eecf1c7

    
           if ( 
        
               "lint" in config_yml 
        
               and "nextflow_config" in config_yml["lint"] 
        
               and "manifest.name" in config_yml["lint"]["nextflow_config"] 
        
           ): 
        
               return param_dict, skip_paths 
        
           # Check that the pipeline name matches the requirements 
        
           if not re.match(r"^[a-z]+$", param_dict["short_name"]): 
        
               raise UserWarning("[red]Invalid workflow name: must be lowercase without punctuation.") 
        
           return param_dict, skip_paths

In theory if you have an .nf-core.yml with manifest.name specified to be skipped then this won't get triggered?
BUT I suppose we didn't think this through because how will you have one in a pipeline you are creating from scratch?

nvnieuwk · 2022-12-09T16:22:40Z

The .nf-core.yml config for the pipeline the command was run on looks like this:

repository_type: pipeline
lint:
  files_exist:
    - assets/nf-core-nfcmggstructural_logo_light.png
    - docs/images/nf-core-nfcmggstructural_logo_light.png
    - docs/images/nf-core-nfcmggstructural_logo_dark.png
    - .github/ISSUE_TEMPLATE/config.yml
    - .github/workflows/awstest.yml
    - .github/workflows/awsfulltest.yml
  nextflow_config:
    - manifest.name
    - manifest.homePage
  multiqc_config:
    - report_comment
  readme:
    - nextflow_badge
  files_unchanged:
    - .github/CONTRIBUTING.md
    - .github/ISSUE_TEMPLATE/bug_report.yml
    - .github/ISSUE_TEMPLATE/feature_request.yml
    - .github/PULL_REQUEST_TEMPLATE.md
    - .github/workflows/linting.yml
    - assets/email_template.txt
    - assets/sendmail_template.txt
    - lib/NfcoreTemplate.groovy
    - .prettierignore
  actions_ci: false

Am I missing something?

awgymer · 2022-12-09T16:28:08Z

I guess it may not be using that file? Could you try running with verbose/debug logging? It should tell you which yml file its using

nvnieuwk · 2022-12-09T16:29:38Z

$ nf-core --verbose lint

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 2.7.1 - https://nf-co.re


DEBUG    Popen(['git', 'cat-file', '--batch-check'], cwd=/home/nvnieuwk/Documents/cmgg/nf-cmgg-germline, universal_newlines=False, shell=None,             cmd.py:931
         istream=<valid stream>)                                                                                                                                     
DEBUG    Using config file: .nf-core.yml                                                                                                                 utils.py:963
DEBUG    Got '.' as path                                                                                                                                 utils.py:215
DEBUG    Found a config cache, loading: /home/nvnieuwk/.nextflow/nf-core/wf-config-cache-653e8a139e1ec6c57ec37bfb4.json                                  utils.py:249
DEBUG    Popen(['git', 'version'], cwd=/home/nvnieuwk/.config/nfcore/nf-core/modules, universal_newlines=False, shell=None, istream=None)                  cmd.py:931
DEBUG    Popen(['git', 'fetch', '-v', '--progress', 'origin'], cwd=/home/nvnieuwk/.config/nfcore/nf-core/modules, universal_newlines=True, shell=None,     cmd.py:931
         istream=None)                                                                                                                                               
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/nvnieuwk/.config/nfcore/nf-core/modules, universal_newlines=False, shell=None, istream=None)       cmd.py:931
DEBUG    Popen(['git', 'merge', 'origin/master'], cwd=/home/nvnieuwk/.config/nfcore/nf-core/modules, universal_newlines=False, shell=None, istream=None)   cmd.py:931
DEBUG    Using config file: /home/nvnieuwk/.config/nfcore/nf-core/modules/.nf-core.yml                                                                   utils.py:963
DEBUG    Using config file: .nf-core.yml                                                                                                                 utils.py:963
DEBUG    The following files were modified by prettier:                                                                                              lint_utils.py:79
                                                                                                                                                                     
         modules.json                                                                                                                                                
                                                                                                                                                                     
                                                                                                                                                                     
DEBUG    No lint config file found: ./.nf-core-lint.yaml                                                                                    components_command.py:182
INFO     Testing pipeline: .                                                                                                                          __init__.py:265
DEBUG    Running lint test: files_exist                                                                                                               __init__.py:328
DEBUG    Running lint test: nextflow_config                                                                                                           __init__.py:328
DEBUG    Running lint test: files_unchanged                                                                                                           __init__.py:328
DEBUG    Using config file: .nf-core.yml                                                                                                                 utils.py:963
ERROR    Invalid workflow name: must be lowercase without punctuation.                                                                                __main__.py:376

awgymer · 2022-12-09T16:33:11Z

Hmm maybe I borked something with .nf-core-lint.yaml there.

I will investigate this in detail and report back.

nvnieuwk · 2022-12-09T16:33:59Z

Thank you!! 🥳

awgymer · 2022-12-09T16:42:18Z

Oh no. I think I know what the issue is! I modified load_tools_config and it returns a tuple now. Most other places this broke anywhere that didn't get updated to reflect that, but because in PipelineCreate we short circuit, first checking "lint" in config_yml it is just never true because "lint" is never one of the values in the tuple.

Will fix later when I am back at my desk.

awgymer · 2022-12-09T22:26:12Z

I think I have the fix in #2122 but could you perhaps either test the branch or provide a minimum reproducible example I can run the code on to confirm @nvnieuwk

nvnieuwk · 2022-12-12T08:35:37Z

Can confirm it works now! Thank you! FYI this is the output:

$ nf-core lint

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 2.8.dev0 - https://nf-co.re


INFO     Testing pipeline: .                                                                                                                                         

╭─ [?] 20 Pipeline Tests Ignored ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                                                   │
│ files_exist: File is ignored: assets/nf-core-nfcmggstructural_logo_light.png                                                                                      │
│ files_exist: File is ignored: docs/images/nf-core-nfcmggstructural_logo_light.png                                                                                 │
│ files_exist: File is ignored: docs/images/nf-core-nfcmggstructural_logo_dark.png                                                                                  │
│ files_exist: File is ignored: .github/ISSUE_TEMPLATE/config.yml                                                                                                   │
│ files_exist: File is ignored: .github/workflows/awstest.yml                                                                                                       │
│ files_exist: File is ignored: .github/workflows/awsfulltest.yml                                                                                                   │
│ nextflow_config: Config variable ignored: manifest.name                                                                                                           │
│ nextflow_config: Config variable ignored: manifest.homePage                                                                                                       │
│ files_unchanged: File ignored due to lint config: .github/CONTRIBUTING.md                                                                                         │
│ files_unchanged: File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml                                                                           │
│ files_unchanged: File does not exist: .github/ISSUE_TEMPLATE/config.yml                                                                                           │
│ files_unchanged: File ignored due to lint config: .github/ISSUE_TEMPLATE/feature_request.yml                                                                      │
│ files_unchanged: File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md                                                                                │
│ files_unchanged: File ignored due to lint config: .github/workflows/linting.yml                                                                                   │
│ files_unchanged: File ignored due to lint config: assets/email_template.txt                                                                                       │
│ files_unchanged: File ignored due to lint config: assets/sendmail_template.txt                                                                                    │
│ files_unchanged: File ignored due to lint config: lib/NfcoreTemplate.groovy                                                                                       │
│ files_unchanged: File ignored due to lint config: .gitignore or .prettierignore or pyproject.toml                                                                 │
│ actions_ci: actions_ci                                                                                                                                            │
│ actions_awstest: 'awstest.yml' workflow not found: ./.github/workflows/awstest.yml                                                                                │
│                                                                                                                                                                   │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ [!] 3 Pipeline Test Warnings ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                                                   │
│ files_exist: File not found: lib/WorkflowNf-cmgg-germline.groovy                                                                                                  │
│ pipeline_name_conventions: Naming does not adhere to nf-core conventions: Contains non alphanumeric characters                                                    │
│ schema_description: No description provided in schema for parameter: vcfanno_lua                                                                                  │
│                                                                                                                                                                   │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ [✗] 3 Pipeline Tests Failed ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                                                   │
│ files_unchanged: .gitattributes does not match the template                                                                                                       │
│ files_unchanged: .github/workflows/linting_comment.yml does not match the template                                                                                │
│ files_unchanged: lib/NfcoreSchema.groovy does not match the template                                                                                              │
│                                                                                                                                                                   │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

╭───────────────────────╮
│ LINT RESULTS SUMMARY  │
├───────────────────────┤
│ [✔] 265 Tests Passed  │
│ [?]  20 Tests Ignored │
│ [!]   3 Test Warnings │
│ [✗]   3 Tests Failed  │
╰───────────────────────╯

Tip: Some of these linting errors can automatically be resolved with the following command:

    nf-core lint   --fix files_unchanged

awgymer · 2022-12-12T10:37:13Z

Hmm. Is it supposed to still be a warning if it's being marked as skip? @mirpedrol

pipeline_name_conventions: Naming does not adhere to nf-core conventions: Contains non alphanumeric characters

nvnieuwk · 2022-12-12T10:38:40Z

I've managed to turn if of by adding pipeline_name_conventions: false to the .nf-core.yml under lint

nvnieuwk · 2022-12-12T10:39:40Z

But yeah it's maybe better to not add the warning if the manifest.name already isn't checked

mirpedrol · 2022-12-12T11:54:57Z

Oh, I thought we had this check for lint but not for create which was when the code failed. Thank you very much for the fix @awgymer. I agree with not adding the warning :)

mirpedrol · 2022-12-12T12:12:26Z

I was testing this now, and it fixes the issue with nf-core lint which now only prints the warning. But still fails with nf-core sync as it calls create() which doesn't take into account the .nf-core.yml file.

nvnieuwk added the enhancement label Dec 9, 2022

mirpedrol added this to the 2.7.2 milestone Dec 9, 2022

awgymer self-assigned this Dec 9, 2022

awgymer mentioned this issue Dec 9, 2022

Fix lint pipeline name #2122

Merged

4 tasks

mirpedrol mentioned this issue Dec 12, 2022

only check pipeline name without dashes if the name is provided by prompt #2123

Merged

4 tasks

ewels closed this as completed Dec 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an option to make the new workflow name checks optional #2117

Add an option to make the new workflow name checks optional #2117

nvnieuwk commented Dec 9, 2022

awgymer commented Dec 9, 2022

nvnieuwk commented Dec 9, 2022

awgymer commented Dec 9, 2022

nvnieuwk commented Dec 9, 2022

awgymer commented Dec 9, 2022

nvnieuwk commented Dec 9, 2022

awgymer commented Dec 9, 2022

awgymer commented Dec 9, 2022

nvnieuwk commented Dec 12, 2022

awgymer commented Dec 12, 2022

nvnieuwk commented Dec 12, 2022

nvnieuwk commented Dec 12, 2022

mirpedrol commented Dec 12, 2022

mirpedrol commented Dec 12, 2022

Add an option to make the new workflow name checks optional #2117

Add an option to make the new workflow name checks optional #2117

Comments

nvnieuwk commented Dec 9, 2022

Description of feature

awgymer commented Dec 9, 2022

nvnieuwk commented Dec 9, 2022

awgymer commented Dec 9, 2022

nvnieuwk commented Dec 9, 2022

awgymer commented Dec 9, 2022

nvnieuwk commented Dec 9, 2022

awgymer commented Dec 9, 2022

awgymer commented Dec 9, 2022

nvnieuwk commented Dec 12, 2022

awgymer commented Dec 12, 2022

nvnieuwk commented Dec 12, 2022

nvnieuwk commented Dec 12, 2022

mirpedrol commented Dec 12, 2022

mirpedrol commented Dec 12, 2022