Update linting to support to changes introduced in nf-schema

[nvnieuwk]

Description of feature

Some changes to the linting have to be made to make the migration from nf-validation to nf-schema possible. IMO we should temporarily support both nf-validation and nf-schema for a couple versions of tools. Eventually we should fully deprecate nf-validation and for everyone to migrate to nf-schema.

Urgent changes

1. Linting currently results in an error with draft 2020-12 schemas because definitions has been changed to defs:

Error

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 2.13.1 - https://nf-co.re


INFO     Testing pipeline: .                                                                                                                                                        
Running lint checks ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7 of 19 » pipeline_todosERROR    [✗] Pipeline schema does not follow nf-core specs:                                                                                                                         
          Schema has allOf, but no definitions                                                                                                                                      
Running lint checks ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10 of 19 » schema_lintERROR    [✗] Pipeline schema does not follow nf-core specs:                                                                                                                         
          Schema has allOf, but no definitions                                                                                                                                      
CRITICAL Critical error: [✗] Pipeline schema does not follow nf-core specs:                                                                                                         
          Schema has allOf, but no definitions                                                                                                                                      
INFO     Stopping tests...                                                                                                                                                          
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/nvnieuwk/anaconda3/envs/nextflow/bin/nf-core:8 in <module>                                 │
│                                                                                                  │
│   5 from nf_core.__main__ import run_nf_core                                                     │
│   6 if __name__ == '__main__':                                                                   │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(run_nf_core())                                                                  │
│   9                                                                                              │
│                                                                                                  │
│ /home/nvnieuwk/anaconda3/envs/nextflow/lib/python3.10/site-packages/nf_core/__main__.py:141 in   │
│ run_nf_core                                                                                      │
│                                                                                                  │
│    138 │   │   │   log.debug(f"Could not check latest version: {e}")                             │
│    139 │   │   stderr.print("\n")                                                                │
│    140 │   # Launch the click cli                                                                │
│ ❱  141 │   nf_core_cli(auto_envvar_prefix="NFCORE")                                              │
│    142                                                                                           │
│    143                                                                                           │
│    144 @tui()                                                                                    │
│                                                                                                  │
│ /home/nvnieuwk/anaconda3/envs/nextflow/lib/python3.10/site-packages/click/core.py:1157 in        │
│ __call__                                                                                         │
│                                                                                                  │
│   1154 │                                                                                         │
│   1155 │   def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:                           │
│   1156 │   │   """Alias for :meth:`main`."""                                                     │
│ ❱ 1157 │   │   return self.main(*args, **kwargs)                                                 │
│   1158                                                                                           │
│   1159                                                                                           │
│   1160 class Command(BaseCommand):                                                               │
│                                                                                                  │
│ /home/nvnieuwk/anaconda3/envs/nextflow/lib/python3.10/site-packages/rich_click/rich_command.py:1 │
│ 26 in main                                                                                       │
│                                                                                                  │
│   123 │   │   try:                                                                               │
│   124 │   │   │   try:                                                                           │
│   125 │   │   │   │   with self.make_context(prog_name, args, **extra) as ctx:                   │
│ ❱ 126 │   │   │   │   │   rv = self.invoke(ctx)                                                  │
│   127 │   │   │   │   │   if not standalone_mode:                                                │
│   128 │   │   │   │   │   │   return rv                                                          │
│   129 │   │   │   │   │   # it's not safe to `ctx.exit(rv)` here!                                │
│                                                                                                  │
│ /home/nvnieuwk/anaconda3/envs/nextflow/lib/python3.10/site-packages/click/core.py:1688 in invoke │
│                                                                                                  │
│   1685 │   │   │   │   super().invoke(ctx)                                                       │
│   1686 │   │   │   │   sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)                    │
│   1687 │   │   │   │   with sub_ctx:                                                             │
│ ❱ 1688 │   │   │   │   │   return _process_result(sub_ctx.command.invoke(sub_ctx))               │
│   1689 │   │                                                                                     │
│   1690 │   │   # In chain mode we create the contexts step by step, but after the                │
│   1691 │   │   # base command has been invoked.  Because at that point we do not                 │
│                                                                                                  │
│ /home/nvnieuwk/anaconda3/envs/nextflow/lib/python3.10/site-packages/click/core.py:1434 in invoke │
│                                                                                                  │
│   1431 │   │   │   echo(style(message, fg="red"), err=True)                                      │
│   1432 │   │                                                                                     │
│   1433 │   │   if self.callback is not None:                                                     │
│ ❱ 1434 │   │   │   return ctx.invoke(self.callback, **ctx.params)                                │
│   1435 │                                                                                         │
│   1436 │   def shell_complete(self, ctx: Context, incomplete: str) -> t.List["CompletionItem"]:  │
│   1437 │   │   """Return a list of completions for the incomplete value. Looks                   │
│                                                                                                  │
│ /home/nvnieuwk/anaconda3/envs/nextflow/lib/python3.10/site-packages/click/core.py:783 in invoke  │
│                                                                                                  │
│    780 │   │                                                                                     │
│    781 │   │   with augment_usage_errors(__self):                                                │
│    782 │   │   │   with ctx:                                                                     │
│ ❱  783 │   │   │   │   return __callback(*args, **kwargs)                                        │
│    784 │                                                                                         │
│    785 │   def forward(                                                                          │
│    786 │   │   __self, __cmd: "Command", *args: t.Any, **kwargs: t.Any  # noqa: B902             │
│                                                                                                  │
│ /home/nvnieuwk/anaconda3/envs/nextflow/lib/python3.10/site-packages/click/decorators.py:33 in    │
│ new_func                                                                                         │
│                                                                                                  │
│    30 │   """                                                                                    │
│    31 │                                                                                          │
│    32 │   def new_func(*args: "P.args", **kwargs: "P.kwargs") -> "R":                            │
│ ❱  33 │   │   return f(get_current_context(), *args, **kwargs)                                   │
│    34 │                                                                                          │
│    35 │   return update_wrapper(new_func, f)                                                     │
│    36                                                                                            │
│                                                                                                  │
│ /home/nvnieuwk/anaconda3/envs/nextflow/lib/python3.10/site-packages/nf_core/__main__.py:607 in   │
│ lint                                                                                             │
│                                                                                                  │
│    604 │                                                                                         │
│    605 │   # Run the lint tests!                                                                 │
│    606 │   try:                                                                                  │
│ ❱  607 │   │   lint_obj, module_lint_obj, subworkflow_lint_obj = run_linting(                    │
│    608 │   │   │   dir,                                                                          │
│    609 │   │   │   release,                                                                      │
│    610 │   │   │   fix,                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: not enough values to unpack (expected 3, got 2)

2. Check if the versions of plugins are pinned, maybe have this as a warning during development, but it should be a failure for the release linting. nf-schema itself will also warn for this, but I think it would be best if we check this for all plugins

Not that urgent changes

1. Lint for the draft version depending on the plugin used. (nf-schema already throws a custom error when the draft version isn't correct)
2. Lint for invalid JSON schemas. The library used in nf-schema gives a very weird error when the JSON schema structure is weird. Some additional linting can definitely help out with debugging.
3. Check if the new configuration options are used instead of the old parameters (the ones starting with validation. (This should only apply if the pipeline uses nf-schema and not when the pipeline is still using nf-validation)
4. Add a warning that migration to nf-schema is advised, but not necessary at this point

Feel free to add additional points :)

[ewels]

Also check the plugin import statements and fail if nf-validation is being imported.

[nvnieuwk]

Good idea especially since this results in weird errors (see nextflow-io/nf-schema#28)

[mirpedrol]

Done by #3116, thanks!