Document formal script syntax

[bentsherman]

Now that the docs are in better shape, we can begin work towards a formal description of the Nextflow language, as well as prepare the docs and tests for the strict syntax that will be introduced by #4613 and #4744 .

Refer to the changes to snippets and tests for examples of strict syntax:

• replace implicit it closure parameter with explicit parameter
• use quotes for process env input/output name
• move top-level statements into process / function / workflow definitions

Currently the syntax description is split across several pages:

• Scripts
• Processes
• Workflows
• Modules

It would be nice to have a single page that describes the entire language. At the same time, I don't want to move these pages around much more, and they are too hefty individually to combine them.

Since these pages cover a lot of other topics like APIs and runtime behavior, maybe it would be better to add a new page e.g. "Reference > Syntax" which gives a top-down syntax description.

Looking to the Groovy docs as an example, I feel like the Scripts page could be expanded to also describe process/workflow/include syntax. It just feels weird to mention them here when there are already dedicated pages.

[netlify]

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	3f6f377
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/671928afdd4b3c000873be4e
😎 Deploy Preview	https://deploy-preview-5336--nextflow-docs-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[christopher-hakkaart]

I agree having everything on the same page would be too much, and I'm also not sure if that would make the most sense for someone accessing the docs.

So I guess the question is where the bulk of the syntax should be described in detail (a single page or separate pages).

I wonder if there can be a compromise where the bulk or common syntax is described on a Syntax page but also have Syntax sections (e.g., Groovy closures). The "main" syntax page could have small sections about the other syntax sections and then link out, rather than build everything into one page.

[bentsherman]

I was thinking something similar, have a top-level page that links out to the existing pages. Two questions around that:

1. Do you think this page should be under "Developing pipelines" or a reference page?
2. Do you think the page should absorb the current "Scripts" content or also link to it?

I think we could just repurpose the "Scripts" page with an outline like this:

• (overview)
• workflows (briefly describe and link out)
• processes (briefly describe and link out)
• functions
• "basic syntax"
  • numbers, strings, lists, variables, closures, etc
  • link out to "Syntax" reference page

Then add a "Syntax" reference page which is actually a full description of the script grammar. This way the "Scripts" page doesn't have to be comprehensive on basic syntax, it can just provide the most important examples. The reference page can get very specific on things like punctuation, what kind of code is allowed where, etc

In any case, we need to have a broader conversation on how to evolve the docs from here, but I think for now I can do a first draft and let us iterate on that

[christopher-hakkaart]

I was thinking something similar:

• Scripts (living under developing pipelines) have a high-level overview with minimal examples and lots of links to reference pages. The purpose is to describe the language and how it fits together but not get into details about every little thing.
• New reference page that describes syntax in detail and gives examples.
  • A focus would be to make everything very findable, so some clear grouping would be essential.

I think this would align nicely with splitting user and reference content too.

[pditommaso]

Please keep PR small

[christopher-hakkaart]

Adding a few comments for visibility now.

[bentsherman]

Okay I think I found a good scope for this PR:

• Add a new syntax reference page which describes the entire Nextflow syntax
• Cleanup existing pages now that they don't have to cover syntax so much, just link to syntax page instead
• Add section to scripts page on how to transition from code snippet to full pipeline script with processes, workflows, etc
• Update code snippets to comply with formal syntax
• Remove unnecessary references to Groovy in favor of Nextflow

Some items for future PRs:

• Update all e2e tests to comply with formal syntax
• Expand standard library docs to cover things like List, Map, Set APIs
• Continue to massage the pages under Developing Pipelines to be more focused

Thanks @christopher-hakkaart for the comments, I will address them shortly

[christopher-hakkaart]

Looking good. I've reviewed every md page - except the new syntax page.

I like the direction of the syntax page but I find some of the descriptions too short. E.g., comparing the comments section here and the groovy sytnax page. What do you think about adding more context to some of these? Do you think it would be too repetitive?

[ewels]

Had a quick run through of just the new syntax.md file and dropped a few first impressions.

This is great!

[christopher-hakkaart]

This is looking awesome.

I've added some suggestions to turn passive into active voice and split some sentences. I don't see any of these as blockers but opportunities to improve clarity. If you're happy with these I think it's good to go.

[christopher-hakkaart]

Thanks for adding my suggestions. The docs look good to me.

[pditommaso]

I've left out the mention to deprecation because deserver further discussion 👉 3f6f377