Merge branch 'nf-core-template-merge-3.0.1' of github.com:nf-core/rar…

…edisease into dev
nf-core · Oct 9, 2024 · 14a7134 · 14a7134
2 parents a087573 + bbdf2a8
commit 14a7134
Show file tree

Hide file tree

Showing 42 changed files with 801 additions and 802 deletions.
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
@@ -19,7 +19,7 @@ If you'd like to write some code for nf-core/raredisease, the standard workflow
 1. Check that there isn't already an issue about your idea in the [nf-core/raredisease issues](https://github.com/nf-core/raredisease/issues) to avoid duplicating work. If there isn't one already, please create one so that others know you're working on this
 2. [Fork](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) the [nf-core/raredisease repository](https://github.com/nf-core/raredisease) to your GitHub account
 3. Make the necessary changes / additions within your forked repository following [Pipeline conventions](#pipeline-contribution-conventions)
-4. Use `nf-core schema build` and add any new parameters to the pipeline JSON schema (requires [nf-core tools](https://github.com/nf-core/tools) >= 1.10).
+4. Use `nf-core pipelines schema build` and add any new parameters to the pipeline JSON schema (requires [nf-core tools](https://github.com/nf-core/tools) >= 1.10).
 5. Submit a Pull Request against the `dev` branch and wait for the code to be reviewed and merged
 
 If you're not used to this workflow with git, you can start with some [docs from GitHub](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests) or even their [excellent `git` resources](https://try.github.io/).
@@ -40,7 +40,7 @@ There are typically two types of tests that run:
 ### Lint tests
 
 `nf-core` has a [set of guidelines](https://nf-co.re/developers/guidelines) which all pipelines must adhere to.
-To enforce these and ensure that all pipelines stay in sync, we have developed a helper tool which runs checks on the pipeline code. This is in the [nf-core/tools repository](https://github.com/nf-core/tools) and once installed can be run locally with the `nf-core lint <pipeline-directory>` command.
+To enforce these and ensure that all pipelines stay in sync, we have developed a helper tool which runs checks on the pipeline code. This is in the [nf-core/tools repository](https://github.com/nf-core/tools) and once installed can be run locally with the `nf-core pipelines lint <pipeline-directory>` command.
 
 If any failures or warnings are encountered, please follow the listed URL for more documentation.
 
@@ -75,7 +75,7 @@ If you wish to contribute a new step, please use the following coding standards:
 2. Write the process block (see below).
 3. Define the output channel if needed (see below).
 4. Add any new parameters to `nextflow.config` with a default (see below).
-5. Add any new parameters to `nextflow_schema.json` with help text (via the `nf-core schema build` tool).
+5. Add any new parameters to `nextflow_schema.json` with help text (via the `nf-core pipelines schema build` tool).
 6. Add sanity checks and validation for all relevant parameters.
 7. Perform local tests to validate that the new code works as expected.
 8. If applicable, add a new test command in `.github/workflow/ci.yml`.
@@ -86,11 +86,11 @@ If you wish to contribute a new step, please use the following coding standards:
 
 Parameters should be initialised / defined with default values in `nextflow.config` under the `params` scope.
 
-Once there, use `nf-core schema build` to add to `nextflow_schema.json`.
+Once there, use `nf-core pipelines schema build` to add to `nextflow_schema.json`.
 
 ### Default processes resource requirements
 
-Sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in the [nf-core pipeline template](https://github.com/nf-core/tools/blob/master/nf_core/pipeline-template/conf/base.config), which has the default process as a single core-process, and then different levels of multi-core configurations for increasingly large memory requirements defined with standardised labels.
+Sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in the [nf-core pipeline template](https://github.com/nf-core/tools/blob/main/nf_core/pipeline-template/conf/base.config), which has the default process as a single core-process, and then different levels of multi-core configurations for increasingly large memory requirements defined with standardised labels.
 
 The process resources can be passed on to the tool dynamically within the process with the `${task.cpus}` and `${task.memory}` variables in the `script:` block.
 
@@ -103,7 +103,7 @@ Please use the following naming schemes, to make it easy to understand what is g
 
 ### Nextflow version bumping
 
-If you are using a new feature from core Nextflow, you may bump the minimum required version of nextflow in the pipeline with: `nf-core bump-version --nextflow . [min-nf-version]`
+If you are using a new feature from core Nextflow, you may bump the minimum required version of nextflow in the pipeline with: `nf-core pipelines bump-version --nextflow . [min-nf-version]`
 
 ### Images and figures
 

diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
@@ -17,7 +17,7 @@ Learn more about contributing: [CONTRIBUTING.md](https://github.com/nf-core/rare
 - [ ] If you've fixed a bug or added code that should be tested, add tests!
 - [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/nf-core/raredisease/tree/master/.github/CONTRIBUTING.md)
 - [ ] If necessary, also make a PR on the nf-core/raredisease _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository.
-- [ ] Make sure your code lints (`nf-core lint`).
+- [ ] Make sure your code lints (`nf-core pipelines lint`).
 - [ ] Ensure the test suite passes (`nextflow run . -profile test,docker --outdir <OUTDIR>`).
 - [ ] Ensure the test suite passes (`nextflow run . -profile test_one_sample,docker --outdir <OUTDIR>`).
 - [ ] Check for unexpected warnings in debug mode (`nextflow run . -profile debug,test,docker --outdir <OUTDIR>`).

diff --git a/.github/workflows/awsfulltest.yml b/.github/workflows/awsfulltest.yml
@@ -1,18 +1,35 @@
 name: nf-core AWS full size tests
-# This workflow is triggered on published releases.
+# This workflow is triggered on PRs opened against the master branch.
 # It can be additionally triggered manually with GitHub actions workflow dispatch button.
 # It runs the -profile 'test_full' on AWS batch
 
 on:
-  release:
-    types: [published]
+  pull_request:
+    branches:
+      - master
   workflow_dispatch:
+  pull_request_review:
+    types: [submitted]
+
 jobs:
   run-platform:
     name: Run AWS full tests
-    if: github.repository == 'nf-core/raredisease'
+    # run only if the PR is approved by at least 2 reviewers and against the master branch or manually triggered
+    if: github.repository == 'nf-core/raredisease' && github.event.review.state == 'approved' && github.event.pull_request.base.ref == 'master' || github.event_name == 'workflow_dispatch'
     runs-on: ubuntu-latest
     steps:
+      - uses: octokit/request-action@v2.x
+        id: check_approvals
+        with:
+          route: GET /repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/reviews
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      - id: test_variables
+        if: github.event_name != 'workflow_dispatch'
+        run: |
+          JSON_RESPONSE='${{ steps.check_approvals.outputs.data }}'
+          CURRENT_APPROVALS_COUNT=$(echo $JSON_RESPONSE | jq -c '[.[] | select(.state | contains("APPROVED")) ] | length')
+          test $CURRENT_APPROVALS_COUNT -ge 2 || exit 1 # At least 2 approvals are required
       - name: Launch workflow via Seqera Platform
         uses: seqeralabs/action-tower-launch@v2
         # TODO nf-core: You can customise AWS full pipeline tests as required

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -7,6 +7,7 @@ on:
   pull_request:
   release:
     types: [published]
+  workflow_dispatch:
 
 env:
   NXF_ANSI_LOG: false
@@ -24,11 +25,11 @@ jobs:
     strategy:
       matrix:
         NXF_VER:
-          - "23.04.0"
+          - "24.04.2"
           - "latest-everything"
         parameters:
-          - "-profile test,docker"
-          - "-profile test_one_sample,docker"
+          - "-profile test"
+          - "-profile test_one_sample"
     steps:
       - name: Check out pipeline code
         uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4
@@ -41,6 +42,16 @@ jobs:
       - name: Disk space cleanup
         uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1
 
-      - name: Run pipeline with test data
+      - name: Run pipeline with test data (docker)
         run: |
-          nextflow run ${GITHUB_WORKSPACE} ${{ matrix.parameters }} -stub --outdir ./results
+          nextflow run ${GITHUB_WORKSPACE} ${{ matrix.parameters }},docker -stub --outdir ./results
+
+      - name: Run pipeline with test data (singularity)
+        run: |
+          nextflow run ${GITHUB_WORKSPACE} ${{ matrix.parameters }},singularity -stub --outdir ./results
+        if: "${{ github.base_ref == 'master' }}"
+
+      - name: Run pipeline with test data (conda)
+        run: |
+          nextflow run ${GITHUB_WORKSPACE} -profile test,conda --outdir ./results
+        if: "${{ github.base_ref == 'master' }}"
diff --git a/.github/workflows/download_pipeline.yml b/.github/workflows/download_pipeline.yml
@@ -1,4 +1,4 @@
-name: Test successful pipeline download with 'nf-core download'
+name: Test successful pipeline download with 'nf-core pipelines download'
 
 # Run the workflow when:
 #  - dispatched manually
@@ -8,7 +8,7 @@ on:
   workflow_dispatch:
     inputs:
       testbranch:
-        description: "The specific branch you wish to utilize for the test execution of nf-core download."
+        description: "The specific branch you wish to utilize for the test execution of nf-core pipelines download."
         required: true
         default: "dev"
   pull_request:
@@ -39,9 +39,11 @@ jobs:
         with:
           python-version: "3.12"
           architecture: "x64"
-      - uses: eWaterCycle/setup-singularity@931d4e31109e875b13309ae1d07c70ca8fbc8537 # v7
+
+      - name: Setup Apptainer
+        uses: eWaterCycle/setup-apptainer@4bb22c52d4f63406c49e94c804632975787312b3 # v2.0.0
         with:
-          singularity-version: 3.8.3
+          apptainer-version: 1.3.4
 
       - name: Install dependencies
         run: |
@@ -54,18 +56,64 @@ jobs:
           echo "REPOTITLE_LOWERCASE=$(basename ${GITHUB_REPOSITORY,,})" >> ${GITHUB_ENV}
           echo "REPO_BRANCH=${{ github.event.inputs.testbranch || 'dev' }}" >> ${GITHUB_ENV}
 
+      - name: Make a cache directory for the container images
+        run: |
+          mkdir -p ./singularity_container_images
+
       - name: Download the pipeline
         env:
-          NXF_SINGULARITY_CACHEDIR: ./
+          NXF_SINGULARITY_CACHEDIR: ./singularity_container_images
         run: |
-          nf-core download ${{ env.REPO_LOWERCASE }} \
+          nf-core pipelines download ${{ env.REPO_LOWERCASE }} \
           --revision ${{ env.REPO_BRANCH }} \
           --outdir ./${{ env.REPOTITLE_LOWERCASE }} \
           --compress "none" \
           --container-system 'singularity' \
-          --container-library "quay.io" -l "docker.io" -l "ghcr.io" \
+          --container-library "quay.io" -l "docker.io" -l "community.wave.seqera.io" \
           --container-cache-utilisation 'amend' \
-          --download-configuration
+          --download-configuration 'yes'
 
       - name: Inspect download
         run: tree ./${{ env.REPOTITLE_LOWERCASE }}
+
+      - name: Count the downloaded number of container images
+        id: count_initial
+        run: |
+          image_count=$(ls -1 ./singularity_container_images | wc -l | xargs)
+          echo "Initial container image count: $image_count"
+          echo "IMAGE_COUNT_INITIAL=$image_count" >> ${GITHUB_ENV}
+
+      - name: Run the downloaded pipeline (stub)
+        id: stub_run_pipeline
+        continue-on-error: true
+        env:
+          NXF_SINGULARITY_CACHEDIR: ./singularity_container_images
+          NXF_SINGULARITY_HOME_MOUNT: true
+        run: nextflow run ./${{ env.REPOTITLE_LOWERCASE }}/$( sed 's/\W/_/g' <<< ${{ env.REPO_BRANCH }}) -stub -profile test,singularity --outdir ./results
+      - name: Run the downloaded pipeline (stub run not supported)
+        id: run_pipeline
+        if: ${{ job.steps.stub_run_pipeline.status == failure() }}
+        env:
+          NXF_SINGULARITY_CACHEDIR: ./singularity_container_images
+          NXF_SINGULARITY_HOME_MOUNT: true
+        run: nextflow run ./${{ env.REPOTITLE_LOWERCASE }}/$( sed 's/\W/_/g' <<< ${{ env.REPO_BRANCH }}) -profile test,singularity --outdir ./results
+
+      - name: Count the downloaded number of container images
+        id: count_afterwards
+        run: |
+          image_count=$(ls -1 ./singularity_container_images | wc -l | xargs)
+          echo "Post-pipeline run container image count: $image_count"
+          echo "IMAGE_COUNT_AFTER=$image_count" >> ${GITHUB_ENV}
+
+      - name: Compare container image counts
+        run: |
+          if [ "${{ env.IMAGE_COUNT_INITIAL }}" -ne "${{ env.IMAGE_COUNT_AFTER }}" ]; then
+            initial_count=${{ env.IMAGE_COUNT_INITIAL }}
+            final_count=${{ env.IMAGE_COUNT_AFTER }}
+            difference=$((final_count - initial_count))
+            echo "$difference additional container images were \n downloaded at runtime . The pipeline has no support for offline runs!"
+            tree ./singularity_container_images
+            exit 1
+          else
+            echo "The pipeline can be downloaded successfully!"
+          fi
diff --git a/.github/workflows/linting.yml b/.github/workflows/linting.yml
@@ -1,6 +1,6 @@
 name: nf-core linting
 # This workflow is triggered on pushes and PRs to the repository.
-# It runs the `nf-core lint` and markdown lint tests to ensure
+# It runs the `nf-core pipelines lint` and markdown lint tests to ensure
 # that the code meets the nf-core guidelines.
 on:
   push:
@@ -41,17 +41,32 @@ jobs:
           python-version: "3.12"
           architecture: "x64"
 
+      - name: read .nf-core.yml
+        uses: pietrobolcato/action-read-yaml@1.1.0
+        id: read_yml
+        with:
+          config: ${{ github.workspace }}/.nf-core.yml
+
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install nf-core
+          pip install nf-core==${{ steps.read_yml.outputs['nf_core_version'] }}
+
+      - name: Run nf-core pipelines lint
+        if: ${{ github.base_ref != 'master' }}
+        env:
+          GITHUB_COMMENTS_URL: ${{ github.event.pull_request.comments_url }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GITHUB_PR_COMMIT: ${{ github.event.pull_request.head.sha }}
+        run: nf-core -l lint_log.txt pipelines lint --dir ${GITHUB_WORKSPACE} --markdown lint_results.md
 
-      - name: Run nf-core lint
+      - name: Run nf-core pipelines lint --release
+        if: ${{ github.base_ref == 'master' }}
         env:
           GITHUB_COMMENTS_URL: ${{ github.event.pull_request.comments_url }}
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           GITHUB_PR_COMMIT: ${{ github.event.pull_request.head.sha }}
-        run: nf-core -l lint_log.txt lint --dir ${GITHUB_WORKSPACE} --markdown lint_results.md
+        run: nf-core -l lint_log.txt pipelines lint --release --dir ${GITHUB_WORKSPACE} --markdown lint_results.md
 
       - name: Save PR number
         if: ${{ always() }}

diff --git a/.github/workflows/linting_comment.yml b/.github/workflows/linting_comment.yml
@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Download lint results
-        uses: dawidd6/action-download-artifact@09f2f74827fd3a8607589e5ad7f9398816f540fe # v3
+        uses: dawidd6/action-download-artifact@bf251b5aa9c2f7eeb574a96ee720e24f801b7c11 # v6
         with:
           workflow: linting.yml
           workflow_conclusion: completed

diff --git a/.github/workflows/release-announcements.yml b/.github/workflows/release-announcements.yml
@@ -12,7 +12,7 @@ jobs:
       - name: get topics and convert to hashtags
         id: get_topics
         run: |
-          echo "topics=$(curl -s https://nf-co.re/pipelines.json | jq -r '.remote_workflows[] | select(.full_name == "${{ github.repository }}") | .topics[]' | awk '{print "#"$0}' | tr '\n' ' ')"  >> $GITHUB_OUTPUT
+          echo "topics=$(curl -s https://nf-co.re/pipelines.json | jq -r '.remote_workflows[] | select(.full_name == "${{ github.repository }}") | .topics[]' | awk '{print "#"$0}' | tr '\n' ' ')" | sed 's/-//g' >> $GITHUB_OUTPUT
 
       - uses: rzr/fediverse-action@master
         with:

diff --git a/.github/workflows/template_version_comment.yml b/.github/workflows/template_version_comment.yml
@@ -0,0 +1,43 @@
+name: nf-core template version comment
+# This workflow is triggered on PRs to check if the pipeline template version matches the latest nf-core version.
+# It posts a comment to the PR, even if it comes from a fork.
+
+on: pull_request_target
+
+jobs:
+  template_version:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out pipeline code
+        uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4
+
+      - name: Read template version from .nf-core.yml
+        uses: pietrobolcato/action-read-yaml@1.0.0
+        id: read_yml
+        with:
+          config: ${{ github.workspace }}/.nf-core.yml
+
+      - name: Install nf-core
+        run: |
+          python -m pip install --upgrade pip
+          pip install nf-core==${{ steps.read_yml.outputs['nf_core_version'] }}
+
+      - name: Check nf-core outdated
+        id: nf_core_outdated
+        run: pip list --outdated | grep nf-core
+
+      - name: Post nf-core template version comment
+        uses: mshick/add-pr-comment@b8f338c590a895d50bcbfa6c5859251edc8952fc # v2
+        if: |
+          ${{ steps.nf_core_outdated.outputs.stdout }} =~ 'nf-core'
+        with:
+          repo-token: ${{ secrets.NF_CORE_BOT_AUTH_TOKEN }}
+          allow-repeats: false
+          message: |
+            ## :warning: Newer version of the nf-core template is available.
+
+            Your pipeline is using an old version of the nf-core template: ${{ steps.read_yml.outputs['nf_core_version'] }}.
+            Please update your pipeline to the latest version.
+
+            For more documentation on how to update your pipeline, please see the [nf-core documentation](https://github.com/nf-core/tools?tab=readme-ov-file#sync-a-pipeline-with-the-template) and [Synchronisation documentation](https://nf-co.re/docs/contributing/sync).
+          #
diff --git a/.gitpod.yml b/.gitpod.yml
@@ -4,17 +4,14 @@ tasks:
     command: |
       pre-commit install --install-hooks
       nextflow self-update
-  - name: unset JAVA_TOOL_OPTIONS
-    command: |
-      unset JAVA_TOOL_OPTIONS
 
 vscode:
   extensions: # based on nf-core.nf-core-extensionpack
-    - esbenp.prettier-vscode # Markdown/CommonMark linting and style checking for Visual Studio Code
+    #- esbenp.prettier-vscode # Markdown/CommonMark linting and style checking for Visual Studio Code
     - EditorConfig.EditorConfig # override user/workspace settings with settings found in .editorconfig files
     - Gruntfuggly.todo-tree # Display TODO and FIXME in a tree view in the activity bar
     - mechatroner.rainbow-csv # Highlight columns in csv files in different colors
-    # - nextflow.nextflow                    # Nextflow syntax highlighting
+    - nextflow.nextflow # Nextflow syntax highlighting
     - oderwat.indent-rainbow # Highlight indentation level
     - streetsidesoftware.code-spell-checker # Spelling checker for source code
     - charliermarsh.ruff # Code linter Ruff
diff --git a/.nf-core.yml b/.nf-core.yml
@@ -1,3 +1,4 @@
+bump_version: null
 lint:
   files_exist:
     - conf/modules.config
@@ -7,6 +8,18 @@ lint:
     - docs/images/nf-core-raredisease_logo_dark.png
     - docs/images/nf-core-raredisease_logo_light.png
     - assets/nf-core-raredisease_logo_light.png
-  modules_config: False
+  modules_config: false
+nf_core_version: 3.0.1
+org_path: null
 repository_type: pipeline
-nf_core_version: "2.14.1"
+template:
+  author: Clinical Genomics Stockholm
+  description: call and score variants from WGS/WES of rare disease patients
+  force: false
+  is_nfcore: true
+  name: raredisease
+  org: nf-core
+  outdir: .
+  skip_features: null
+  version: 2.3.0dev
+update: null