Skip to content

Commit

Permalink
Merge branch 'dev' into parabricks_alignment
Browse files Browse the repository at this point in the history
  • Loading branch information
maxulysse committed Apr 11, 2024
2 parents bd1981e + e32d0df commit 881bd34
Show file tree
Hide file tree
Showing 517 changed files with 21,564 additions and 3,067 deletions.
10 changes: 1 addition & 9 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,7 @@
"vscode": {
// Set *default* container specific settings.json values on container create.
"settings": {
"python.defaultInterpreterPath": "/opt/conda/bin/python",
"python.linting.enabled": true,
"python.linting.pylintEnabled": true,
"python.formatting.autopep8Path": "/opt/conda/bin/autopep8",
"python.formatting.yapfPath": "/opt/conda/bin/yapf",
"python.linting.flake8Path": "/opt/conda/bin/flake8",
"python.linting.pycodestylePath": "/opt/conda/bin/pycodestyle",
"python.linting.pydocstylePath": "/opt/conda/bin/pydocstyle",
"python.linting.pylintPath": "/opt/conda/bin/pylint"
"python.defaultInterpreterPath": "/opt/conda/bin/python"
},

// Add the IDs of extensions you want installed when the container is created.
Expand Down
15 changes: 14 additions & 1 deletion .editorconfig
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,20 @@ end_of_line = unset
insert_final_newline = unset
trim_trailing_whitespace = unset
indent_style = unset
indent_size = unset
[/subworkflows/nf-core/**]
charset = unset
end_of_line = unset
insert_final_newline = unset
trim_trailing_whitespace = unset
indent_style = unset

[/assets/email*]
indent_size = unset

# ignore Readme
[README.md]
indent_style = unset

# ignore python
[*.{py,md}]
indent_style = unset
14 changes: 8 additions & 6 deletions .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,8 @@ Please use the pre-filled template to save time.
However, don't be put off by this template - other more general issues and suggestions are welcome!
Contributions to the code are even more welcome ;)

:::info
If you need help using or modifying nf-core/sarek then the best place to ask is on the nf-core Slack [#sarek](https://nfcore.slack.com/channels/sarek) channel ([join our Slack here](https://nf-co.re/join/slack)).
:::
> [!NOTE]
> If you need help using or modifying nf-core/sarek then the best place to ask is on the nf-core Slack [#sarek](https://nfcore.slack.com/channels/sarek) channel ([join our Slack here](https://nf-co.re/join/slack)).
## Contribution workflow

Expand All @@ -27,8 +26,11 @@ If you're not used to this workflow with git, you can start with some [docs from

## Tests

You can optionally test your changes by running the pipeline locally. Then it is recommended to use the `debug` profile to
receive warnings about process selectors and other debug info. Example: `nextflow run . -profile debug,test,docker --outdir <OUTDIR>`.
You have the option to test your changes locally by running the pipeline. For receiving warnings about process selectors and other `debug` information, it is recommended to use the debug profile. Execute all the tests with the following command:

```bash
nf-test test --profile debug,test,docker --verbose
```

When you create a pull request with changes, [GitHub Actions](https://github.com/features/actions) will run automatic tests.
Typically, pull-requests are only fully reviewed when these tests are passing, though of course we can help out before then.
Expand Down Expand Up @@ -90,7 +92,7 @@ Once there, use `nf-core schema build` to add to `nextflow_schema.json`.

Sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in the [nf-core pipeline template](https://github.com/nf-core/tools/blob/master/nf_core/pipeline-template/conf/base.config), which has the default process as a single core-process, and then different levels of multi-core configurations for increasingly large memory requirements defined with standardised labels.

The process resources can be passed on to the tool dynamically within the process with the `${task.cpu}` and `${task.memory}` variables in the `script:` block.
The process resources can be passed on to the tool dynamically within the process with the `${task.cpus}` and `${task.memory}` variables in the `script:` block.

### Naming schemes

Expand Down
2 changes: 1 addition & 1 deletion .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Learn more about contributing: [CONTRIBUTING.md](https://github.com/nf-core/sare
- [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/nf-core/sarek/tree/master/.github/CONTRIBUTING.md)
- [ ] If necessary, also make a PR on the nf-core/sarek _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository.
- [ ] Make sure your code lints (`nf-core lint`).
- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker --outdir <OUTDIR>`).
- [ ] Ensure the test suite passes (`nf-test test tests/ --verbose --profile +docker`).
- [ ] Check for unexpected warnings in debug mode (`nextflow run . -profile debug,test,docker --outdir <OUTDIR>`).
- [ ] Usage Documentation in `docs/usage.md` is updated.
- [ ] Output Documentation in `docs/output.md` is updated.
Expand Down
36 changes: 23 additions & 13 deletions .github/RELEASE_CHECKLIST.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,22 @@
> This checklist is for our own reference, to help us prepare a new release
1. Check that everything is ready to go
- Desired [PRs](https://github.com/nf-core/sarek/pulls) are merged
- [GHA CI](https://github.com/nf-core/sarek/actions/workflows/ci.yml) are passing on `dev`
- [GHA linting](https://github.com/nf-core/sarek/actions/workflows/linting.yml) are passing on `dev`

- Desired [PRs](https://github.com/nf-core/sarek/pulls) are merged
- [GHA CI](https://github.com/nf-core/sarek/actions/workflows/ci.yml) are passing on `dev`
- [GHA linting](https://github.com/nf-core/sarek/actions/workflows/linting.yml) are passing on `dev`

2. Increase version number following [semantic versioning](http://semver.org/spec/v2.0.0.html)
3. Choose an appropriate codename for the release (if major or minor)
- i.e. Peaks in [Sarek National Park](https://en.wikipedia.org/wiki/Sarek_National_Park#Topography)

- i.e. Peaks in [Sarek National Park](https://en.wikipedia.org/wiki/Sarek_National_Park#Topography)

4. Sync `dev` and checkout a new branch for the release
5. Bump version:
- `nf-core bump-version . <VERSION>`
- edit `CHANGELOG`

- `nf-core bump-version . <VERSION>`
- edit `CHANGELOG`

6. Make a PR to `master`
7. Wait for reviews
8. Merge said PR
Expand All @@ -21,15 +27,19 @@
11. RT the nf-core automated tweet about the new released version
12. Make a new branch from `dev`
13. Checkout the `CHANGELOG.md` from `master`
- `git checkout upstream/master -- CHANGELOG.md`

- `git checkout upstream/master -- CHANGELOG.md`

14. Add a new `Unreleased` section in `CHANGELOG.md` for the `dev` version
15. Checkout figures from `master`
- `git checkout upstream/master -- docs/images/sarek_indices_subway.svg`
- `git checkout upstream/master -- docs/images/sarek_indices_subway.png`
- `git checkout upstream/master -- docs/images/sarek_subway.svg`
- `git checkout upstream/master -- docs/images/sarek_subway.png`
- `git checkout upstream/master -- docs/images/sarek_workflow.svg`
- `git checkout upstream/master -- docs/images/sarek_workflow.png`

- `git checkout upstream/master -- docs/images/sarek_indices_subway.svg`
- `git checkout upstream/master -- docs/images/sarek_indices_subway.png`
- `git checkout upstream/master -- docs/images/sarek_subway.svg`
- `git checkout upstream/master -- docs/images/sarek_subway.png`
- `git checkout upstream/master -- docs/images/sarek_workflow.svg`
- `git checkout upstream/master -- docs/images/sarek_workflow.png`

16. Make a PR to `dev`
17. Wait for review
18. Merge said PR
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/branch.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ jobs:
# NOTE - this doesn't currently work if the PR is coming from a fork, due to limitations in GitHub actions secrets
- name: Post PR comment
if: failure()
uses: mshick/add-pr-comment@v1
uses: mshick/add-pr-comment@b8f338c590a895d50bcbfa6c5859251edc8952fc # v2
with:
message: |
## This PR is against the `master` branch :x:
Expand Down
57 changes: 54 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: test
name: nf-core CI
# This workflow runs the pipeline with the minimal test dataset to check that it completes without any syntax errors
on:
pull_request:
Expand Down Expand Up @@ -52,6 +52,17 @@ jobs:
NXF_VER:
- "23.04.0"
- "latest-everything"
exclude:
- tags: "sentieon/bwamem"
- tags: "sentieon/dedup"
- tags: "sentieon/dnascope"
- tags: "sentieon/dnascope_joint_germline"
- tags: "sentieon/dnascope_skip_filter"
- tags: "sentieon/haplotyper"
- tags: "sentieon/haplotyper_joint_germline"
- tags: "sentieon/haplotyper_skip_filter"
- NXF_VER: "latest-everything"
tags: "joint_germline"
env:
NXF_ANSI_LOG: false
TEST_DATA_BASE: "${{ github.workspace }}/test-datasets"
Expand Down Expand Up @@ -136,6 +147,9 @@ jobs:
if: matrix.profile == 'conda'
run: conda clean -a

- name: Disk space cleanup
uses: jlumbroso/free-disk-space@v1.3.1

- name: Run pytest-workflow
uses: Wandalen/wretry.action@v1
with:
Expand Down Expand Up @@ -200,16 +214,47 @@ jobs:
- "23.04.0"
- "latest-everything"
exclude:
- tags: "bcftools/concat"
- tags: "bcftools/mpileup"
- tags: "bcftools/sort"
- tags: "bwa/index"
- tags: "bwa/mem"
- tags: "bwamem2/index"
- tags: "bwamem2/mem"
- tags: "cat/cat"
- tags: "cat/fastq"
- tags: "custom/dumpsoftwareversions"
- tags: "cnvkit/batch"
- tags: "deepvariant"
- tags: "dragmap/align"
- tags: "dragmap/hashtable"
- tags: "ensemblvep/download"
- tags: "ensemblvep/vep"
- tags: "fastp"
- tags: "fastqc"
- tags: "fgbio/fastqtobam"
- tags: "freebayes"
- tags: "gatk4/applybqsr"
- tags: "gatk4/baserecalibrator"
- tags: "gatk4/estimatelibrarycomplexity"
- tags: "gatk4/genomicsdbimport"
- tags: "gatk4/haplotypecaller"
- tags: "gatk4/markduplicates"
- tags: "gatk4/mergevcfs"
- tags: "gatk4/mutect2"
- tags: "gatk4spark/applybqsr"
- tags: "gatk4spark/markduplicates"
- tags: "mosdepth"
- tags: "multiqc"
- tags: "samblaster"
- tags: "samtools/mpileup"
- tags: "samtools/stats"
- tags: "snpeff/snpeff"
- tags: "strelka/germline"
- tags: "strelka/somatic"
- tags: "subworkflows/utils_nfvalidation_plugin"
- tags: "tabix/bgziptabix"
- tags: "tabix/tabix"
- tags: "tiddit/sv"
- tags: "untar"
env:
NXF_ANSI_LOG: false
Expand All @@ -223,8 +268,11 @@ jobs:
with:
distribution: "temurin"
java-version: "17"
- name: Setup Nextflow

- name: Install Nextflow ${{ matrix.NXF_VER }}
uses: nf-core/setup-nextflow@v1
with:
version: "${{ matrix.NXF_VER }}"

- name: Cache nf-test installation
id: cache-software
Expand Down Expand Up @@ -279,6 +327,9 @@ jobs:
SENTIEON_AUTH_DATA_BASE64=$(echo -n "$SENTIEON_AUTH_DATA" | base64 -w 0)
nextflow secrets set SENTIEON_AUTH_DATA_BASE64 $SENTIEON_AUTH_DATA_BASE64
- name: Disk space cleanup
uses: jlumbroso/free-disk-space@v1.3.1

# Test the module
- name: Run nf-test
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/clean-up.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
issues: write
pull-requests: write
steps:
- uses: actions/stale@v7
- uses: actions/stale@28ca1036281a5e5922ead5184a1bbf96e5fc984e # v9
with:
stale-issue-message: "This issue has been tagged as awaiting-changes or awaiting-feedback by an nf-core contributor. Remove stale label or add a comment otherwise this issue will be closed in 20 days."
stale-pr-message: "This PR has been tagged as awaiting-changes or awaiting-feedback by an nf-core contributor. Remove stale label or add a comment if it is still useful."
Expand Down
72 changes: 72 additions & 0 deletions .github/workflows/download_pipeline.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
name: Test successful pipeline download with 'nf-core download'

# Run the workflow when:
# - dispatched manually
# - when a PR is opened or reopened to master branch
# - the head branch of the pull request is updated, i.e. if fixes for a release are pushed last minute to dev.
on:
workflow_dispatch:
inputs:
testbranch:
description: "The specific branch you wish to utilize for the test execution of nf-core download."
required: true
default: "dev"
pull_request:
types:
- opened
branches:
- master
pull_request_target:
branches:
- master

env:
NXF_ANSI_LOG: false

jobs:
download:
runs-on: ubuntu-latest
steps:
- name: Install Nextflow
uses: nf-core/setup-nextflow@v1

- uses: actions/setup-python@0a5c61591373683505ea898e09a3ea4f39ef2b9c # v5
with:
python-version: "3.11"
architecture: "x64"
- uses: eWaterCycle/setup-singularity@931d4e31109e875b13309ae1d07c70ca8fbc8537 # v7
with:
singularity-version: 3.8.3

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install git+https://github.com/nf-core/tools.git@dev
- name: Get the repository name and current branch set as environment variable
run: |
echo "REPO_LOWERCASE=${GITHUB_REPOSITORY,,}" >> ${GITHUB_ENV}
echo "REPOTITLE_LOWERCASE=$(basename ${GITHUB_REPOSITORY,,})" >> ${GITHUB_ENV}
echo "REPO_BRANCH=${{ github.event.inputs.testbranch || 'dev' }}" >> ${GITHUB_ENV}
- name: Download the pipeline
env:
NXF_SINGULARITY_CACHEDIR: ./
run: |
nf-core download ${{ env.REPO_LOWERCASE }} \
--revision ${{ env.REPO_BRANCH }} \
--outdir ./${{ env.REPOTITLE_LOWERCASE }} \
--compress "none" \
--container-system 'singularity' \
--container-library "quay.io" -l "docker.io" -l "ghcr.io" \
--container-cache-utilisation 'amend' \
--download-configuration
- name: Inspect download
run: tree ./${{ env.REPOTITLE_LOWERCASE }}

- name: Run the downloaded pipeline
env:
NXF_SINGULARITY_CACHEDIR: ./
NXF_SINGULARITY_HOME_MOUNT: true
run: nextflow run ./${{ env.REPOTITLE_LOWERCASE }}/$( sed 's/\W/_/g' <<< ${{ env.REPO_BRANCH }}) -profile test,singularity --outdir ./results
Loading

0 comments on commit 881bd34

Please sign in to comment.