Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More complete CI #806

Merged
merged 50 commits into from
Nov 4, 2022
Merged
Show file tree
Hide file tree
Changes from 46 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
e06695f
starting to reorganize conf files
maxulysse Nov 2, 2022
5fee072
fix lint
maxulysse Nov 2, 2022
c2f6ee9
fix lint
maxulysse Nov 2, 2022
2e728fc
add first tests
maxulysse Nov 2, 2022
a0173b8
fix path
maxulysse Nov 2, 2022
72b4140
fix workflow
maxulysse Nov 2, 2022
34965c0
fixing aligner tests
maxulysse Nov 2, 2022
1085042
fix aligner tests
maxulysse Nov 2, 2022
3122950
update annotation tests
maxulysse Nov 2, 2022
68da3de
test only with docker for now
maxulysse Nov 2, 2022
4c05d1c
testing out adding an extra layer of tags
maxulysse Nov 2, 2022
986fe91
add info as comments instead
maxulysse Nov 2, 2022
b9de3cf
sort files
maxulysse Nov 2, 2022
7715760
fix file names
maxulysse Nov 2, 2022
630eab9
alignement_to_fastq
maxulysse Nov 2, 2022
767c9ef
markduplicates
maxulysse Nov 2, 2022
66d770b
rename files
maxulysse Nov 2, 2022
83ed74a
prepare_recalibration + recalibrate
maxulysse Nov 2, 2022
13ab016
Merge remote-tracking branch 'upstream/dev' into dev_more_complete_CI
maxulysse Nov 3, 2022
9fc7b31
code polish + intervals tests
maxulysse Nov 3, 2022
a3f502c
prettier . -w
maxulysse Nov 3, 2022
c5a4531
recalibrate
maxulysse Nov 3, 2022
b6ea03e
trimming
maxulysse Nov 3, 2022
e14cc7f
umi
maxulysse Nov 3, 2022
4a3685d
default + trimming | split fastq > fastp
maxulysse Nov 3, 2022
bfc72f6
target > intervals
maxulysse Nov 3, 2022
0cc8fac
pair > default
maxulysse Nov 3, 2022
20ba4bd
cnvkit
maxulysse Nov 3, 2022
10c74f1
controlfreec
maxulysse Nov 3, 2022
12fc3b7
more cnvkit
maxulysse Nov 3, 2022
2298738
deepvariant
maxulysse Nov 3, 2022
696796b
freebayes
maxulysse Nov 3, 2022
e136625
haplotypecaller
maxulysse Nov 3, 2022
a922a39
manta
maxulysse Nov 3, 2022
7f56a74
mpileup + controlfreec
maxulysse Nov 3, 2022
ec13c41
mutect2
maxulysse Nov 3, 2022
cfc634f
msisensorpro + strelka + strelka_bp + tiddit
maxulysse Nov 3, 2022
8ca37fa
test with singularity and conda as well
maxulysse Nov 3, 2022
1694983
update GHA workflows
maxulysse Nov 3, 2022
09241c5
remove tags
maxulysse Nov 3, 2022
7faac67
remove more conda tests
maxulysse Nov 3, 2022
5c6b995
remove more tags
maxulysse Nov 3, 2022
e8c20a1
update default
maxulysse Nov 3, 2022
9184570
less default tests
maxulysse Nov 3, 2022
819d284
less default tests
maxulysse Nov 3, 2022
f495369
exclude conda from annotation tests
maxulysse Nov 3, 2022
c31691f
exclude more tests + remove md5sum for some conda tests
maxulysse Nov 4, 2022
973bbcb
more conda md5sum removed
maxulysse Nov 4, 2022
8af8cc4
comments
maxulysse Nov 4, 2022
93f4db2
more conda md5sum removed + remove duplicate tests
maxulysse Nov 4, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 5 additions & 91 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,12 @@ name: nf-core CI
# This workflow runs the pipeline with the minimal test dataset to check that it completes without any syntax errors
on:
push:
branches:
- dev
branches: [dev]
pull_request:
release:
types: [published]

env:
NXF_ANSI_LOG: false

# Cancel if a newer run is started
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
Expand All @@ -34,9 +31,11 @@ jobs:
exclude:
- NXF_VER: "latest-everything"
profile: "conda"
env:
NXF_ANSI_LOG: false
steps:
- name: Check out pipeline code
uses: actions/checkout@v2
uses: actions/checkout@v3

- name: Set up Python
uses: actions/setup-python@v2
Expand Down Expand Up @@ -107,88 +106,3 @@ jobs:
/home/runner/pytest_workflow_*/*/work
!/home/runner/pytest_workflow_*/*/work/conda
!/home/runner/pytest_workflow_*/*/work/singularity

test_all:
name: Run pipeline with test data (complete)
# Only run on push if this is the nf-core dev branch (merged PRs)
if: "${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'nf-core/sarek') }}"
runs-on: ubuntu-latest
strategy:
# HACK Remove after DSL2 rewrite is done
fail-fast: false
matrix:
NXF_VER:
- "21.10.3"
test:
- "aligner"
- "alignment_to_fastq"
- "annotation"
- "cnvkit"
- "controlfreec"
- "deepvariant"
- "freebayes"
- "gatk4_spark"
- "haplotypecaller"
- "intervals"
- "manta"
- "markduplicates"
- "mpileup"
- "msisensorpro"
- "mutect2"
- "prepare_recalibration"
- "recalibrate"
- "save_mapped"
- "save_output_as_bam"
- "skip_markduplicates"
- "skip_qc"
- "split_fastq"
- "strelka"
- "strelkabp"
- "targeted"
- "tiddit"
- "trimming"
- "tumor_normal_pair"
- "umi"
- "variantcalling_channel"
profile: ["docker"]
steps:
- name: Check out pipeline code
uses: actions/checkout@v2

- name: Install Nextflow
uses: nf-core/setup-nextflow@v1
with:
version: "${{ matrix.NXF_VER }}"

- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: "3.x"

- name: Install dependencies
run: python -m pip install --upgrade pip pytest-workflow

- name: Run pipeline with tests settings
uses: Wandalen/wretry.action@v1.0.11
with:
command: TMPDIR=~ PROFILE=${{ matrix.profile }} pytest --tag ${{ matrix.test }} --symlink --kwdof --git-aware --color=yes
attempt_limit: 3

- name: Output log on failure
if: failure()
run: |
sudo apt install bat > /dev/null
batcat --decorations=always --color=always /home/runner/pytest_workflow_*/*/log.{out,err}

- name: Upload logs on failure
if: failure()
uses: actions/upload-artifact@v2
with:
name: logs-${{ matrix.profile }}
path: |
/home/runner/pytest_workflow_*/*/.nextflow.log
/home/runner/pytest_workflow_*/*/log.out
/home/runner/pytest_workflow_*/*/log.err
/home/runner/pytest_workflow_*/*/work
!/home/runner/pytest_workflow_*/*/work/conda
!/home/runner/pytest_workflow_*/*/work/singularity
126 changes: 126 additions & 0 deletions .github/workflows/pytest-workflow.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
name: pytest-workflow
# This workflow runs the pipeline with the minimal test dataset to check that it completes without any syntax errors
on:
pull_request:
branches: [dev]

# Cancel if a newer run is started
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

jobs:
changes:
name: Check for changes
runs-on: ubuntu-latest
outputs:
# Expose matched filters as job 'tags' output variable
tags: ${{ steps.filter.outputs.changes }}
steps:
- uses: actions/checkout@v3
- uses: dorny/paths-filter@v2
id: filter
with:
filters: "tests/config/pytest_tags.yml"

test:
name: ${{ matrix.tags }} ${{ matrix.profile }}
runs-on: ubuntu-latest
needs: changes
if: needs.changes.outputs.tags != '[]'
strategy:
fail-fast: false
matrix:
tags: ["${{ fromJson(needs.changes.outputs.tags) }}"]
profile: ["docker", "singularity", "conda"]
NXF_VER:
- "21.10.3"
- "latest-everything"
exclude:
- profile: "conda"
NXF_VER: "latest-everything"
- profile: "conda"
tags: deepvariant
- profile: "conda"
tags: haplotypecaller
- profile: "conda"
tags: merge
- profile: "conda"
tags: snpeff
- profile: "conda"
tags: vep
maxulysse marked this conversation as resolved.
Show resolved Hide resolved
env:
NXF_ANSI_LOG: false
steps:
- name: Check out pipeline code
uses: actions/checkout@v3

- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: "3.x"

- uses: actions/cache@v2
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-

- name: Install Python dependencies
run: python -m pip install --upgrade pip pytest-workflow

- uses: actions/cache@v2
with:
path: /usr/local/bin/nextflow
key: ${{ runner.os }}
restore-keys: |
${{ runner.os }}-nextflow-

- name: Install Nextflow
uses: nf-core/setup-nextflow@v1
with:
version: "${{ matrix.NXF_VER }}"

- name: Set up Singularity
if: matrix.profile == 'singularity'
uses: eWaterCycle/setup-singularity@v5
with:
singularity-version: 3.7.1

- name: Set up miniconda
if: matrix.profile == 'conda'
uses: conda-incubator/setup-miniconda@v2
with:
auto-update-conda: true
channels: conda-forge,bioconda,defaults
python-version: ${{ matrix.python-version }}

- name: Conda clean
if: matrix.profile == 'conda'
run: conda clean -a

- name: Run pytest-workflow
uses: Wandalen/wretry.action@v1.0.11
with:
command: TMPDIR=~ PROFILE=${{ matrix.profile }} pytest --tag ${{ matrix.tags }} --symlink --kwdof --git-aware --color=yes
attempt_limit: 3

- name: Output log on failure
if: failure()
run: |
sudo apt install bat > /dev/null
batcat --decorations=always --color=always /home/runner/pytest_workflow_*/*/log.{out,err}

- name: Upload logs on failure
if: failure()
uses: actions/upload-artifact@v2
with:
name: logs-${{ matrix.profile }}
path: |
/home/runner/pytest_workflow_*/*/.nextflow.log
/home/runner/pytest_workflow_*/*/log.out
/home/runner/pytest_workflow_*/*/log.err
/home/runner/pytest_workflow_*/*/work
!/home/runner/pytest_workflow_*/*/work/conda
!/home/runner/pytest_workflow_*/*/work/singularity
4 changes: 4 additions & 0 deletions .nf-core.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
repository_type: pipeline
lint:
files_exist:
- conf/modules.config
- conf/test.config
- conf/test_full.config
files_unchanged:
- assets/multiqc_config.yml
- assets/nf-core-sarek_logo_light.png
Expand Down
Loading