Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enable pytest with conda and singularity #691

Merged
merged 6 commits into from
Jul 26, 2022
Merged

enable pytest with conda and singularity #691

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7e835d1
enable pytest with conda and singularity
maxulysse Jul 26, 2022
74e3a0e
default CI is run with docker
maxulysse Jul 26, 2022
bb09801
update CHANGELOG
maxulysse Jul 26, 2022
0a47ac1
add paths to check for modification cc @jfy133
maxulysse Jul 26, 2022
5310f32
prettier
maxulysse Jul 26, 2022
2fb8c01
will play with paths in a different PR
maxulysse Jul 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ jobs:
- "tumor_normal_pair"
- "umi"
- "variantcalling_channel"

profile: ["docker"]
steps:
- name: Check out pipeline code
uses: actions/checkout@v2
Expand All @@ -88,7 +88,7 @@ jobs:
- name: Run pipeline with tests settings
uses: Wandalen/wretry.action@v1.0.11
with:
command: pytest --tag ${{ matrix.test }} --symlink --kwdof --git-aware --color=yes
command: PROFILE=${{ matrix.profile }} pytest --tag ${{ matrix.test }} --symlink --kwdof --git-aware --color=yes
attempt_limit: 3

- name: Output log on failure
Expand All @@ -107,6 +107,6 @@ jobs:
/tmp/pytest_workflow_*/*/log.out
/tmp/pytest_workflow_*/*/log.err
/tmp/pytest_workflow_*/*/work
/tmp/pytest_workflow_*/**/.command.log
!/tmp/pytest_workflow_*/*/work/conda
!/tmp/pytest_workflow_*/*/work/singularity
/tmp/pytest_workflow_*/**/.command.log
3 changes: 3 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- [#691](https://github.com/nf-core/sarek/pull/691) - Enable `PROFILE=conda`, `PROFILE=docker` and `PROFILE=singularity` for pytest

### Changed

- [#679](https://github.com/nf-core/sarek/pull/679) - Back to `dev`
- [#691](https://github.com/nf-core/sarek/pull/691) - To run the same pytest as before locally, use `PROFILE=docker`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- [#691](https://github.com/nf-core/sarek/pull/691) - To run the same pytest as before locally, use `PROFILE=docker`
- [#691](https://github.com/nf-core/sarek/pull/691) - To run the same pytest as before locally, use TMPDIR=~/tmp, and `PROFILE=conda`, `PROFILE=docker` or `PROFILE=singularity`.

or something like that.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before we had just docker, all the rest is new, let's not document that too much for now, as we're still figuring it out ;-)


### Fixed

Expand Down
27 changes: 27 additions & 0 deletions conf/test.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,3 +233,30 @@ process {
ext.args = { "--info-key CNN_1D --indel-tranche 0" }
}
}

// ENABLE CI containers for testing
if (System.getenv('PROFILE')) {
if ("$PROFILE" == "conda") {
params.enable_conda = true
docker.enabled = false
singularity.enabled = false
podman.enabled = false
shifter.enabled = false
charliecloud.enabled = false
} else if ("$PROFILE" == "docker") {
docker.enabled = true
docker.userEmulation = { params.use_gatk_spark ? false : true }.call()
singularity.enabled = false
podman.enabled = false
shifter.enabled = false
charliecloud.enabled = false
fixOwnership = true
} else if ("$PROFILE" == "singularity") {
singularity.enabled = true
singularity.autoMounts = true
docker.enabled = false
podman.enabled = false
shifter.enabled = false
charliecloud.enabled = false
}
}
2 changes: 1 addition & 1 deletion tests/test_aligner.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
- name: Run bwa-mem2
command: nextflow run main.nf -profile test,docker --aligner bwa-mem2 --save_reference
command: nextflow run main.nf -profile test --aligner bwa-mem2 --save_reference
tags:
- aligner
- bwa-mem2
Expand Down
10 changes: 5 additions & 5 deletions tests/test_annotation.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
- name: Run snpEff
command: nextflow run main.nf -profile test,annotation,docker --tools snpeff
command: nextflow run main.nf -profile test,annotation --tools snpeff
tags:
- annotation
- snpeff
Expand All @@ -11,7 +11,7 @@
- path: results/reports/snpeff/test/test_snpEff.csv
- path: results/reports/snpeff/test/test_snpEff.genes.txt
- name: Run VEP
command: nextflow run main.nf -profile test,annotation,docker --tools vep --skip_tools multiqc
command: nextflow run main.nf -profile test,annotation --tools vep --skip_tools multiqc
tags:
- annotation
- vep
Expand All @@ -20,7 +20,7 @@
- path: results/annotation/test/test_VEP.ann.vcf.gz.tbi
- path: results/reports/EnsemblVEP/test/test_VEP.summary.html
- name: Run snpEff followed by VEP
command: nextflow run main.nf -profile test,annotation,docker --tools merge --skip_tools multiqc
command: nextflow run main.nf -profile test,annotation --tools merge --skip_tools multiqc
tags:
- annotation
- merge
Expand All @@ -47,7 +47,7 @@
- path: results/reports/EnsemblVEP/test/test_VEP.summary.html
should_exist: false
- name: Run snpEff, VEP and both consecutively
command: nextflow run main.nf -profile test,annotation,docker --tools merge,snpeff,vep --skip_tools multiqc
command: nextflow run main.nf -profile test,annotation --tools merge,snpeff,vep --skip_tools multiqc
tags:
- annotation
- merge
Expand All @@ -66,7 +66,7 @@
- path: results/reports/snpeff/test/test_snpEff.csv
- path: results/reports/snpeff/test/test_snpEff.genes.txt
- name: Run VEP with fasta
command: nextflow run main.nf -profile test,annotation,docker --tools vep --vep_include_fasta --skip_tools multiqc
command: nextflow run main.nf -profile test,annotation --tools vep --vep_include_fasta --skip_tools multiqc
tags:
- annotation
- vep
Expand Down
2 changes: 1 addition & 1 deletion tests/test_bam_remap.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
- name: Run alignment to fastq and then remap on bam files
command: nextflow run main.nf -profile test,alignment_to_fastq,docker
command: nextflow run main.nf -profile test,alignment_to_fastq
tags:
- alignment_to_fastq
files:
Expand Down
6 changes: 3 additions & 3 deletions tests/test_default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
- name: Run default pipeline
command: nextflow run main.nf -profile test,docker
command: nextflow run main.nf -profile test
tags:
- default
- preprocessing
Expand Down Expand Up @@ -28,7 +28,7 @@
- path: results/reports/samtools/test/test.md.cram.stats
- path: results/reports/samtools/test/test.recal.cram.stats
- name: Run default pipeline without intervals
command: nextflow run main.nf -profile test,no_intervals,docker
command: nextflow run main.nf -profile test,no_intervals
tags:
- default
- no_intervals
Expand Down Expand Up @@ -61,7 +61,7 @@
- path: results/reports/samtools/test/test.md.cram.stats
- path: results/reports/samtools/test/test.recal.cram.stats
- name: Run default pipeline with VC
command: nextflow run main.nf -profile test,docker --tools strelka
command: nextflow run main.nf -profile test --tools strelka
tags:
- default
- preprocessing
Expand Down
2 changes: 1 addition & 1 deletion tests/test_gatk_spark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
- name: Run default pipeline with gatk_spark
command: nextflow run main.nf -profile test,use_gatk_spark,docker
command: nextflow run main.nf -profile test,use_gatk_spark
tags:
- gatk4
- gatk4_spark
Expand Down
4 changes: 2 additions & 2 deletions tests/test_markduplicates.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
- name: Run markduplicates starting from BAM
command: nextflow run main.nf -profile test,markduplicates_bam,docker
command: nextflow run main.nf -profile test,markduplicates_bam
tags:
- bam
- markduplicates
Expand Down Expand Up @@ -28,7 +28,7 @@
- path: results/reports/samtools/test/test.md.cram.stats
- path: results/reports/samtools/test/test.recal.cram.stats
- name: Run markduplicates starting from CRAM
command: nextflow run main.nf -profile test,markduplicates_cram,docker
command: nextflow run main.nf -profile test,markduplicates_cram
tags:
- cram
- markduplicates
Expand Down
4 changes: 2 additions & 2 deletions tests/test_only_paired_VC.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
- name: Run variant calling on somatic samples with cnvkit and skip variant calling on matched normal
command: nextflow run main.nf -profile test,tools_somatic,docker --tools cnvkit --only_paired_variant_calling
command: nextflow run main.nf -profile test,tools_somatic --tools cnvkit --only_paired_variant_calling
tags:
- cnvkit
- somatic
Expand All @@ -21,7 +21,7 @@
- path: results/variant_calling/cnvkit/sample4_vs_sample3/test2.paired_end.recalibrated.sorted.cns
- path: results/variant_calling/cnvkit/sample4_vs_sample3/test2.paired_end.recalibrated.sorted.targetcoverage.cnn
- name: Skip variant calling on matched normal
command: nextflow run main.nf -profile test,variantcalling_channels,docker --tools strelka --only_paired_variant_calling
command: nextflow run main.nf -profile test,variantcalling_channels --tools strelka --only_paired_variant_calling
tags:
- somatic
- strelka
Expand Down
2 changes: 1 addition & 1 deletion tests/test_pair.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
- name: Run default pipeline for tumor normal pair
command: nextflow run main.nf -profile test,pair,docker
command: nextflow run main.nf -profile test,pair
tags:
- preprocessing
- tumor_normal_pair
Expand Down
8 changes: 4 additions & 4 deletions tests/test_prepare_recalibration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
- name: Run prepare_recalibration starting from bam
command: nextflow run main.nf -profile test,prepare_recalibration_bam,docker
command: nextflow run main.nf -profile test,prepare_recalibration_bam
tags:
- bam
- prepare_recalibration
Expand All @@ -20,7 +20,7 @@
- path: results/reports/mosdepth/test/test.recal.regions.bed.gz.csi
- path: results/reports/samtools/test/test.recal.cram.stats
- name: Run prepare_recalibration starting from cram
command: nextflow run main.nf -profile test,prepare_recalibration_cram,docker
command: nextflow run main.nf -profile test,prepare_recalibration_cram
tags:
- cram
- prepare_recalibration
Expand All @@ -38,7 +38,7 @@
- path: results/reports/mosdepth/test/test.recal.regions.bed.gz.csi
- path: results/reports/samtools/test/test.recal.cram.stats
- name: Run prepare_recalibration starting from bam and skip baserecalibration
command: nextflow run main.nf -profile test,prepare_recalibration_bam,skip_bqsr,docker --tools strelka
command: nextflow run main.nf -profile test,prepare_recalibration_bam,skip_bqsr --tools strelka
tags:
- bam
- prepare_recalibration
Expand Down Expand Up @@ -69,7 +69,7 @@
- path: results/reports/samtools_stats
should_exist: false
- name: Run prepare_recalibration starting from cram and skip baserecalibration
command: nextflow run main.nf -profile test,prepare_recalibration_cram,skip_bqsr,docker --tools strelka
command: nextflow run main.nf -profile test,prepare_recalibration_cram,skip_bqsr --tools strelka
tags:
- cram
- prepare_recalibration
Expand Down
8 changes: 4 additions & 4 deletions tests/test_recalibrate.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
- name: Run Recalibration starting from bam
command: nextflow run main.nf -profile test,recalibrate_bam,docker
command: nextflow run main.nf -profile test,recalibrate_bam
tags:
- bam
- recalibrate
Expand All @@ -18,7 +18,7 @@
- path: results/reports/mosdepth/test/test.recal.regions.bed.gz.csi
- path: results/reports/samtools/test/test.recal.cram.stats
- name: Run Recalibration starting from cram
command: nextflow run main.nf -profile test,recalibrate_cram,docker
command: nextflow run main.nf -profile test,recalibrate_cram
tags:
- cram
- recalibrate
Expand All @@ -35,7 +35,7 @@
- path: results/reports/mosdepth/test/test.recal.regions.bed.gz.csi
- path: results/reports/samtools/test/test.recal.cram.stats
- name: Run Recalibration starting from bam and skip baserecalibration
command: nextflow run main.nf -profile test,recalibrate_bam,skip_bqsr,docker --tools strelka
command: nextflow run main.nf -profile test,recalibrate_bam,skip_bqsr --tools strelka
tags:
- bam
- recalibrate
Expand Down Expand Up @@ -74,7 +74,7 @@
- path: results/reports/samtools/test/test.recal.cram.stats
should_exist: false
- name: Run Recalibration starting from cram and skip baserecalibration
command: nextflow run main.nf -profile test,recalibrate_cram,skip_bqsr,docker --tools strelka
command: nextflow run main.nf -profile test,recalibrate_cram,skip_bqsr --tools strelka
tags:
- cram
- recalibrate
Expand Down
2 changes: 1 addition & 1 deletion tests/test_save_bam_mapped.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
- name: Run save_bam_mapped
command: nextflow run main.nf -profile test,docker --save_bam_mapped
command: nextflow run main.nf -profile test --save_bam_mapped
tags:
- preprocessing
- save_bam_mapped
Expand Down
4 changes: 2 additions & 2 deletions tests/test_save_output_bam.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
- name: Run save_output_as_bam
command: nextflow run main.nf -profile test,docker --save_output_as_bam
command: nextflow run main.nf -profile test --save_output_as_bam
tags:
- preprocessing
- save_output_as_bam
Expand Down Expand Up @@ -28,7 +28,7 @@
- path: results/reports/samtools/test/test.md.cram.stats
- path: results/reports/samtools/test/test.recal.cram.stats
- name: Run save_output_as_bam with gatk4 spark
command: nextflow run main.nf -profile test,use_gatk_spark,docker --save_output_as_bam
command: nextflow run main.nf -profile test,use_gatk_spark --save_output_as_bam
tags:
- preprocessing
- save_output_as_bam
Expand Down
6 changes: 3 additions & 3 deletions tests/test_skip_markduplicates.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
- name: Run default pipeline with skipping MarkDuplicates
command: nextflow run main.nf -profile test,docker,skip_markduplicates
command: nextflow run main.nf -profile test,skip_markduplicates
tags:
- preprocessing
- skip_markduplicates
Expand Down Expand Up @@ -33,7 +33,7 @@
- path: results/csv/markduplicates_no_table.csv
should_exist: false
- name: Run skip markduplicates bam from step markduplicates
command: nextflow run main.nf -profile test,docker,markduplicates_bam,skip_markduplicates
command: nextflow run main.nf -profile test,markduplicates_bam,skip_markduplicates
tags:
- bam
- preprocessing
Expand Down Expand Up @@ -64,7 +64,7 @@
- path: results/csv/markduplicates_no_table.csv
should_exist: false
- name: Run skip markduplicates cram from step markduplicates
command: nextflow run main.nf -profile test,docker,markduplicates_cram,skip_markduplicates
command: nextflow run main.nf -profile test,markduplicates_cram,skip_markduplicates
tags:
- cram
- preprocessing
Expand Down
4 changes: 2 additions & 2 deletions tests/test_skip_qc.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
- name: Run default pipeline with skipping all QC steps
command: nextflow run main.nf -profile test,docker --skip_tools fastqc,markduplicates_report,mosdepth,multiqc,samtools
command: nextflow run main.nf -profile test --skip_tools fastqc,markduplicates_report,mosdepth,multiqc,samtools
tags:
- skip_qc
- preprocessing
Expand All @@ -24,7 +24,7 @@
should_exist: false

- name: Run spark pipeline with skipping all QC steps
command: nextflow run main.nf -profile test,use_gatk_spark,docker --skip_tools fastqc,markduplicates_report,mosdepth,multiqc,samtools
command: nextflow run main.nf -profile test,use_gatk_spark --skip_tools fastqc,markduplicates_report,mosdepth,multiqc,samtools
tags:
- skip_qc
- preprocessing
Expand Down
2 changes: 1 addition & 1 deletion tests/test_split_fastq.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
- name: Run split fastq module
command: nextflow run main.nf -profile test,split_fastq,docker
command: nextflow run main.nf -profile test,split_fastq
tags:
- split_fastq
files:
Expand Down
4 changes: 2 additions & 2 deletions tests/test_targeted.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
- name: Run default pipeline with target bed
command: nextflow run main.nf -profile test,targeted,docker
command: nextflow run main.nf -profile test,targeted
tags:
- preprocessing
- targeted
Expand Down Expand Up @@ -33,7 +33,7 @@
- path: results/reports/samtools/test/test.recal.cram.stats

- name: Run intervals false pipeline
command: nextflow run main.nf -profile test,docker --intervals false
command: nextflow run main.nf -profile test --intervals false
tags:
- default
- preprocessing
Expand Down
Loading