More improvments (#18)

* Improve cpu usage * Use same font for nf-core and sarek in ascii art * Improve syntax * Add --no-reports option for tests + add snpEff,VEP,merge to MULTIPLE test * Use --no-reports for TravisCI testing * Add possibility to download other genome for sareksnpeff and sarekvep containers * Update docs * Add --no-reports for all tests but MULTIPLE * Use sensible label selectors for configs * Enhance the old --noReports with --skip to skip QC on demand * Replace --noReports by --skip all * Update logos with new nf-core/sarek logo * Add logo to MultiQC report * Remove createMultiQCconfig() * Improve docs about offline usage with bianca * Code polishing * Update CHANGELOG
nf-core · Jul 5, 2019 · 0d66968 · 0d66968
1 parent 10eff4e
commit 0d66968
Show file tree

Hide file tree

Showing 60 changed files with 6,543 additions and 4,010 deletions.
diff --git a/.travis.yml b/.travis.yml
@@ -43,4 +43,4 @@ before_script:
 
 # Actual tests
 script:
-  - "${TRAVIS_BUILD_DIR}/scripts/run_tests.sh --test $TEST --verbose"
+  - "${TRAVIS_BUILD_DIR}/scripts/run_tests.sh --test $TEST --verbose --no-reports"
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -6,9 +6,11 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
 ## Unreleased
+
 Initial release of `nf-core/sarek`, created with the [nf-core](http://nf-co.re/) template.
 
 ### `Added`
+
 -   [#2](https://github.com/nf-core/sarek/pull/2) - Create `nf-core/sarek` `environment.yml` file
 -   [#2](https://github.com/nf-core/sarek/pull/2), [#3](https://github.com/nf-core/sarek/pull/3), [#4](https://github.com/nf-core/sarek/pull/4), [#5](https://github.com/nf-core/sarek/pull/5), [#7](https://github.com/nf-core/sarek/pull/7), [#9](https://github.com/nf-core/sarek/pull/9), [#11](https://github.com/nf-core/sarek/pull/11), [#12](https://github.com/nf-core/sarek/pull/12) - Add CI for `nf-core/sarek`
 -   [#3](https://github.com/nf-core/sarek/pull/3) - Add preprocessing to `nf-core/sarek`
@@ -32,9 +34,14 @@ Initial release of `nf-core/sarek`, created with the [nf-core](http://nf-co.re/)
 -   [#13](https://github.com/nf-core/sarek/pull/13) - Add Citation documentation
 -   [#13](https://github.com/nf-core/sarek/pull/13) - Add `BamQC` process
 -   [#13](https://github.com/nf-core/sarek/pull/13) - Add `CompressVCFsnpEff` and `CompressVCFvep` processes
+-   [#18](https://github.com/nf-core/sarek/pull/18) - Add --no-reports option for tests + add snpEff,VEP,merge to MULTIPLE test
+-   [#18](https://github.com/nf-core/sarek/pull/18) - Add possibility to download other genome for sareksnpeff and sarekvep containers
+-   [#18](https://github.com/nf-core/sarek/pull/18) - Add params `--skip` to skip specified QC tools
+-   [#18](https://github.com/nf-core/sarek/pull/18) - Add logo to MultiQC report
 
 ### `Changed`
--   [#1](https://github.com/nf-core/sarek/pull/1), [#2](https://github.com/nf-core/sarek/pull/2), [#3](https://github.com/nf-core/sarek/pull/3), [#4](https://github.com/nf-core/sarek/pull/4), [#5](https://github.com/nf-core/sarek/pull/5), [#6](https://github.com/nf-core/sarek/pull/6), [#7](https://github.com/nf-core/sarek/pull/7), [#8](https://github.com/nf-core/sarek/pull/8), [#9](https://github.com/nf-core/sarek/pull/9), [#11](https://github.com/nf-core/sarek/pull/11), [#12](https://github.com/nf-core/sarek/pull/12) - Update docs
+
+-   [#1](https://github.com/nf-core/sarek/pull/1), [#2](https://github.com/nf-core/sarek/pull/2), [#3](https://github.com/nf-core/sarek/pull/3), [#4](https://github.com/nf-core/sarek/pull/4), [#5](https://github.com/nf-core/sarek/pull/5), [#6](https://github.com/nf-core/sarek/pull/6), [#7](https://github.com/nf-core/sarek/pull/7), [#8](https://github.com/nf-core/sarek/pull/8), [#9](https://github.com/nf-core/sarek/pull/9), [#11](https://github.com/nf-core/sarek/pull/11), [#12](https://github.com/nf-core/sarek/pull/12), [#18](https://github.com/nf-core/sarek/pull/18) - Update docs
 -   [#4](https://github.com/nf-core/sarek/pull/4) - Update `cancerit-allelecount` from `2.1.2` to `4.0.2`
 -   [#4](https://github.com/nf-core/sarek/pull/4) - Update `gatk4` from `4.1.1.0` to `4.1.2.0`
 -   [#7](https://github.com/nf-core/sarek/pull/7) - `--sampleDir` is now deprecated, use `--sample` instead
@@ -53,28 +60,38 @@ Initial release of `nf-core/sarek`, created with the [nf-core](http://nf-co.re/)
 -   [#13](https://github.com/nf-core/sarek/pull/13) - Merge `BamQCmapped` and `BamQCrecalibrated` processes into `BamQC` process
 -   [#13](https://github.com/nf-core/sarek/pull/13) - Split `CompressVCF` process into `CompressVCFsnpEff` and `CompressVCFvep` processes
 -   [#16](https://github.com/nf-core/sarek/pull/16) - Make scripts in `bin/` and `scripts/` executable
+-   [#18](https://github.com/nf-core/sarek/pull/18) - Use --no-reports for TravisCI testing
+-   [#18](https://github.com/nf-core/sarek/pull/18) - Add --no-reports for all tests but MULTIPLE in Jenkins
+-   [#18](https://github.com/nf-core/sarek/pull/18) - `--noReports` is now `--skip all`
+-   [#18](https://github.com/nf-core/sarek/pull/18) - Update logo
 
 ### `Removed`
 
 -   [#9](https://github.com/nf-core/sarek/pull/9) - Removed `relatedness2` graph from `vcftools stats`
 -   [#13](https://github.com/nf-core/sarek/pull/13) - Removed `BamQCmapped` and `BamQCrecalibrated` processes
 -   [#13](https://github.com/nf-core/sarek/pull/13) - Removed `CompressVCF`
+-   [#18](https://github.com/nf-core/sarek/pull/18) - Removed params `--noReports`
 
 ### `Fixed`
+
 -   [#3](https://github.com/nf-core/sarek/pull/3) - Fix Docker ownership
 -   [#11](https://github.com/nf-core/sarek/pull/11) - Fix MergeMpileup PublishDir
 -   [#13](https://github.com/nf-core/sarek/pull/13) - Fix merge in annotation
 -   [#14](https://github.com/nf-core/sarek/pull/14) - Fix output name for vcf files
 -   [#16](https://github.com/nf-core/sarek/pull/16) - Fix path to Rscript
+-   [#18](https://github.com/nf-core/sarek/pull/18) - Improve cpu usage
+-   [#18](https://github.com/nf-core/sarek/pull/18) - Use same font for nf-core and sarek in ascii art
 
 ## [2.3.FIX1] - 2019-03-04
 
 ### `Fixed`
+
 -   [#742](https://github.com/SciLifeLab/Sarek/pull/742) - Fix output dirs (HaplotypeCaller that was not recognized by annotate.nf introduced by [#728](https://github.com/SciLifeLab/Sarek/pull/728))
 
 ## [2.3] - Äpar - 2019-02-27
 
 ### `Added`
+
 -   [#628](https://github.com/SciLifeLab/Sarek/pull/628), [#722](https://github.com/SciLifeLab/Sarek/pull/722) - `ASCAT` now use `.gc` file
 -   [#712](https://github.com/SciLifeLab/Sarek/pull/712), [#718](https://github.com/SciLifeLab/Sarek/pull/718) - Added possibilities to run Sarek with `conda`
 -   [#719](https://github.com/SciLifeLab/Sarek/pull/719) - Annotation documentation
@@ -116,13 +133,15 @@ Initial release of `nf-core/sarek`, created with the [nf-core](http://nf-co.re/)
 -   [#732](https://github.com/SciLifeLab/Sarek/pull/732) - Update VEP from `95.1` to `95.2`
 
 ### `Removed`
+
 -   [#715](https://github.com/SciLifeLab/Sarek/pull/715) - Remove `defReferencesFiles` function from `buildReferences.nf`
 -   [#719](https://github.com/SciLifeLab/Sarek/pull/719) - `snpEff` base container is no longer used
 -   [#721](https://github.com/SciLifeLab/Sarek/pull/721) - Remove COSMIC docs
 -   [#728](https://github.com/SciLifeLab/Sarek/pull/728) - Remove `defineDirectoryMap()`
 -   [#732](https://github.com/SciLifeLab/Sarek/pull/732) - Removed `--database` option for VEP cf: [VEP docs](https://www.ensembl.org/info/docs/tools/vep/script/vep_other.html)
 
 ### `Fixed`
+
 -   [#720](https://github.com/SciLifeLab/Sarek/pull/720) - bamQC is now run on the recalibrated bams, and not after MarkDuplicates
 -   [#726](https://github.com/SciLifeLab/Sarek/pull/726) - Fix Ascat ref file input (one file can't be a set)
 -   [#727](https://github.com/SciLifeLab/Sarek/pull/727) - bamQC outputs are no longer overwritten (name of dir is now the file instead of sample)

diff --git a/Jenkinsfile b/Jenkinsfile
@@ -8,7 +8,7 @@ pipeline {
     stages {
         stage('Docker setup') {
             steps {
-                sh "./scripts/download_image.sh -n docker -t ALL"
+                 sh "./scripts/download_image.sh -n docker -t ALL -T dev -g smallGRCh37"
             }
         }
         stage('Build references') {
@@ -21,23 +21,23 @@ pipeline {
             steps {
                 sh "rm -rf data/"
                 sh "git clone --single-branch --branch sarek https://github.com/nf-core/test-datasets.git data"
-                sh "./scripts/run_tests.sh --test GERMLINE"
+                sh "./scripts/run_tests.sh --test GERMLINE --no-reports"
                 sh "rm -rf data/"
             }
         }
         stage('Somatic') {
             steps {
-                sh "./scripts/run_tests.sh --test SOMATIC"
+                sh "./scripts/run_tests.sh --test SOMATIC --no-reports"
             }
         }
         stage('Targeted') {
             steps {
-                sh "./scripts/run_tests.sh --test TARGETED"
+                sh "./scripts/run_tests.sh --test TARGETED --no-reports"
             }
         }
         stage('Annotation') {
             steps {
-                sh "./scripts/run_tests.sh --test ANNOTATEALL"
+                sh "./scripts/run_tests.sh --test ANNOTATEBOTH --no-reports"
             }
         }
         stage('Multiple') {

diff --git a/assets/multiqc_config.yaml b/assets/multiqc_config.yaml
@@ -1,9 +1,22 @@
+custom_logo: ../../../docs/images/nf-core_sarek_logo.png
+custom_logo_url: https://github.com/nf-core/sarek/
+custom_logo_title: 'nf-core/sarek'
+
 report_comment: >
     This report has been generated by the <a href="https://github.com/nf-core/sarek" target="_blank">nf-core/sarek</a>
     analysis pipeline. For information about how to interpret these results, please see the
-    <a href="https://github.com/nf-core/sarek" target="_blank">documentation</a>.
+    <a href="https://github.com/nf-core/sarek/blob/master/docs/output.md" target="_blank">documentation</a>.
 report_section_order:
     nf-core/sarek-software-versions:
         order: -1000
 
 export_plots: true
+
+top_modules:
+- 'fastqc'
+- 'picard'
+- 'samtools'
+- 'qualimap'
+- 'bcftools'
+- 'vcftools'
+- 'snpeff'
diff --git a/build.nf b/build.nf
@@ -387,12 +387,12 @@ def nfcoreHeader(){
     ${c_blue}  |\\ | |__  __ /  ` /  \\ |__) |__         ${c_yellow}}  {${c_reset}
     ${c_blue}  | \\| |       \\__, \\__/ |  \\ |___     ${c_green}\\`-._,-`-,${c_reset}
                                             ${c_green}`._,._,\'${c_reset}
-    ${c_white}       ____      ${c_blue}  _____               _ ${c_reset}
-    ${c_white}     .' _  `.    ${c_blue} / ____|             | | ${c_reset}
-    ${c_white}    /  ${c_green}|\\${c_white}`-_${c_white} \\ ${c_blue}  | (___  ___  _ __ __ | | __ ${c_reset}
-    ${c_white}   |   ${c_green}| \\  ${c_white}`-${c_white}| ${c_blue}  \\___ \\/__ \\| ´__/ _\\| |/ / ${c_reset}
-    ${c_white}    \\ ${c_green}|   \\  ${c_white}/ ${c_blue}   ____) | __ | | |  __|   < ${c_reset}
-    ${c_white}     `${c_green}|${c_white}____${c_green}\\${c_white}'   ${c_blue} |_____/\\____|_|  \\__/|_|\\_\\ ${c_reset}
+        ${c_white}____${c_reset}
+      ${c_white}.´ _  `.${c_reset}
+     ${c_white}/  ${c_green}|\\${c_reset}`-_ \\${c_reset}     ${c_blue} __        __   ___     ${c_reset}
+    ${c_white}|   ${c_green}| \\${c_reset}  `-|${c_reset}    ${c_blue}|__`  /\\  |__) |__  |__/${c_reset}
+     ${c_white}\\ ${c_green}|   \\${c_reset}  /${c_reset}     ${c_blue}.__| /¯¯\\ |  \\ |___ |  \\${c_reset}
+      ${c_white}`${c_green}|${c_reset}____${c_green}\\${c_reset}´${c_reset}
 
     ${c_purple}  nf-core/sarek v${workflow.manifest.version}${c_reset}
     ${c_dim}----------------------------------------------------${c_reset}

diff --git a/conf/base.config b/conf/base.config
@@ -3,15 +3,15 @@
  *  nf-core/sarek Nextflow base config file
  * -------------------------------------------------
  * A 'blank slate' config file, appropriate for general
- * use on most high performace compute environments.
+ * use on most high performance compute environments.
  * Assumes that all software is installed and available
  * on the PATH. Runs in `local` mode - all jobs will be
  * run on the logged in environment.
  */
 
 params {
   // Defaults only, expecting to be overwritten
-  cpus = 10
+  cpus = 8
   igenomes_base = 's3://ngi-igenomes/igenomes/'
   markdup_java_options = '"-Xms4000m -Xmx7g"' //Established values for markDuplicate memory consumption, see issue PR #689 for details
   max_cpus = 16 // Base specifications
@@ -21,79 +21,68 @@ params {
 }
 
 process {
-  cpus = {check_max(params.cpus * task.attempt)}
-  memory = {check_max(15.GB * task.attempt)}
-  time = {check_max(24.h * task.attempt)}
+  cpus = {check_resource(params.cpus * task.attempt)}
+  memory = {check_resource(params.singleCPUMem * task.attempt)}
+  time = {check_resource(24.h * task.attempt)}
   shell = ['/bin/bash', '-euo', 'pipefail']
 
   errorStrategy = {task.exitStatus in [143,137,104,134,139] ? 'retry' : 'finish'}
   maxErrors = '-1'
   maxRetries = 3
 
-  withLabel:singleCPUmem_x_task {
-    memory = {check_max(params.singleCPUMem * task.attempt)}
+  withLabel:cpus_1 {
+    cpus = {check_resource(1)}
   }
-  withLabel:singleCPUmem_x2_task {
-    memory = {check_max(params.singleCPUMem * task.attempt * task.attempt)}
-    }
-  withLabel:singleCPUmem_2x_task {
-    memory = {check_max(params.singleCPUMem * 2 * task.attempt)}
+  withLabel:cpus_2 {
+    cpus = {check_resource(2)}
   }
-  withLabel:max_cpus {
+  withLabel:cpus_4 {
+    cpus = {check_resource(4)}
+  }
+  withLabel:cpus_8 {
+    cpus = {check_resource(8)}
+  }
+  withLabel:cpus_16 {
+    cpus = {check_resource(16)}
+  }
+  withLabel:cpus_max {
     cpus = {params.max_cpus}
   }
-  withLabel:max_memory {
-    memory = {check_max(params.max_memory)}
+
+  withLabel:memory_singleCPU_2_task {
+    memory = {check_resource(params.singleCPUMem * 2 * task.attempt)}
   }
-  withName:BamQC {
-    cpus = {check_max(16)}
+  withLabel:memory_singleCPU_task_sq {
+    memory = {check_resource(params.singleCPUMem * task.attempt * task.attempt)}
   }
-  withName:BaseRecalibrator {
-    cpus = {check_max(16)}
+
+  withLabel:memory_max {
+    memory = {check_resource(params.max_memory)}
   }
+
   withName:ConcatVCF {
-    cpus = {check_max(8)}
     // For unknown reasons, ConcatVCF sometimes fails with SIGPIPE
     // (exit code 141). Rerunning the process will usually work.
     errorStrategy = {task.exitStatus == 141 ? 'retry' : 'terminate'}
   }
   withName:FastQCBAM {
-    cpus = {check_max(2)}
     errorStrategy = {task.exitStatus == 143 ? 'retry' : 'ignore'}
   }
   withName:FastQCFQ {
-    // FastQC is only capable of running one thread per fastq file.
-    cpus = {check_max(2)}
     errorStrategy = {task.exitStatus == 143 ? 'retry' : 'ignore'}
   }
-  withName:GatherBQSRReports {
-    cpus = {check_max(2)}
-  }
   withName:MapReads {
-    memory = {check_max(check_max(60.GB * task.attempt))}
-  }
-  withName:MarkDuplicates {
-    cpus = {check_max(16)}
-  }
-  withName:MergeBamMapped {
-    cpus = {check_max(8)}
-  }
-  withName:MergeBamRecal {
-    cpus = {check_max(8)}
+    memory = {check_resource(check_resource(60.GB * task.attempt))}
   }
   withName:MultiQC {
     errorStrategy = {task.exitStatus == 143 ? 'retry' : 'ignore'}
   }
-  withName:SamtoolsStats {
-    cpus = {check_max(2)}
-  }
   withName:Snpeff {
     container = {(params.annotation_cache && params.snpEff_cache) ? 'nfcore/sarek:dev' : "nfcore/sareksnpeff:dev.${params.genome}"}
     errorStrategy = {task.exitStatus == 143 ? 'retry' : 'ignore'}
   }
   withLabel:VEP {
     container = {(params.annotation_cache && params.vep_cache) ? 'nfcore/sarek:dev' : "nfcore/sarekvep:dev.${params.genome}"}
-    cpus = {check_max(4)}
     errorStrategy = {task.exitStatus == 143 ? 'retry' : 'ignore'}
   }
 }
diff --git a/docs/containers.md b/docs/containers.md
@@ -47,6 +47,37 @@ For annotation, the main container can be used, but the cache has to be download
 - Contain **[VEP][vep-link]** 96.0
 - Contain cache for `GRCh37`, `GRCh38`, or `GRCm38`
 
+## Using helper script
+
+An helper script, used for testing can also be used to help with pulling docker containers, or building singularity images.
+The following parameters can be used:
+
+### Engine: -n
+
+Specify which container engine to use: `docker` or `singularity`.
+Default:`docker`
+
+### Containers: -c
+
+Specify which containers to build: `SNPEFF`, `VEP` or `ALL`.
+Default:`ALL`
+
+### Version: -T
+
+Specify which release to pull or build: any tagged release, or `dev`.
+Default:`dev`
+
+### Genome: -g
+Specify which release genome to use for annotation containers (`sareksnpeff`, `sarekvep`): `GRCh37`, `GRCh38`, `smallGRCh37`, `CanFan3.1`, `GRCm38`.
+Default:`smallGRCh37`
+
+### Singularity
+To specify where to build singularity image, use the Nextflow ENV variable `NXF_SINGULARITY_CACHEDIR`, ie:
+```bash
+NXF_SINGULARITY_CACHEDIR=/data/singularity ./scripts/download_image.sh -n singularity -t ALL -T dev -g GRCh38
+```
+That will build the main container, plus the annotation containers (`sareksnpeff`, `sarekvep`) for `GRCh38`, in the `/data/singularity` folder 
+
 ## Building your own
 Our containers are designed using [Conda](https://conda.io/).
 The `environment.yml` file can easilly be modified if particular versions of tools are more suited to your needs.

diff --git a/docs/images/BTB.svg → docs/images/BTB_logo.svg b/docs/images/BTB.svg → docs/images/BTB_logo.svg
diff --git a/docs/images/CAW_logo.png b/docs/images/CAW_logo.png