Absorbs latest nfcore and cgpu fork changes (#6) (#7)

* nf-core bump-version . 2.5.1dev

* Remove PublishDirMode from test profile (nf-core#40)

* remove PublishDirMode from test profile

* update all tools

* minor updates + typo fix (nf-core#42)

* minor updates + typo fix

* fix VEP automated builds

* add location for abstracts

* remove reference to old buil.nf script

* update CHANGELOG

* Update docs/reference.md

Co-Authored-By: Szilveszter Juhos <szilveszter.juhos@scilifelab.se>

* Update docs/reference.md

* Update docs/reference.md

* Worfklow (nf-core#45)

* Add workflow figure
* Include workflow figure in readme
* Update CHANGELOG

* add minimal genome and update some processes

* Start adding mouse data

* Update iGenomes.config

* Add tbi

* Drop ASCAT files

* apply changes from 2.5.1 to dev

* bump version to 2.5.2dev

* update CHANGELOG

* update tiddit to 2.8.1

* Use Version 98 of Mouse

* Add for grcm38

* Adjust mus musculus DB

* Annotation

* add smallerGRCh37 and minimalGRCh37

* use bwa aln when no knowIndels, otherwise use bwa mem, noIntervals currently in the process of being added everywhere

* don't use bwa aln

* add automatic generation of intervals file based on fastaFai file

* Adjusted genomes.config

* Should be list

* Set genomes_base to something

* Revert back

* enable CreateIntervalsBed for intervals_list from GATK Bundle

* Add proper calling list

* Use the bed file

* remove temp file

* update CHANGELOG

* Fix genome fa.fai

* Add in mgpv5

* Try short track

* Add in species handling

* Document new parameter species

* Add changelog

* Fix iGenomes stuff

* Add in note about GRCm38

* Fix small fai index issue

* Adjusted quotes in genomes.config

* And the same for igenomes

* Better folder structure for Mouse Genome Project data

* Minor adjustment to propoer paths

* Apply suggestions from code review

Add changes by Maxime

Co-Authored-By: Maxime Garcia <max.u.garcia@gmail.com>

* Remove space

* Move it up

* Update CHANGELOG.md

Co-Authored-By: Maxime Garcia <max.u.garcia@gmail.com>

* add minimal tests

* fix processes with no intervals

* add comments

* params noIntervals -> no_intervals

* sort genomes + add news

* code polishing

* update CHANGELOG

* add split_fastq params to split the fastq files with the splitFastq() nf method

* add tests

* temporarely remove TIDDIT tests

* add sention for bwa mem

* disable docker and singularity

* disable container

* add fastaFai for bwamem

* remove module samtools from label sentieon

* fix output from bwa mem

* fix output channel BamMapped from MapReads

* set params.sentieon to null by default

* add SentieonDedup process

* fix typo

* add fastaFai to SentieonDedup process

* fix bam indexing

* fix bam indexing

* fix bam indexing

* add SentieonBQSR

* add label sentieon to SentieonBQSR

* fix metrics output for SentieonBQSR

* increase cpus for Sentieon BQSR

* remove indexing

* add index for dedup

* bwa mem sentieon specific process

* TSV file for sentieon Dedup

* TSV for every step for Sentieon

* recal -> deduped

* fix input for TSV recalibrate

* enable restart from recalibrate with TSV with Sentieon

* fix sention variant calling from mapping and recalibrate

* code polishing

* add dump tag for imput sample

* add dump tag for bamDedupedSentieon

* code polishing

* code polishing

* code polishing

* code polishing

* remove when statement

* fix typo

* remove tsv for recalibrate with sentieon

* add dnascope dnaseq

* fix dnascope

* add TNscope process

* fix TNscope output

* add pon for TNscope

* add params.pon_index

* add annotation for sention DNAseq, DNAscope, TNscope

* add default pon_index

* typo

* fix typo

* improve automatic annotation

* typo

* typo

* add condition on when statement on TNscope

* clean up

* code polish

* add CODEOWNERS file

* add when statement on all sentieon processes with params.sentieon

* remove munin sentieon specific configs from config

* load sarek specific config

* update path to specific config

* update docs

* remove Freebayes

* update workflow image

* remove old logo

* fix tests

* add docs about params split_fastq

* update CHANGELOG

* improve docs

* more tests but less NF versions

* actually run the tests

* typo

* simplify configs

* add test for mpileup

* go crazy with tests

* fix tests

* includ test.config

* restore FreeBayes

* remove label memory_max from BaseRecalibrator process to fix nf-core#72

* add --skipQC all and --tools Manta,mpileup,Strelka to minimal genome tests

* update Nextflow version

* update Nextflow version

* update Nextflow

* add --step annotation to profile

* don't need to specify step here

* move params initalization

* add docs

* fix markdownlint

* more complete docs + sort genomes

* improve tests

* update docs

* update CHANGELOG

* improve script

* fix tests

* better comments

* better comments

* fix error on channel name

* fix output for MergeBamRecal

* fix MergeBamRecal output

* fix TSV file

* update comments and docs

* add warning for sentieon only processes

* nf-core bump-version . 2.5.2

* manual bump-version . 2.5.2

* update workflow image

* downgrade tools for release

* update CHANGELOG

* clean up and update workflow image

* allow a

* fix workflow image

* Apply suggestions from code review

* Apply suggestions from code review

* Update docs/output.md

* Apply suggestions from code review

* Apply suggestions from code review

* Apply suggestions from code review

* Reformats `bwa mem | samtools sort` command; WIP suboptimal resource usage

* Addresses #5 ;WIP

* Removes max_ resource alloc labels from MarkDuplicatesSpark

* Replaces .md.bam.bai->.md.bai (same as nf-core)

* Add ${markdup_java_options} to MarkDuplicatesSpark (same as nf-core MarkDuplicates)

* Changes MarkDuplicates --verbosity, DEBUG->INFO

* Changes intervalBed.simpleName->intervalBed.baseName; nf-cored

* Removes label cpus_1 from BaseRecalibratorSpark

* Remove cpus_2 labels from ApplyBQSRSpark; DEBUG->INFO

* Changes pseudo file "no_vepFile.txt" from https to s3 link

* Removes java options from ApplyBQSRSpark

* Removes java options from MarkDupesSpark

* Add java-options to MarkDupesSpark; verbosity INFO->ERROR

* Fixes dupe --java-options; 🤦

* Attempt to fix MarkDupesSpark; "--lower-case"->"-CAP"; Removed tmp

* Adds soft-coded allocation of resources to MapReads

* Initialise params for MapReads split resource alloc

* Adds neglected curlies around params

* Adds neglected \ to bash vars

* Adds neglected \ to bash vars

* WIP; MapReads optimisations

* Implement resource alloc between bwa and samtools

* Adds max, med soft coded resource alloc

* Re-labels processes (from hard coded resources to soft)

* Adds extra curlies to addrees priority of eval

* Add explicit declaration of maxForks/process

* Update med resource allocation function

* Add echo true and echo of  ${bwa_cpus} and ${sort_cpus}

* Hard code heap in MarkDuplicatesSpark at 8g

* Correct expected output bai in MarkDupes

* Removes Spark versions; Not stable with low resources

* Removes sorting; Picard might sort?

* Do not assume sorting in MarkDupes

* Adds explicit --ASSUME_SORT_ORDER unsorted

* Adds missing \\

* Omits -k 23

* Bringing sorted back

* Eliminating pipes in mapping step

* Adds bwa -k 23 and GenomeChronicler as tool (cgpu#16)

- [x] Adds -k 23 (bwa mem seed length)
- [x] Exposes as params bwa_cpus, sort_cpus
- [x] Adds GenomeChronicler in tools (sarek logic)

Co-authored-by: Alexander Peltzer <apeltzer@users.noreply.github.com>
Co-authored-by: Maxime Garcia <maxime.garcia@scilifelab.se>
Co-authored-by: Szilveszter Juhos <szilveszter.juhos@scilifelab.se>

Co-authored-by: Alexander Peltzer <apeltzer@users.noreply.github.com>
Co-authored-by: Maxime Garcia <maxime.garcia@scilifelab.se>
Co-authored-by: Szilveszter Juhos <szilveszter.juhos@scilifelab.se>

.circleci/config.yml

@@ -10,11 +10,11 @@ jobs:
       - checkout
       - setup_remote_docker
       - run:
-          command: docker build -t nfcore/sareksnpeff:2.5.1.${GENOME} containers/snpeff/. --build-arg GENOME=${GENOME} --build-arg SNPEFF_CACHE_VERSION=${SNPEFF_CACHE_VERSION}
+          command: docker build -t nfcore/sareksnpeff:2.5.2.${GENOME} containers/snpeff/. --build-arg GENOME=${GENOME} --build-arg SNPEFF_CACHE_VERSION=${SNPEFF_CACHE_VERSION}
       - run:
           command: |
             echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin
-            docker push nfcore/sareksnpeff:2.5.1.${GENOME}
+            docker push nfcore/sareksnpeff:2.5.2.${GENOME}
 
   snpeffgrch38:
     << : *buildsnpeff
@@ -45,10 +45,10 @@ jobs:
       - checkout
       - setup_remote_docker
       - run:
-          command: docker build -t nfcore/sarekvep:2.5.1.${GENOME} containers/vep/. --build-arg GENOME=${GENOME} --build-arg SPECIES=${SPECIES} --build-arg VEP_VERSION=${VEP_VERSION}
+          command: docker build -t nfcore/sarekvep:2.5.2.${GENOME} containers/vep/. --build-arg GENOME=${GENOME} --build-arg SPECIES=${SPECIES} --build-arg VEP_VERSION=${VEP_VERSION}
           no_output_timeout: 3h
       - run:
-          command: echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin ; docker push nfcore/sarekvep:2.5.1.${GENOME}
+          command: echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin ; docker push nfcore/sarekvep:2.5.2.${GENOME}
 
   vepgrch38:
     << : *buildvep

.github/CODEOWNERS

@@ -0,0 +1 @@
+*   @MaxUlysse

.github/CONTRIBUTING.md

@@ -1,47 +1,57 @@
 # nf-core/sarek: Contributing Guidelines
 
-Hi there! Many thanks for taking an interest in improving nf-core/sarek.
+Hi there!
+Many thanks for taking an interest in improving nf-core/sarek.
 
-We try to manage the required tasks for nf-core/sarek using GitHub issues, you probably came to this page when creating one. Please use the pre-filled template to save time.
-
-However, don't be put off by this template - other more general issues and suggestions are welcome! Contributions to the code are even more welcome ;)
-
-> If you need help using or modifying nf-core/sarek then the best place to ask is on the pipeline channel on [Slack](https://nf-core-invite.herokuapp.com/).
+We try to manage the required tasks for nf-core/sarek using GitHub issues, you probably came to this page when creating one.
+Please use the pre-filled template to save time.
 
+However, don't be put off by this template - other more general issues and suggestions are welcome!
+Contributions to the code are even more welcome ;)
 
+> If you need help using or modifying nf-core/sarek then the best place to ask is on the nf-core Slack [#sarek](https://nfcore.slack.com/channels/sarek) channel ([join our Slack here](https://nf-co.re/join/slack)).
 
 ## Contribution workflow
-If you'd like to write some code for nf-core/sarek, the standard workflow
-is as follows:
 
-1. Check that there isn't already an issue about your idea in the
-   [nf-core/sarek issues](https://github.com/nf-core/sarek/issues) to avoid
-   duplicating work.
+If you'd like to write some code for nf-core/sarek, the standard workflow is as follows:
+
+1. Check that there isn't already an issue about your idea in the [nf-core/sarek issues](https://github.com/nf-core/sarek/issues) to avoid duplicating work
     * If there isn't one already, please create one so that others know you're working on this
-2. Fork the [nf-core/sarek repository](https://github.com/nf-core/sarek) to your GitHub account
+2. [Fork](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) the [nf-core/sarek repository](https://github.com/nf-core/sarek) to your GitHub account
 3. Make the necessary changes / additions within your forked repository
-4. Submit a Pull Request against the `dev` branch and wait for the code to be reviewed and merged.
-
-If you're not used to this workflow with git, you can start with some [basic docs from GitHub](https://help.github.com/articles/fork-a-repo/) or even their [excellent interactive tutorial](https://try.github.io/).
+4. Submit a Pull Request against the `dev` branch and wait for the code to be reviewed and merged
 
+If you're not used to this workflow with git, you can start with some [docs from GitHub](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests) or even their [excellent `git` resources](https://try.github.io/).
 
 ## Tests
-When you create a pull request with changes, [Travis CI](https://travis-ci.org/) will run automatic tests.
+
+When you create a pull request with changes, [GitHub Actions](https://github.com/features/actions) will run automatic tests.
 Typically, pull-requests are only fully reviewed when these tests are passing, though of course we can help out before then.
 
 There are typically two types of tests that run:
 
 ### Lint Tests
-The nf-core has a [set of guidelines](http://nf-co.re/guidelines) which all pipelines must adhere to.
+
+`nf-core` has a [set of guidelines](https://nf-co.re/developers/guidelines) which all pipelines must adhere to.
 To enforce these and ensure that all pipelines stay in sync, we have developed a helper tool which runs checks on the pipeline code. This is in the [nf-core/tools repository](https://github.com/nf-core/tools) and once installed can be run locally with the `nf-core lint <pipeline-directory>` command.
 
 If any failures or warnings are encountered, please follow the listed URL for more documentation.
 
 ### Pipeline Tests
-Each nf-core pipeline should be set up with a minimal set of test-data.
-Travis CI then runs the pipeline on this data to ensure that it exists successfully.
+
+Each `nf-core` pipeline should be set up with a minimal set of test-data.
+`GitHub Actions` then runs the pipeline on this data to ensure that it exits successfully.
 If there are any failures then the automated tests fail.
-These tests are run both with the latest available version of Nextflow and also the minimum required version that is stated in the pipeline code.
+These tests are run both with the latest available version of `Nextflow` and also the minimum required version that is stated in the pipeline code.
+
+## Patch
+
+: warning: Only in the unlikely and regretful event of a release happening with a bug.
+
+* On your own fork, make a new branch `patch` based on `upstream/master`.
+* Fix the bug, and bump version (X.Y.Z+1).
+* A PR should be made on `master` from patch to directly this particular bug.
 
 ## Getting help
-For further information/help, please consult the [nf-core/sarek documentation](https://github.com/nf-core/sarek#documentation) and don't hesitate to get in touch on the [sarek pipeline channel](https://nfcore.slack.com/channels/sarek) on [Slack](https://nf-co.re/join/slack).
+
+For further information/help, please consult the [nf-core/sarek documentation](https://nf-co.re/sarek/docs) and don't hesitate to get in touch on the nf-core Slack [#sarek](https://nfcore.slack.com/channels/sarek) channel ([join our Slack here](https://nf-co.re/join/slack)).

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,31 +1,42 @@
+# nf-core/sarek bug report
+
 Hi there!
 
-Thanks for telling us about a problem with the pipeline. Please delete this text and anything that's not relevant from the template below:
+Thanks for telling us about a problem with the pipeline.
+Please delete this text and anything that's not relevant from the template below:
+
+## Describe the bug
 
-#### Describe the bug
 A clear and concise description of what the bug is.
 
-#### Steps to reproduce
+## Steps to reproduce
+
 Steps to reproduce the behaviour:
+
 1. Command line: `nextflow run ...`
 2. See error: _Please provide your error message_
 
-#### Expected behaviour
+## Expected behaviour
+
 A clear and concise description of what you expected to happen.
 
-#### System:
- - Hardware: [e.g. HPC, Desktop, Cloud...]
- - Executor: [e.g. slurm, local, awsbatch...]
- - OS: [e.g. CentOS Linux, macOS, Linux Mint...]
- - Version [e.g. 7, 10.13.6, 18.3...]
+## System
+
+- Hardware: <!-- [e.g. HPC, Desktop, Cloud...] -->
+- Executor: <!-- [e.g. slurm, local, awsbatch...] -->
+- OS: <!-- [e.g. CentOS Linux, macOS, Linux Mint...] -->
+- Version <!-- [e.g. 7, 10.13.6, 18.3...] -->
+
+## Nextflow Installation
+
+- Version: <!-- [e.g. 19.10.0] -->
+
+## Container engine
 
-#### Nextflow Installation:
- - Version: [e.g. 0.31.0]
+- Engine: <!-- [e.g. Conda, Docker or Singularity] -->
+- version: <!-- [e.g. 1.0.0] -->
+- Image tag: <!-- [e.g. nfcore/sarek:2.5.1] -->
 
-#### Container engine:
- - Engine: [e.g. Conda, Docker or Singularity]
- - version: [e.g. 1.0.0]
- - Image tag: [e.g. nfcore/sarek:1.0.0]
+## Additional context
 
-#### Additional context
 Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,16 +1,24 @@
+# nf-core/sarek feature request
+
 Hi there!
 
-Thanks for suggesting a new feature for the pipeline! Please delete this text and anything that's not relevant from the template below:
+Thanks for suggesting a new feature for the pipeline!
+Please delete this text and anything that's not relevant from the template below:
+
+## Is your feature request related to a problem? Please describe
 
-#### Is your feature request related to a problem? Please describe.
 A clear and concise description of what the problem is.
+
 Ex. I'm always frustrated when [...]
 
-#### Describe the solution you'd like
+## Describe the solution you'd like
+
 A clear and concise description of what you want to happen.
 
-#### Describe alternatives you've considered
+## Describe alternatives you've considered
+
 A clear and concise description of any alternative solutions or features you've considered.
 
-#### Additional context
+## Additional context
+
 Add any other context about the feature request here.

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,16 +1,19 @@
-Many thanks to contributing to nf-core/sarek!
+# nf-core/sarek pull request
+
+Many thanks for contributing to nf-core/sarek!
 
 Please fill in the appropriate checklist below (delete whatever is not relevant).
 These are the most common things requested on pull requests (PRs).
 
 ## PR checklist
- - [ ] This comment contains a description of changes (with reason)
- - [ ] If you've fixed a bug or added code that should be tested, add tests!
- - [ ] If necessary, also make a PR on the [nf-core/sarek branch on the nf-core/test-datasets repo](https://github.com/nf-core/test-datasets/pull/new/nf-core/sarek)
- - [ ] Ensure the test suite passes (`nextflow run . -profile test,docker`).
- - [ ] Make sure your code lints (`nf-core lint .`).
- - [ ] Documentation in `docs` is updated
- - [ ] `CHANGELOG.md` is updated
- - [ ] `README.md` is updated
 
-**Learn more about contributing:** [guidelines](https://github.com/nf-core/sarek/tree/master/.github/CONTRIBUTING.md)
+- [ ] This comment contains a description of changes (with reason)
+- [ ] If you've fixed a bug or added code that should be tested, add tests!
+- [ ] If necessary, also make a PR on the [nf-core/sarek branch on the nf-core/test-datasets repo](https://github.com/nf-core/test-datasets/pull/new/nf-core/sarek)
+- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker`).
+- [ ] Make sure your code lints (`nf-core lint .`).
+- [ ] Documentation in `docs` is updated
+- [ ] `CHANGELOG.md` is updated
+- [ ] `README.md` is updated
+
+**Learn more about contributing:** [CONTRIBUTING.md](https://github.com/nf-core/sarek/tree/master/.github/CONTRIBUTING.md)

.github/markdownlint.yml

@@ -9,4 +9,6 @@ no-duplicate-header:
     siblings_only: true
 no-inline-html: 
     allowed_elements:
-        - img
+        - a
+        - img
+        - p

.github/workflows/branch.yml

@@ -7,10 +7,10 @@ on:
 
 jobs:
   test:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-18.04
     steps:
       # PRs are only ok if coming from an nf-core dev branch
       - uses: actions/checkout@v1
       - name: Check PRs
         run: |
-          { [[ $(git remote get-url origin) == *nf-core/sarek ]] && [[ ${GITHUB_BASE_REF} = "master" ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } || [[ ${GITHUB_HEAD_REF} == patch* ]]
+          { [[ $(git remote get-url origin) == *nf-core/sarek ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } || [[ ${GITHUB_HEAD_REF} == "patch" ]]