Integrate cellranger workflow

.github/CONTRIBUTING.md

@@ -15,8 +15,7 @@ Contributions to the code are even more welcome ;)
 
 If you'd like to write some code for nf-core/scrnaseq, the standard workflow is as follows:
 
-1. Check that there isn't already an issue about your idea in the [nf-core/scrnaseq issues](https://github.com/nf-core/scrnaseq/issues) to avoid duplicating work
-    * If there isn't one already, please create one so that others know you're working on this
+1. Check that there isn't already an issue about your idea in the [nf-core/scrnaseq issues](https://github.com/nf-core/scrnaseq/issues) to avoid duplicating work. If there isn't one already, please create one so that others know you're working on this
 2. [Fork](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) the [nf-core/scrnaseq repository](https://github.com/nf-core/scrnaseq) to your GitHub account
 3. Make the necessary changes / additions within your forked repository following [Pipeline conventions](#pipeline-contribution-conventions)
 4. Use `nf-core schema build` and add any new parameters to the pipeline JSON schema (requires [nf-core tools](https://github.com/nf-core/tools) >= 1.10).
@@ -49,9 +48,9 @@ These tests are run both with the latest available version of `Nextflow` and als
 
 :warning: Only in the unlikely and regretful event of a release happening with a bug.
 
-* On your own fork, make a new branch `patch` based on `upstream/master`.
-* Fix the bug, and bump version (X.Y.Z+1).
-* A PR should be made on `master` from patch to directly this particular bug.
+- On your own fork, make a new branch `patch` based on `upstream/master`.
+- Fix the bug, and bump version (X.Y.Z+1).
+- A PR should be made on `master` from patch to directly this particular bug.
 
 ## Getting help
 
@@ -73,7 +72,7 @@ If you wish to contribute a new step, please use the following coding standards:
 6. Add sanity checks and validation for all relevant parameters.
 7. Perform local tests to validate that the new code works as expected.
 8. If applicable, add a new test command in `.github/workflow/ci.yml`.
-9. Update MultiQC config `assets/multiqc_config.yaml` so relevant suffixes, file name clean up and module plots are in the appropriate order. If applicable, add a [MultiQC](https://https://multiqc.info/) module.
+9. Update MultiQC config `assets/multiqc_config.yml` so relevant suffixes, file name clean up and module plots are in the appropriate order. If applicable, add a [MultiQC](https://https://multiqc.info/) module.
 10. Add a description of the output files and if relevant any appropriate images from the MultiQC report to `docs/output.md`.
 
 ### Default values
@@ -92,8 +91,8 @@ The process resources can be passed on to the tool dynamically within the proces
 
 Please use the following naming schemes, to make it easy to understand what is going where.
 
-* initial process channel: `ch_output_from_<process>`
-* intermediate and terminal channels: `ch_<previousprocess>_for_<nextprocess>`
+- initial process channel: `ch_output_from_<process>`
+- intermediate and terminal channels: `ch_<previousprocess>_for_<nextprocess>`
 
 ### Nextflow version bumping
 

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -1,9 +1,7 @@
-
 name: Bug report
 description: Report something that is broken or incorrect
 labels: bug
 body:
-
   - type: markdown
     attributes:
       value: |

.github/PULL_REQUEST_TEMPLATE.md

@@ -10,16 +10,15 @@ Remember that PRs should be made against the dev branch, unless you're preparing
 
 Learn more about contributing: [CONTRIBUTING.md](https://github.com/nf-core/scrnaseq/tree/master/.github/CONTRIBUTING.md)
 -->
-<!-- markdownlint-disable ul-indent -->
 
 ## PR checklist
 
 - [ ] This comment contains a description of changes (with reason).
 - [ ] If you've fixed a bug or added code that should be tested, add tests!
-    - [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/nf-core/scrnaseq/tree/master/.github/CONTRIBUTING.md)
-    - [ ] If necessary, also make a PR on the nf-core/scrnaseq _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository.
+  - [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/nf-core/scrnaseq/tree/master/.github/CONTRIBUTING.md)
+  - [ ] If necessary, also make a PR on the nf-core/scrnaseq _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository.
 - [ ] Make sure your code lints (`nf-core lint`).
-- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker`).
+- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker --outdir <OUTDIR>`).
 - [ ] Usage Documentation in `docs/usage.md` is updated.
 - [ ] Output Documentation in `docs/output.md` is updated.
 - [ ] `CHANGELOG.md` is updated.

.github/workflows/branch.yml

@@ -15,7 +15,6 @@ jobs:
         run: |
           { [[ ${{github.event.pull_request.head.repo.full_name }} == nf-core/scrnaseq ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]]
 
-
       # If the above check failed, post a comment on the PR explaining the failure
       # NOTE - this doesn't currently work if the PR is coming from a fork, due to limitations in GitHub actions secrets
       - name: Post PR comment
@@ -43,4 +42,3 @@ jobs:
             Thanks again for your contribution!
           repo-token: ${{ secrets.GITHUB_TOKEN }}
           allow-repeats: false
-

.github/workflows/ci.yml

@@ -33,6 +33,7 @@ jobs:
             "test,docker --aligner alevin",
             "test,docker --aligner kallisto",
             "test,docker --aligner star",
+            "test,docker --aligner cellranger"
           ]
     steps:
       - name: Check out pipeline code

.github/workflows/linting.yml

@@ -1,107 +1,43 @@
 name: nf-core linting
 # This workflow is triggered on pushes and PRs to the repository.
-# It runs the `nf-core lint` and markdown lint tests to ensure that the code meets the nf-core guidelines
+# It runs the `nf-core lint` and markdown lint tests to ensure
+# that the code meets the nf-core guidelines.
 on:
   push:
   pull_request:
   release:
     types: [published]
 
 jobs:
-  Markdown:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v2
-      - uses: actions/setup-node@v1
-        with:
-          node-version: '10'
-      - name: Install markdownlint
-        run: npm install -g markdownlint-cli
-      - name: Run Markdownlint
-        run: markdownlint .
-
-      # If the above check failed, post a comment on the PR explaining the failure
-      - name: Post PR comment
-        if: failure()
-        uses: mshick/add-pr-comment@v1
-        with:
-          message: |
-            ## Markdown linting is failing
-
-            To keep the code consistent with lots of contributors, we run automated code consistency checks.
-            To fix this CI test, please run:
-
-            * Install `markdownlint-cli`
-                * On Mac: `brew install markdownlint-cli`
-                * Everything else: [Install `npm`](https://www.npmjs.com/get-npm) then [install `markdownlint-cli`](https://www.npmjs.com/package/markdownlint-cli) (`npm install -g markdownlint-cli`)
-            * Fix the markdown errors
-                * Automatically: `markdownlint . --fix`
-                * Manually resolve anything left from `markdownlint .`
-
-            Once you push these changes the test should pass, and you can hide this comment :+1:
-
-            We highly recommend setting up markdownlint in your code editor so that this formatting is done automatically on save. Ask about it on Slack for help!
-
-            Thanks again for your contribution!
-          repo-token: ${{ secrets.GITHUB_TOKEN }}
-          allow-repeats: false
-
   EditorConfig:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
 
-      - uses: actions/setup-node@v1
-        with:
-          node-version: '10'
+      - uses: actions/setup-node@v2
 
       - name: Install editorconfig-checker
         run: npm install -g editorconfig-checker
 
       - name: Run ECLint check
-        run: editorconfig-checker -exclude README.md $(git ls-files | grep -v test)
+        run: editorconfig-checker -exclude README.md $(find .* -type f | grep -v '.git\|.py\|.md\|json\|yml\|yaml\|html\|css\|work\|.nextflow\|build\|nf_core.egg-info\|log.txt\|Makefile')
 
-  YAML:
+  Prettier:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v1
-      - uses: actions/setup-node@v1
-        with:
-          node-version: '10'
-      - name: Install yaml-lint
-        run: npm install -g yaml-lint
-      - name: Run yaml-lint
-        run: yamllint $(find ${GITHUB_WORKSPACE} -type f -name "*.yml" -o -name "*.yaml")
-
-      # If the above check failed, post a comment on the PR explaining the failure
-      - name: Post PR comment
-        if: failure()
-        uses: mshick/add-pr-comment@v1
-        with:
-          message: |
-            ## YAML linting is failing
-
-            To keep the code consistent with lots of contributors, we run automated code consistency checks.
-            To fix this CI test, please run:
-
-            * Install `yaml-lint`
-                * [Install `npm`](https://www.npmjs.com/get-npm) then [install `yaml-lint`](https://www.npmjs.com/package/yaml-lint) (`npm install -g yaml-lint`)
-            * Fix the markdown errors
-                * Run the test locally: `yamllint $(find . -type f -name "*.yml" -o -name "*.yaml")`
-                * Fix any reported errors in your YAML files
+      - uses: actions/checkout@v2
 
-            Once you push these changes the test should pass, and you can hide this comment :+1:
+      - uses: actions/setup-node@v2
 
-            We highly recommend setting up yaml-lint in your code editor so that this formatting is done automatically on save. Ask about it on Slack for help!
+      - name: Install Prettier
+        run: npm install -g prettier
 
-            Thanks again for your contribution!
-          repo-token: ${{ secrets.GITHUB_TOKEN }}
-          allow-repeats: false
+      - name: Run Prettier --check
+        run: prettier --check ${GITHUB_WORKSPACE}
 
   nf-core:
     runs-on: ubuntu-latest
     steps:
-
       - name: Check out pipeline code
         uses: actions/checkout@v2
 
@@ -112,10 +48,10 @@ jobs:
           wget -qO- get.nextflow.io | bash
           sudo mv nextflow /usr/local/bin/
 
-      - uses: actions/setup-python@v1
+      - uses: actions/setup-python@v3
         with:
-          python-version: '3.6'
-          architecture: 'x64'
+          python-version: "3.6"
+          architecture: "x64"
 
       - name: Install dependencies
         run: |
@@ -142,4 +78,3 @@ jobs:
             lint_log.txt
             lint_results.md
             PR_number.txt
-

.github/workflows/linting_comment.yml

@@ -1,4 +1,3 @@
-
 name: nf-core linting comment
 # This workflow is triggered after the linting action is complete
 # It posts an automated comment to the PR, even if the PR is coming from a fork
@@ -27,4 +26,3 @@ jobs:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           number: ${{ steps.pr_number.outputs.pr_number }}
           path: linting-logs/lint_results.md
-

.nf-core.yml

@@ -0,0 +1 @@
+repository_type: pipeline

.prettierrc.yml

@@ -0,0 +1 @@
+printWidth: 120

README.md

@@ -18,9 +18,12 @@
 
 ## Introduction
 
-The **nf-core/scrnaseq** is a bioinformatics best-practise analysis pipeline for transciptomic data of single-cell RNAs. The pipeline creates count matrices from FASTQ sequence reads. It is suitable for droplets-based sequencing technologies.
-The pipeline is built using [Nextflow](https://www.nextflow.io), a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It comes with docker containers making installation trivial and results highly reproducible.
+<!-- TODO nf-core: Write a 1-2 sentence summary of what data the pipeline is for and what it does -->
+**nf-core/scrnaseq** is a bioinformatics best-practice analysis pipeline for processing 10x Genomics single-cell RNA-seq data.
 
+The pipeline is built using [Nextflow](https://www.nextflow.io), a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. The [Nextflow DSL2](https://www.nextflow.io/docs/latest/dsl2.html) implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. Where possible, these processes have been submitted to and installed from [nf-core/modules](https://github.com/nf-core/modules) in order to make them available to all nf-core pipelines, and to everyone within the Nextflow community!
+
+<!-- TODO nf-core: Add full-sized test dataset and amend the paragraph below if applicable -->
 On release, automated continuous integration tests run the pipeline on a full-sized dataset on the AWS cloud infrastructure. This ensures that the pipeline runs on AWS, has sensible resource allocation defaults set to run on real-world datasets, and permits the persistent storage of results to benchmark between pipeline releases and other analysis sources. The results obtained from the full-sized test can be viewed on the [nf-core website](https://nf-co.re/scrnaseq/results).
 
 ## Pipeline summary

bin/check_samplesheet.py

@@ -112,7 +112,8 @@ def check_samplesheet(file_in, file_out):
                 sample_mapping_dict[sample] = [sample_info]
             else:
                 if sample_info in sample_mapping_dict[sample]:
-                    print_error("Samplesheet contains duplicate rows!", "Line", line)
+                    # print_error("Samplesheet contains duplicate rows!", "Line", line)
+                    sample_mapping_dict[sample].append(sample_info)
                 else:
                     sample_mapping_dict[sample].append(sample_info)
 
@@ -129,7 +130,7 @@ def check_samplesheet(file_in, file_out):
                     print_error("Multiple runs of a sample must be of the same datatype!", "Sample: {}".format(sample))
 
                 for idx, val in enumerate(sample_mapping_dict[sample]):
-                    fout.write(",".join(["{}_T{}".format(sample, idx + 1)] + val) + "\n")
+                    fout.write(",".join(["{}".format(sample)] + val) + "\n")
     else:
         print_error("No entries to process!", "Samplesheet: {}".format(file_in))
 

conf/modules.config

@@ -44,7 +44,7 @@ if(params.aligner == "cellranger") {
             ext.args = "--attribute=gene_biotype:protein_coding --attribute=gene_biotype:lncRNA --attribute=gene_biotype:pseudogene"
         }
         withName: CELLRANGER_MKREF {
-            publishDir = null
+            publishDir = []
         }
         withName: CELLRANGER_COUNT {
             publishDir = [

conf/test.config

@@ -20,10 +20,10 @@ params {
     max_time   = '6.h'
 
     // Input data
-    input = 'https://github.com/nf-core/test-datasets/raw/scrnaseq/samplesheet-2-0.csv'
+    input        = 'https://github.com/nf-core/test-datasets/raw/scrnaseq/samplesheet-2-0.csv'
     genome_fasta = 'https://github.com/nf-core/test-datasets/raw/scrnaseq/reference/GRCm38.p6.genome.chr19.fa'
-    gtf = 'https://github.com/nf-core/test-datasets/raw/scrnaseq/reference/gencode.vM19.annotation.chr19.gtf'
-    protocol = '10XV2'
+    gtf          = 'https://github.com/nf-core/test-datasets/raw/scrnaseq/reference/gencode.vM19.annotation.chr19.gtf'
+    protocol     = '10XV2'
     // Ignore `--input` as otherwise the parameter validation will throw an error
     schema_ignore_params = 'genomes,input_paths,input'
 }

docs/README.md

@@ -2,9 +2,9 @@
 
 The nf-core/scrnaseq documentation is split into the following pages:
 
-* [Usage](usage.md)
-    * An overview of how the pipeline works, how to run it and a description of all of the different command-line flags.
-* [Output](output.md)
-    * An overview of the different results produced by the pipeline and how to interpret them.
+- [Usage](usage.md)
+  - An overview of how the pipeline works, how to run it and a description of all of the different command-line flags.
+- [Output](output.md)
+  - An overview of the different results produced by the pipeline and how to interpret them.
 
 You can find a lot more documentation about installing, configuring and running nf-core pipelines on the website: [https://nf-co.re](https://nf-co.re)

lib/NfcoreSchema.groovy

@@ -27,7 +27,7 @@ class NfcoreSchema {
     /* groovylint-disable-next-line UnusedPrivateMethodParameter */
     public static void validateParameters(workflow, params, log, schema_filename='nextflow_schema.json') {
         def has_error = false
-        //=====================================================================//
+        //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~//
         // Check for nextflow core params and unexpected params
         def json = new File(getSchemaPath(workflow, schema_filename=schema_filename)).text
         def Map schemaParams = (Map) new JsonSlurper().parseText(json).get('definitions')
@@ -135,7 +135,7 @@ class NfcoreSchema {
             }
         }
 
-        //=====================================================================//
+        //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~//
         // Validate parameters against the schema
         InputStream input_stream = new File(getSchemaPath(workflow, schema_filename=schema_filename)).newInputStream()
         JSONObject raw_schema = new JSONObject(new JSONTokener(input_stream))

modules.json

@@ -4,13 +4,13 @@
     "repos": {
         "nf-core/modules": {
             "cellranger/count": {
-                "git_sha": "e2ba70ed9a1d2f09aa77b0744ea50b447c35f696"
+                "git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
             },
             "cellranger/mkgtf": {
-                "git_sha": "e2ba70ed9a1d2f09aa77b0744ea50b447c35f696"
+                "git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
             },
             "cellranger/mkref": {
-                "git_sha": "e2ba70ed9a1d2f09aa77b0744ea50b447c35f696"
+                "git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
             },
             "custom/dumpsoftwareversions": {
                 "git_sha": "20d8250d9f39ddb05dfb437603aaf99b5c0b2b41"
@@ -41,4 +41,4 @@
             }
         }
     }
-}
+}

modules/local/kallistobustools_count.nf

@@ -8,6 +8,10 @@ process KALLISTOBUSTOOLS_COUNT {
         'quay.io/biocontainers/kb-python:0.25.1--py_0' }"
 
     input:
+    //
+    // Input reads are expected to come as: [ meta, [ pair1_read1, pair1_read2, pair2_read1, pair2_read2 ] ]
+    // Input array for a sample is created in the same order reads appear in samplesheet as pairs from replicates are appended to array.
+    //
     tuple   val(meta),  path(reads)
     path    index
     path    t2g
@@ -38,7 +42,7 @@ process KALLISTOBUSTOOLS_COUNT {
         -x $technology \\
         $args \\
         -o ${prefix}_kallistobustools_count \\
-        ${reads[0]} ${reads[1]}
+        ${reads.join( " " )}
 
     cat <<-END_VERSIONS > versions.yml
     "${task.process}":