Support paired-end reads and custom genomes #105

kelly-sovacool · 2023-09-29T16:57:53Z

Changes

This is a major refactor with new features:

Support paired end reads. A mix of single or paired end samples can be provided in the same sample sheet, as long as each chip + input pair is the same type.
PREP_GENOME is a new subworkflow that allows users to give a fasta and gtf file to create a custom reference genome.
Custom reusable modules are pulled from CCBR/nf-modules

Issues

Resolves Support paired-end reads #24
Resolves Optional subworkflow to prepare the genome #86

PR Checklist

(~~Strikethrough~~ any points that are not applicable.)

This comment contains a description of changes with justifications, with any relevant issues linked.
~~[ ] Write unit tests for any new features, bug fixes, or other code changes.~~ testing framework not yet implemented
~~[ ] Update docs if there are any API changes.~~ on hold until before public release
If a new nextflow process is implemented, define the process container and stub.
Update CHANGELOG.md with a short description of any user-facing changes and reference the PR number. Guidelines: https://keepachangelog.com/en/1.1.0/

towards #24

…s separate subworkflow

see https://github.com/nf-core/modules/blob/29595ea381f4be8308c4af075eb64764c0249739/modules/nf-core/phantompeakqualtools/main.nf

breaks for paired end

we do need to use -f 66 with ppqt. will need to fix samtools view -q 6 for bam with vishal's custom script This reverts commit caa2ec2.

source https://github.com/CCBR/Pipeliner/blob/86c6ccaa3d58381a0ffd696bbf9c047e4f991f9e/Results-template/Scripts/bam_filter_by_mapq.py

…into paired-end

it needs absolute paths in the config file

head -n 94 /data/CCBR_Pipeliner/db/PipeDB/Indices/hg38_basic/indexes/hg38.blacklist_v3.chrM.chr_rDNA.fa > tests/data/test.blacklist

from https://raw.githubusercontent.com/dpryan79/ChromosomeMappings/4ead5cc20b6a5d394191cc0bac897759deb02a80/R64-1-1_ensembl2UCSC.txt

kelly-sovacool added 30 commits September 28, 2023 14:27

chore: create module & subworkflow scaffolding for prepare_genome

e1b4511

feat: support paired-end reads in ALIGN_BLACKLIST

691349c

towards #24

feat: use tmpdir in ppqt

c218d80

feat: support paired end & tmpdir in ppqt

6a79397

feat: use Picard markduplicates for paired end; refactor with dedup a…

73c6cdc

…s separate subworkflow

refactor: likely don't need bai w/ picard bam

18a4b0f

chore: Merge branch 'main' into paired-end

d231416

feat: use paired test dataset

72ffc5b

feat: accept paired end reads for all peak callers

305ab38

fix: ppqt accepts paired BAM

caa2ec2

see https://github.com/nf-core/modules/blob/29595ea381f4be8308c4af075eb64764c0249739/modules/nf-core/phantompeakqualtools/main.nf

chore: cleanup unused variables

fa01f91

fix: bedtools output filename

36347d3

fix: sicer channel doesn't need format

4079b3c

feat: create test samplesheet w/ mixed single & paired end

27a90b4

fix: output dedup flagstat for qc table

9119083

feat: support paired end in deeptools fingerprint & bamCoverage

dbcdc32

fix: don't use getBaseName(2) in fastqc

5df4ea4

breaks for paired end

fix: Revert "fix: ppqt accepts paired BAM"

8bf1ff6

we do need to use -f 66 with ppqt. will need to fix samtools view -q 6 for bam with vishal's custom script This reverts commit caa2ec2.

feat: add special bam PE filter script from old pipeliner

9427160

source https://github.com/CCBR/Pipeliner/blob/86c6ccaa3d58381a0ffd696bbf9c047e4f991f9e/Results-template/Scripts/bam_filter_by_mapq.py

chore: Merge branch 'paired-end' of https://github.com/CCBR/CHAMPAGNE …

d61187c

…into paired-end

fix: use picard for bam_to_fastq when paired reads

3c06d39

style: don't need equals sign in process container def

d1e2f2a

fix: use BAM_TO_FASTQ

31c4ff6

refactor: simplify logic for ppqt bam preparation

93f9b42

refactor: get index files from PREPARE_GENOME subworkflow [WIP]

247e4fe

fix: explicitly pass blacklist index name

8b5fd58

refactor: use ternary operator for blacklist name

bce5ba4

refactor: get effective genome size from PREP_GENOME subwf

f1304ad

refactor: macs accepts efg as input

e020e27

fix: typo

bf5bca1

kelly-sovacool added 24 commits October 7, 2023 13:02

fix: stub for bwa mem

076b75a

fix: add/update stubs

cd044cc

fix: fastqc stub output to avoid file name collisions

4c16946

feat: use test_paired for ci_stub

a5761cf

fix: meta of fasta files

1932261

feat: optionally skip fastq-screen for CI

ef815b5

it needs absolute paths in the config file

fix: collect bwa index output channel

03fe12f

test: update configs

2847a3a

ci: run real test data on github actions

3e0b161

ci: generate fake blacklist for CI

729f143

head -n 94 /data/CCBR_Pipeliner/db/PipeDB/Indices/hg38_basic/indexes/hg38.blacklist_v3.chrM.chr_rDNA.fa > tests/data/test.blacklist

fix: specify tag for cvbio container

7c518b1

fix: docker container name

0586403

fix: must escape backslashes for nextflow

9153591

ci: only launch full run if requested from manual dispatch

d444780

ci: always upload artifact

e006ba8

fix: don't emit metadata, add it in the subwf

7fbb28e

fix: switch to ccbr/nf-modules khmer with stub

7133d34

feat: download chrom mapping for R64

fb33772

from https://raw.githubusercontent.com/dpryan79/ChromosomeMappings/4ead5cc20b6a5d394191cc0bac897759deb02a80/R64-1-1_ensembl2UCSC.txt

fix: typos

1539825

fix: mitochondrial contig is labeled 'MT' in this genome

092b3e0

fix: update cvbio container w/ older java

035b74f

test: tweak resources

bd99177

fix: make DIY filter for negative GEM peaks

631f854

fix: parse chrom lengths

44b3735

kelly-sovacool marked this pull request as ready for review October 11, 2023 14:17

kelly-sovacool and others added 3 commits October 11, 2023 10:31

test: consolidate test & test_paired into one profile

757e068

docs: update w/ new custom genome option

4abed35

Merge branch 'main' into paired-end

fa91bbd

kelly-sovacool merged commit 509a082 into main Oct 11, 2023
2 checks passed

kelly-sovacool deleted the paired-end branch October 11, 2023 14:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support paired-end reads and custom genomes #105

Support paired-end reads and custom genomes #105

kelly-sovacool commented Sep 29, 2023 •

edited

Loading

Support paired-end reads and custom genomes #105

Support paired-end reads and custom genomes #105

Conversation

kelly-sovacool commented Sep 29, 2023 • edited Loading

Changes

Issues

PR Checklist

kelly-sovacool commented Sep 29, 2023 •

edited

Loading