Move long read preprocessing into a subworkflow #674

muabnezor · 2024-09-20T12:56:49Z

This PR moves long read preprocessing into a subworkflow. It also adds the option to run porechop-abi, instead of porechop, for long read adaptertrimming.

PR checklist

…module to porechop/abi

…/modules.config

github-actions · 2024-09-20T12:59:50Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 2cfd5e5

+| ✅ 308 tests passed       |+
#| ❔   2 tests were ignored |#
!| ❗   5 tests had warnings |!

❗ Test warnings:

pipeline_todos - TODO string in main.nf: Remove this line if you don't need a FASTA file [TODO: try and test using for --host_fasta and --host_genome]
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!

❔ Tests ignored:

nextflow_config - Config default ignored: params.phix_reference
nextflow_config - Config default ignored: params.lambda_reference

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-mag_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-mag_logo_light.png
files_exist - File found: docs/images/nf-core-mag_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-mag_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowMag.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.igenomes_base= s3://ngi-igenomes/igenomes/
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.max_cpus= 16
nextflow_config - Config default value correct: params.max_memory= 128.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/
nextflow_config - Config default value correct: params.spades_fix_cpus= -1
nextflow_config - Config default value correct: params.spadeshybrid_fix_cpus= -1
nextflow_config - Config default value correct: params.metabat_rng_seed= 1
nextflow_config - Config default value correct: params.clip_tool= fastp
nextflow_config - Config default value correct: params.reads_minlength= 15
nextflow_config - Config default value correct: params.fastp_qualified_quality= 15
nextflow_config - Config default value correct: params.fastp_cut_mean_quality= 15
nextflow_config - Config default value correct: params.adapterremoval_minquality= 2
nextflow_config - Config default value correct: params.adapterremoval_adapter1= AGATCGGAAGAGCACACGTCTGAACTCCAGTCACNNNNNNATCTCGTATGCCGTCTTCTGCTTG
nextflow_config - Config default value correct: params.adapterremoval_adapter2= AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTAGATCTCGGTGGTCGCCGTATCATT
nextflow_config - Config default value correct: params.bbnorm_target= 100
nextflow_config - Config default value correct: params.bbnorm_min= 5
nextflow_config - Config default value correct: params.longreads_min_length= 1000
nextflow_config - Config default value correct: params.longreads_keep_percent= 90
nextflow_config - Config default value correct: params.longreads_length_weight= 10
nextflow_config - Config default value correct: params.longread_adaptertrimming_tool= porechop_abi
nextflow_config - Config default value correct: params.gtdb_db= https://data.ace.uq.edu.au/public/gtdb/data/releases/release220/220.0/auxillary_files/gtdbtk_package/full_package/gtdbtk_r220_data.tar.gz
nextflow_config - Config default value correct: params.gtdbtk_min_completeness= 50.0
nextflow_config - Config default value correct: params.gtdbtk_max_contamination= 10.0
nextflow_config - Config default value correct: params.gtdbtk_min_perc_aa= 10.0
nextflow_config - Config default value correct: params.gtdbtk_min_af= 0.65
nextflow_config - Config default value correct: params.gtdbtk_pplacer_cpus= 1.0
nextflow_config - Config default value correct: params.gtdbtk_pplacer_useram= false
nextflow_config - Config default value correct: params.genomad_min_score= 0.7
nextflow_config - Config default value correct: params.genomad_splits= 1
nextflow_config - Config default value correct: params.binning_map_mode= group
nextflow_config - Config default value correct: params.min_contig_size= 1500
nextflow_config - Config default value correct: params.min_length_unbinned_contigs= 1000000
nextflow_config - Config default value correct: params.max_unbinned_contigs= 100
nextflow_config - Config default value correct: params.bin_domain_classification_tool= tiara
nextflow_config - Config default value correct: params.tiara_min_length= 3000
nextflow_config - Config default value correct: params.binqc_tool= busco
nextflow_config - Config default value correct: params.checkm_download_url= https://data.ace.uq.edu.au/public/CheckM_databases/checkm_data_2015_01_16.tar.gz
nextflow_config - Config default value correct: params.refine_bins_dastool_threshold= 0.5
nextflow_config - Config default value correct: params.postbinning_input= raw_bins_only
nextflow_config - Config default value correct: params.gunc_database_type= progenomes
nextflow_config - Config default value correct: params.pydamage_accuracy= 0.5
nextflow_config - Config default value correct: params.freebayes_ploidy= 1
nextflow_config - Config default value correct: params.freebayes_min_basequality= 20
nextflow_config - Config default value correct: params.freebayes_minallelefreq= 0.33
nextflow_config - Config default value correct: params.bcftools_view_high_variant_quality= 30
nextflow_config - Config default value correct: params.bcftools_view_medium_variant_quality= 20
nextflow_config - Config default value correct: params.bcftools_view_minimal_allelesupport= 3
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-mag_logo_light.png matches the template
files_unchanged - docs/images/nf-core-mag_logo_light.png matches the template
files_unchanged - docs/images/nf-core-mag_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
readme - README Zenodo placeholder was replaced with DOI.
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (330 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains a matching 'report_comment'.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
base_config - BOWTIE2_HOST_REMOVAL_BUILD found in conf/base.config and Nextflow scripts.
base_config - BOWTIE2_HOST_REMOVAL_ALIGN found in conf/base.config and Nextflow scripts.
base_config - BOWTIE2_PHIX_REMOVAL_ALIGN found in conf/base.config and Nextflow scripts.
base_config - PORECHOP_PORECHOP found in conf/base.config and Nextflow scripts.
base_config - NANOLYSE found in conf/base.config and Nextflow scripts.
base_config - FILTLONG found in conf/base.config and Nextflow scripts.
base_config - CENTRIFUGE_CENTRIFUGE found in conf/base.config and Nextflow scripts.
base_config - KRAKEN2 found in conf/base.config and Nextflow scripts.
base_config - KRONA_KTIMPORTTAXONOMY found in conf/base.config and Nextflow scripts.
base_config - CAT_DB_GENERATE found in conf/base.config and Nextflow scripts.
base_config - CAT found in conf/base.config and Nextflow scripts.
base_config - GTDBTK_CLASSIFYWF found in conf/base.config and Nextflow scripts.
base_config - MEGAHIT found in conf/base.config and Nextflow scripts.
base_config - SPADES found in conf/base.config and Nextflow scripts.
base_config - SPADESHYBRID found in conf/base.config and Nextflow scripts.
base_config - BOWTIE2_ASSEMBLY_ALIGN found in conf/base.config and Nextflow scripts.
base_config - METABAT2_METABAT2 found in conf/base.config and Nextflow scripts.
base_config - MAG_DEPTHS found in conf/base.config and Nextflow scripts.
base_config - BUSCO found in conf/base.config and Nextflow scripts.
base_config - MAXBIN2 found in conf/base.config and Nextflow scripts.
base_config - DASTOOL_DASTOOL found in conf/base.config and Nextflow scripts.
modules_config - conf/modules.config found and not ignored.
modules_config - FASTQC_RAW found in conf/modules.config and Nextflow scripts.
modules_config - FASTP found in conf/modules.config and Nextflow scripts.
modules_config - ADAPTERREMOVAL_PE found in conf/modules.config and Nextflow scripts.
modules_config - ADAPTERREMOVAL_SE found in conf/modules.config and Nextflow scripts.
modules_config - BOWTIE2_PHIX_REMOVAL_ALIGN found in conf/modules.config and Nextflow scripts.
modules_config - BOWTIE2_HOST_REMOVAL_ALIGN found in conf/modules.config and Nextflow scripts.
modules_config - FASTQC_TRIMMED found in conf/modules.config and Nextflow scripts.
modules_config - BBMAP_BBNORM found in conf/modules.config and Nextflow scripts.
modules_config - PORECHOP_PORECHOP found in conf/modules.config and Nextflow scripts.
modules_config - PORECHOP_ABI found in conf/modules.config and Nextflow scripts.
modules_config - FILTLONG found in conf/modules.config and Nextflow scripts.
modules_config - NANOLYSE found in conf/modules.config and Nextflow scripts.
modules_config - NANOPLOT_RAW found in conf/modules.config and Nextflow scripts.
modules_config - NANOPLOT_FILTERED found in conf/modules.config and Nextflow scripts.
modules_config - CENTRIFUGE_CENTRIFUGE found in conf/modules.config and Nextflow scripts.
modules_config - CENTRIFUGE_KREPORT found in conf/modules.config and Nextflow scripts.
modules_config - KRAKEN2 found in conf/modules.config and Nextflow scripts.
modules_config - KREPORT2KRONA_CENTRIFUGE found in conf/modules.config and Nextflow scripts.
modules_config - KRONA_KTIMPORTTAXONOMY found in conf/modules.config and Nextflow scripts.
modules_config - MEGAHIT found in conf/modules.config and Nextflow scripts.
modules_config - SPADES found in conf/modules.config and Nextflow scripts.
modules_config - SPADESHYBRID found in conf/modules.config and Nextflow scripts.
modules_config - QUAST found in conf/modules.config and Nextflow scripts.
modules_config - GENOMAD_ENDTOEND found in conf/modules.config and Nextflow scripts.
modules_config - BOWTIE2_ASSEMBLY_ALIGN found in conf/modules.config and Nextflow scripts.
modules_config - MAG_DEPTHS_PLOT found in conf/modules.config and Nextflow scripts.
modules_config - BIN_SUMMARY found in conf/modules.config and Nextflow scripts.
modules_config - BUSCO_DB_PREPARATION found in conf/modules.config and Nextflow scripts.
modules_config - BUSCO found in conf/modules.config and Nextflow scripts.
modules_config - BUSCO_SAVE_DOWNLOAD found in conf/modules.config and Nextflow scripts.
modules_config - BUSCO_SUMMARY found in conf/modules.config and Nextflow scripts.
modules_config - ARIA2_UNTAR found in conf/modules.config and Nextflow scripts.
modules_config - CHECKM_LINEAGEWF found in conf/modules.config and Nextflow scripts.
modules_config - CHECKM_QA found in conf/modules.config and Nextflow scripts.
modules_config - COMBINE_CHECKM_TSV found in conf/modules.config and Nextflow scripts.
modules_config - GUNC_DOWNLOADDB found in conf/modules.config and Nextflow scripts.
modules_config - GUNC_RUN found in conf/modules.config and Nextflow scripts.
modules_config - GUNC_MERGECHECKM found in conf/modules.config and Nextflow scripts.
modules_config - CAT_DB_GENERATE found in conf/modules.config and Nextflow scripts.
modules_config - CAT found in conf/modules.config and Nextflow scripts.
modules_config - CAT_SUMMARY found in conf/modules.config and Nextflow scripts.
modules_config - GTDBTK_CLASSIFYWF found in conf/modules.config and Nextflow scripts.
modules_config - GTDBTK_SUMMARY found in conf/modules.config and Nextflow scripts.
modules_config - PROKKA found in conf/modules.config and Nextflow scripts.
modules_config - PRODIGAL found in conf/modules.config and Nextflow scripts.
modules_config - FREEBAYES found in conf/modules.config and Nextflow scripts.
modules_config - BCFTOOLS_VIEW found in conf/modules.config and Nextflow scripts.
modules_config - BCFTOOLS_CONSENSUS found in conf/modules.config and Nextflow scripts.
modules_config - BCFTOOLS_INDEX found in conf/modules.config and Nextflow scripts.
modules_config - PYDAMAGE_ANALYZE found in conf/modules.config and Nextflow scripts.
modules_config - PYDAMAGE_FILTER found in conf/modules.config and Nextflow scripts.
modules_config - SAMTOOLS_FAIDX found in conf/modules.config and Nextflow scripts.
modules_config - METABAT2_JGISUMMARIZEBAMCONTIGDEPTHS found in conf/modules.config and Nextflow scripts.
modules_config - METABAT2_METABAT2 found in conf/modules.config and Nextflow scripts.
modules_config - MAXBIN2 found in conf/modules.config and Nextflow scripts.
modules_config - ADJUST_MAXBIN2_EXT found in conf/modules.config and Nextflow scripts.
modules_config - CONCOCT_ found in conf/modules.config and Nextflow scripts.
modules_config - SPLIT_FASTA found in conf/modules.config and Nextflow scripts.
modules_config - DASTOOL_FASTATOCONTIG2BIN_METABAT2 found in conf/modules.config and Nextflow scripts.
modules_config - DASTOOL_FASTATOCONTIG2BIN_MAXBIN2 found in conf/modules.config and Nextflow scripts.
modules_config - DASTOOL_FASTATOCONTIG2BIN_CONCOCT found in conf/modules.config and Nextflow scripts.
modules_config - DASTOOL_FASTATOCONTIG2BIN_TIARA found in conf/modules.config and Nextflow scripts.
modules_config - DASTOOL_DASTOOL found in conf/modules.config and Nextflow scripts.
modules_config - RENAME_POSTDASTOOL found in conf/modules.config and Nextflow scripts.
modules_config - TIARA_TIARA found in conf/modules.config and Nextflow scripts.
modules_config - TIARA_CLASSIFY found in conf/modules.config and Nextflow scripts.
modules_config - TIARA_SUMMARY found in conf/modules.config and Nextflow scripts.
modules_config - MMSEQS_DATABASES found in conf/modules.config and Nextflow scripts.
modules_config - METAEUK_EASYPREDICT found in conf/modules.config and Nextflow scripts.
modules_config - MULTIQC found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 2.14.1

Run details

nf-core/tools version 2.14.1
Run at 2024-10-04 08:52:39

jfy133 · 2024-09-20T14:06:56Z

@muabnezor feel free to open the PR to 'ready for review' when ready :) but very brief skim look good!

muabnezor · 2024-09-23T12:45:15Z

Hi,

In this pull request I exchanged porechop with porechop-abi for preprocessing of long-reads, I think this is an ok exchange. I also added the module chopper, but decided it is better to integrate this tool in a future PR. Maybe it would have a cleaner PR just focusing on refactoring the code, and waiting with introducing new modules altogether. Should I move the short-read preprocessing to a subworkflow in this PR, or wait with this?

jfy133

Looking really good @muabnezor !

I have some requests first:

Please don't remove PORECHOP_PORECHOP or FILTLONG yet, I would like to keep them in for backwards compatibility (maybe some people don't 'trust' yet the newer tools), please have them alongside the new tools but wrapped in an if statement, and a user can select their preferred tool with a new param (e.g. --longread_preprocessing_tool or something).

Please also don't forget:

Update CHANGELOG
Add any new tools to citations.md
Add yourself to the list of contibutors/developers on the GitHub README !

Hi

In this pull request I exchanged porechop with porechop-abi for preprocessing of long-reads, I think this is an ok exchange. I also added the module chopper, but decided it is better to integrate this tool in a future PR. Maybe it would have a cleaner PR just focusing on refactoring the code, and waiting with introducing new modules altogether. Should I move the short-read preprocessing to a subworkflow in this PR, or wait with this?

Yes, lets keep it clean and do one PR per thing you've suggested once you've addressed by comments here (short-read prepocessing next, then chopper), good plan :)!

subworkflows/local/lr_preprocessing.nf

muabnezor · 2024-09-24T07:19:47Z

Sounds great @jfy133! To run any newer modules, they need to be specified with an option, otherwise, we’ll leave the functionality as is for now. I’ll take care of that, along with your additional comments. I apologize for the limited time I have this week to work on this, but I’ll get to it as soon as I can.

jfy133 · 2024-09-24T09:14:29Z

Sounds perfect!

No worries about the time, this is already great progress 👍

…long read preprocessing tools. Currently only has the option to specify porechop_abi, but I decided this is a nice solution for future tools that the user might want to chose among

muabnezor · 2024-09-24T21:45:03Z

What's new

Added parameter longread_preprocessing_tools a comma-separated string
Added filtlong and porechop to multiqc
changed local filtlong to nf-core/filtlong

Fix linting problem

jfy133 · 2024-09-25T07:23:31Z

What's new

* Added parameter longread_preprocessing_tools a comma-separated string

* Added filtlong and porechop to multiqc

* changed local filtlong to nf-core/filtlong

Great, thanks for adding filtling/porechop!

While it's clever from a conciseness PoV I personally don't like this as it's more error prone by users. For example, we then can't get nice dropdowns on the nf-core launch GUI (and other GUI launch platforms).

Finally it's inconsistent with how other parts of the pipeline select a given tool.

I would be grateful if you switch back to one parameter per preprocessing step.

(Note however, I much prefer this development approach of you taking the initiative of a possible optimisation and incorporating it in as a proof of concept, even if I disagree, so if it doesn't annoy you to revert, please continue to take this approch :) )

muabnezor · 2024-09-25T08:04:27Z

Finally it's inconsistent with how other parts of the pipeline select a given tool.

I would be grateful if you switch back to one parameter per preprocessing step.

No problems @jfy133, and I agree, makes more sense to have one parameter per tool. I'll add a --lr_adaptertrimming_tool parameter instead.

I just need to check the multiqc stuff, as filtlong doesn't seem to get included in the report for some reason. Then I'll update all additional documents, and the PR should be ready for review.

jfy133 · 2024-09-25T09:02:13Z

Finally it's inconsistent with how other parts of the pipeline select a given tool.
I would be grateful if you switch back to one parameter per preprocessing step.

No problems @jfy133, and I agree, makes more sense to have one parameter per tool. I'll add a --lr_adaptertrimming_tool parameter instead.

Great :) following the other parameters please don't use shorthand: --longreads_adaptertrimming_tool is fine :)

I just need to check the multiqc stuff, as filtlong doesn't seem to get included in the report for some reason. Then I'll update all additional documents, and the PR should be ready for review.

Huh interesting, I'm pretty sure it does get displayed in MultiQC in taxprofiler, it may only be included in the GeneralStats table?

subworkflows/local/lr_preprocessing.nf

muabnezor · 2024-09-30T10:46:41Z

Still scratching my head about the filtlong log and MultiQC. The log file is passed to MultiQC in the same way as in TaxProfiler, but no filtlong data is included in the MultiQC report.

…module in multiqc

assets/multiqc_config.yml

Co-authored-by: James A. Fellows Yates <jfy133@gmail.com>

muabnezor · 2024-10-04T09:18:33Z

This should be ready @jfy133. Think I'm getting a grip on nf-core now, next PR should be a lot cleaner!

muabnezor · 2024-10-11T10:07:58Z

Have you had time to look at this yet @jfy133?

jfy133 · 2024-10-11T10:47:40Z

Sorry, didn't get to it before because of the meta-omics hackathon and preparing for the nextflow summit, but I have time now!

jfy133

OK, LGTM to me now @muabnezor !

I'm going to make one change to the output docs for you (something that wasn't documented previously), then you can merge in!

I'll give the green tick once I've made the addition

This is a very nice and clean PR, thank you very much! Hopefully the first of many :D

jfy133 · 2024-10-11T10:48:11Z

CHANGELOG.md

 ### `Changed`

+- [#674](https://github.com/nf-core/mag/pull/674/files) - Changed to porechop-abi as default adapter trimming tool for long reads. User can still use porechop if prefered.


Suggested change

- [#674](https://github.com/nf-core/mag/pull/674/files) - Changed to porechop-abi as default adapter trimming tool for long reads. User can still use porechop if prefered.

- [#674](https://github.com/nf-core/mag/pull/674/files) - Changed to porechop-abi as default adapter trimming tool for long reads. User can still use porechop if prefered. (added by @muabnezor)

jfy133 · 2024-10-11T10:51:14Z

CITATIONS.md

@@ -116,6 +116,8 @@

 - [Porechop](https://github.com/rrwick/Porechop)

+- [Porechop-abi](https://github.com/bonsai-team/Porechop_ABI)


Suggested change

- [Porechop-abi](https://github.com/bonsai-team/Porechop_ABI)

- [Porechop-abi](https://github.com/bonsai-team/Porechop_ABI)

> Bonenfant, Q., Noé, L., & Touzet, H. (2022). Porechop_ABI: discovering unknown adapters in ONT sequencing reads for downstream trimming. In BioRxiv. doi: 10.1101/2022.07.07.499093

CHANGELOG.md

jfy133 · 2024-10-11T11:03:55Z

@nf-core-bot fix linting

jfy133

Once tests pass, you can merge @muabnezor ! Thanks again! Lets chat on Slack on next step if unclear :)

jfy133 · 2024-10-11T11:11:34Z

Ah wait, I've not done the template merge yet so it won't let you pass because of the linting, I'll merge for you (and next PR hopefully you can merge in yourself).

adro added 3 commits September 20, 2024 08:25

Added modules porechop/abi and chopper for QC of long reads

08445cf

Move long read preprocessing into subworkflow, and swapping porechop …

5aa3049

…module to porechop/abi

Remove module import from main workflow, and add PORECHOP_ABI in conf…

451612a

…/modules.config

muabnezor requested a review from jfy133 September 20, 2024 12:56

jfy133 marked this pull request as draft September 20, 2024 12:57

muabnezor added the WIP Work in progress label Sep 23, 2024

jfy133 reviewed Sep 23, 2024

View reviewed changes

subworkflows/local/lr_preprocessing.nf Outdated Show resolved Hide resolved

subworkflows/local/lr_preprocessing.nf Outdated Show resolved Hide resolved

subworkflows/local/lr_preprocessing.nf Outdated Show resolved Hide resolved

muabnezor added 3 commits September 24, 2024 21:40

Exchange local filtlong module to nf-core/filtlong

59ef702

Add filtlong and porechop logs to multiqc

cd5ce2f

Added --longread_preprocessing_tools parameters, to let user specify …

5820fb4

…long read preprocessing tools. Currently only has the option to specify porechop_abi, but I decided this is a nice solution for future tools that the user might want to chose among

muabnezor added 2 commits September 25, 2024 07:13

Merge branch 'dev' into move_lr_qc

5335fc5

Update modules/nf-core/filtlong/main.nf

1496ce9

Fix linting problem

prototaxites reviewed Sep 25, 2024

View reviewed changes

subworkflows/local/lr_preprocessing.nf Outdated Show resolved Hide resolved

muabnezor added 4 commits September 30, 2024 08:12

make subworkflow name more verbose

c11601a

make --longread_adaptertrimming_tool as enum porechop or porechop_abi

31deb5c

Make prefix for porechop/porechop-abi more verbose

b2da50b

lint

8955e31

Change default search pattern for filtlong.log files for the filtlong…

644e2b1

…module in multiqc

muabnezor commented Oct 1, 2024

View reviewed changes

assets/multiqc_config.yml Show resolved Hide resolved

muabnezor changed the title ~~Work in Progress: Move long read preprocessing into a subworkflow~~ Move long read preprocessing into a subworkflow Oct 1, 2024

muabnezor and others added 14 commits October 3, 2024 13:42

Update nextflow_schema.json

10ecdc5

Co-authored-by: James A. Fellows Yates <jfy133@gmail.com>

Update nextflow.config

ea6f9a4

Co-authored-by: James A. Fellows Yates <jfy133@gmail.com>

Update subworkflows/local/longread_preprocessing.nf

b43b9f6

Co-authored-by: James A. Fellows Yates <jfy133@gmail.com>

Update subworkflows/local/longread_preprocessing.nf

25981a1

Co-authored-by: James A. Fellows Yates <jfy133@gmail.com>

Update subworkflows/local/longread_preprocessing.nf

edf8dab

Co-authored-by: James A. Fellows Yates <jfy133@gmail.com>

Merge branch 'dev' into move_lr_qc

0a06944

Linting fix

669a854

make porechop-abi default long read adapter trimming tool

a1098f3

Fix changelog

17ba45c

Fix porechop_abi pattern in modules.config

f54c967

Merge branch 'dev' into move_lr_qc

110b5dd

remove chopper citation for now

3472c87

retrigger checks

ab5f482

Merge branch 'dev' into move_lr_qc

2cfd5e5

jfy133 linked an issue Oct 11, 2024 that may be closed by this pull request

Add Porechop_ABI as alternative to Porechop #660

Closed

jfy133 reviewed Oct 11, 2024

View reviewed changes

Apply suggestions from code review

7b31394

jfy133 reviewed Oct 11, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

jfy133 added 2 commits October 11, 2024 12:57

Apply suggestions from code review

78edf25

Add previouysly undocumented output files to docs

3c3b46a

jfy133 self-requested a review October 11, 2024 11:04

[automated] Fix code linting

7c7e954

jfy133 approved these changes Oct 11, 2024

View reviewed changes

jfy133 merged commit db33efd into nf-core:dev Oct 11, 2024
14 of 15 checks passed

jfy133 mentioned this pull request Oct 27, 2024

Bin QC Improvements #707

Open

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move long read preprocessing into a subworkflow #674

Move long read preprocessing into a subworkflow #674

muabnezor commented Sep 20, 2024 •

edited

Loading

github-actions bot commented Sep 20, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

jfy133 commented Sep 20, 2024

muabnezor commented Sep 23, 2024

jfy133 left a comment

muabnezor commented Sep 24, 2024

jfy133 commented Sep 24, 2024

muabnezor commented Sep 24, 2024 •

edited

Loading

jfy133 commented Sep 25, 2024 •

edited

Loading

muabnezor commented Sep 25, 2024

jfy133 commented Sep 25, 2024

muabnezor commented Sep 30, 2024

muabnezor commented Oct 4, 2024

muabnezor commented Oct 11, 2024

jfy133 commented Oct 11, 2024

jfy133 left a comment

jfy133 Oct 11, 2024

jfy133 Oct 11, 2024

jfy133 commented Oct 11, 2024

jfy133 left a comment

jfy133 commented Oct 11, 2024

		### `Changed`

		- [#674](https://github.com/nf-core/mag/pull/674/files) - Changed to porechop-abi as default adapter trimming tool for long reads. User can still use porechop if prefered.

		@@ -116,6 +116,8 @@

		- [Porechop](https://github.com/rrwick/Porechop)

		- [Porechop-abi](https://github.com/bonsai-team/Porechop_ABI)

Move long read preprocessing into a subworkflow #674

Move long read preprocessing into a subworkflow #674

Conversation

muabnezor commented Sep 20, 2024 • edited Loading

PR checklist

github-actions bot commented Sep 20, 2024 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

jfy133 commented Sep 20, 2024

muabnezor commented Sep 23, 2024

jfy133 left a comment

Choose a reason for hiding this comment

muabnezor commented Sep 24, 2024

jfy133 commented Sep 24, 2024

muabnezor commented Sep 24, 2024 • edited Loading

jfy133 commented Sep 25, 2024 • edited Loading

muabnezor commented Sep 25, 2024

jfy133 commented Sep 25, 2024

muabnezor commented Sep 30, 2024

muabnezor commented Oct 4, 2024

muabnezor commented Oct 11, 2024

jfy133 commented Oct 11, 2024

jfy133 left a comment

Choose a reason for hiding this comment

jfy133 Oct 11, 2024

Choose a reason for hiding this comment

jfy133 Oct 11, 2024

Choose a reason for hiding this comment

jfy133 commented Oct 11, 2024

jfy133 left a comment

Choose a reason for hiding this comment

jfy133 commented Oct 11, 2024

muabnezor commented Sep 20, 2024 •

edited

Loading

github-actions bot commented Sep 20, 2024 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️

muabnezor commented Sep 24, 2024 •

edited

Loading

jfy133 commented Sep 25, 2024 •

edited

Loading