kallisto|bustools 0.28.2 update #294

gennadyFauna · 2024-01-22T21:57:40Z

Starting work on bumping up kb version to 0.28.2. The nucleus and lamanno workflows are obsolete as of 0.28.0, so the interface for generating count matrices needs to be updated.

PR checklist

Release v2.5.0

github-actions · 2024-01-23T07:41:55Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 600619f

+| ✅ 171 tests passed       |+
#| ❔   4 tests were ignored |#
!| ❗   3 tests had warnings |!

❗ Test warnings:

pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!

❔ Tests ignored:

files_exist - File is ignored: lib/Utils.groovy
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
template_strings - template_strings
schema_params - schema_params

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-scrnaseq_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-scrnaseq_logo_light.png
files_exist - File found: docs/images/nf-core-scrnaseq_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-scrnaseq_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/WorkflowScrnaseq.groovy
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 2.6.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.aligner= alevin
nextflow_config - Config default value correct: params.protocol= auto
nextflow_config - Config default value correct: params.igenomes_base= s3://ngi-igenomes/igenomes/
nextflow_config - Config default value correct: params.simpleaf_rlen= 91
nextflow_config - Config default value correct: params.star_feature= Gene
nextflow_config - Config default value correct: params.kb_workflow= standard
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.max_cpus= 16
nextflow_config - Config default value correct: params.max_memory= 128.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-scrnaseq_logo_light.png matches the template
files_unchanged - docs/images/nf-core-scrnaseq_logo_light.png matches the template
files_unchanged - docs/images/nf-core-scrnaseq_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
files_unchanged - pyproject.toml matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
readme - README Zenodo placeholder was replaced with DOI.
pipeline_name_conventions - Name adheres to nf-core convention
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: awstest.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - 'assets/multiqc_config.yml' contains report_section_order
multiqc_config - 'assets/multiqc_config.yml' contains export_plots
multiqc_config - 'assets/multiqc_config.yml' contains report_comment
multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
multiqc_config - 'assets/multiqc_config.yml' contains a matching 'report_comment'.
multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.13.1
Run at 2024-03-07 11:52:16

Patch release v2.5.1

gennadyFauna · 2024-01-24T19:25:50Z

I am working on testing the current version to have a baseline. There is a potential existing bug: using the kallisto pseudoaligner with the lamanno workflow works, but only if Nextflow builds a new reference. Specifying an existing reference using kallisto_gene_map (the t2g.txt file?) and kallisto_index (the kb_ref_out.idx file) does not work, and throws an error pointing at

scrnaseq/workflows/scrnaseq.nf

Line 147 in 90cb6a4

protocol_config['protocol'],

with the following explanation:

Missing process or function mix([DataflowStream[?], [DataflowGetPropertyExpression(value=null)]])

I suspect that this is because KALLISTOBUSTOOLS_COUNT takes in the cDNA/intron FASTA files in this line:

scrnaseq/subworkflows/local/kallisto_bustools.nf

Line 59 in 90cb6a4

t1c,

But I do not believe these variables (t1c and t2c) are ever defined if the index is provided:

scrnaseq/subworkflows/local/kallisto_bustools.nf

Line 43 in 90cb6a4

if (!kallisto_index) {

I assume fixing this bug would require changing the kallisto workflow parameters. Though it does look as though these are already present in the following line:

scrnaseq/modules/nf-core/kallistobustools/count/meta.yml

Line 35 in 90cb6a4

- t1c:

That is not really what this argument does – it just concatenates the gene_version and transcript_version fields to IDs.

The txp2gene file should really be generated by kallisto|bustools. The t2g.py script generates a 3-column rather than 7-column t2g, which seems to be incompatible with kb count.

This .collect() is incompatible with passing in an existing t2g, and superfluous given the .collect() after the kb ref call.

gennadyFauna · 2024-02-12T20:00:13Z

Fixed some issues in kallisto_bustools.nf: the way the code was set up made it impossible to pass in an existing kallisto reference. Removed t2g.py. This script is from 2017 and a few versions behind; it may be more reliable to just use the t2g.txt generated by kallisto.

In the test run, kb exits with a SIGKILL error: https://github.com/nf-core/scrnaseq/actions/runs/7876828275/job/21493304177#step:5:93. This is actually likely related to the binary issue in pachterlab/kallisto#411.

t1c and t2c are required to run nac and lamanno count workflows. Since they are already defined when running kb ref, they cannot be easily defined through ext.args when using an existing reference

scrnaseq does not actually use kallisto_gene_map anywhere. Changed kallisto icon from a salmon to a rainbow (there is no bear icon, but bustools uses the Pink Floyd-esque prism image, so it seems most appropriate)

grst

Hi @gennadyFauna,

thanks for working on this! I'd like to get in #291 before we merge this to ensure everything is properly tested.

You can request to be added to the nf-core github organization in the #github-invitations channel - then github actions should run for you without anyone approving every single time.

grst · 2024-02-13T07:33:11Z

modules/nf-core/kallistobustools/count/main.nf

-        'https://depot.galaxyproject.org/singularity/kb-python:0.27.2--pyhdfd78af_0' :
-        'biocontainers/kb-python:0.27.2--pyhdfd78af_0' }"


Once everything works, please update the modules in nf-core/modules and re-install them into the pipeline using nf-core modules update. We don't allow modules in the pipelines to diverge from the central version stored in modules.

Thank you, I have put in a request.
I appreciate the clarification regarding the nf-core/modules source – so I would need to make changes in https://github.com/nf-core/modules/tree/master/modules/nf-core/kallistobustools, then pull them in.

If I understand correctly, these module essentially have the input/output conventions and a link to the docker image, which does not actually work.
Is there any way to rebuild the binary for the kallisto dependency?

Exactly. Whenever possible, we try to use the docker images provided by Biocontainers, which are derived from the bioconda recipe. So if the container is broken, it would be ideal to make it work there. If this is not possible for whatever reason, we can also use custom docker images in nf-core/modules.

To just rebuild, you can make a PR to bioconda recipes that increases the build number: https://github.com/bioconda/bioconda-recipes/blob/2094e1c611eedfe500a145bd442640f0440f68a5/recipes/kb-python/meta.yaml#L14

The bioconda recipe has been fixed. I have opened a PR at nf-core/modules#4981; this should be complete relatively soon. In the meantime, I have updated the tests and adjusted the downstream mtx operations.

@grst the nf-core module PR is merged and I have updated the kb modules in the PR branch. It still fails linting due to other files – not sure what to do; would love any suggestions. Running nf-core lint locally does not give any failures.

Maybe it is possible to somehow inject them with ext.args, but this is a fairly standard pipeline option, and should probably be exposed.

this is not a real snapshot

mtx should take in nascent/ambiguous/mature matrices. The genes.names txt contains gene names, which are typically what is desired (e.g. cellranger has names rather than IDs)

grst · 2024-03-04T19:10:47Z

@nf-core-bot fix linting

grst

LGTM, thanks for working on this!

grst · 2024-03-05T07:19:32Z

Don't worry about the linting failures. This is related to an update in the nf-core template that has not been yet integrated into the scrnaseq pipeline.

grst · 2024-03-07T11:10:40Z

@nf-core-bot fix linting

grst and others added 2 commits January 10, 2024 07:13

Merge pull request nf-core#283 from nf-core/dev

e03cd46

Release v2.5.0

first commit: bump up kb versions

19132ff

Merge pull request nf-core#293 from nf-core/dev

90cb6a4

Patch release v2.5.1

gennadyFauna added 7 commits January 24, 2024 13:17

Merge branch 'nf-core:master' into kb_update

9e176ac

renaming parameters

4ef499e

Memory arg might need to be before fastq list?

fb6db73

Update main.nf

3ccd083

Removed t2g versions call

45e0b19

That is not really what this argument does – it just concatenates the gene_version and transcript_version fields to IDs.

removing t2g.py

50d3948

The txp2gene file should really be generated by kallisto|bustools. The t2g.py script generates a 3-column rather than 7-column t2g, which seems to be incompatible with kb count.

Remove txp2gene.collect() call

b0d7c5d

This .collect() is incompatible with passing in an existing t2g, and superfluous given the .collect() after the kb ref call.

gennadyFauna added 3 commits February 12, 2024 13:21

add t1c/t2c parameters to schema

8630ef9

t1c and t2c are required to run nac and lamanno count workflows. Since they are already defined when running kb ref, they cannot be easily defined through ext.args when using an existing reference

Fixed kallisto_gene_map documentation

e503e21

scrnaseq does not actually use kallisto_gene_map anywhere. Changed kallisto icon from a salmon to a rainbow (there is no bear icon, but bustools uses the Pink Floyd-esque prism image, so it seems most appropriate)

Update nextflow_schema.json

752d051

grst reviewed Feb 13, 2024

View reviewed changes

gennadyFauna and others added 13 commits February 13, 2024 10:03

de-duplicating txp2gene args (lint complains)

4d90d60

Update nextflow_schema.json

efb1487

txp2gene has to be moved if it's shared

5ffdc88

adding explicit t1c and t2c arguments for kb

38344a5

Maybe it is possible to somehow inject them with ext.args, but this is a fairly standard pipeline option, and should probably be exposed.

Merge branch 'dev' into kb_update

336d5f4

harmonizing with PR while that's in progress

bf27685

Update nextflow_schema.json

ac7e7cd

Update main.nf

8c4b872

Update kallisto_bustools.nf

8db2f95

Update kallisto_bustools.nf

a1fdcda

docker container update

2aca75b

manually updated md5s

0f56093

this is not a real snapshot

Update main_pipeline_kallisto.test.snap

aae5fc5

gennadyFauna and others added 7 commits February 27, 2024 13:01

update mtx construction

740a5dc

mtx should take in nascent/ambiguous/mature matrices. The genes.names txt contains gene names, which are typically what is desired (e.g. cellranger has names rather than IDs)

update seurat too

e7f5994

this must have gotten fixed a few kb versions ago

25ecada

...and the R changes have to be updated in md5

072d187

correcting var

e006cdf

features too

0187f8c

version bump

35dc973

[automated] Fix linting with Prettier

f12cfbb

grst approved these changes Mar 5, 2024

View reviewed changes

grst enabled auto-merge March 5, 2024 07:19

grst mentioned this pull request Mar 5, 2024

Fix issue 81, "call empty droplets" #301

Merged

Merge branch 'dev' into kb_update

73e958c

grst and others added 4 commits March 7, 2024 12:39

Delete .nf-test.log

fdda7a5

Update nextflow_schema.json

6b46540

Fix nextflow schema

904743e

Update changelog

600619f

grst merged commit ae5b28b into nf-core:dev Mar 7, 2024
10 checks passed

grst mentioned this pull request Mar 7, 2024

Bump up kb to 0.28.0 #280

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kallisto|bustools 0.28.2 update #294

kallisto|bustools 0.28.2 update #294

gennadyFauna commented Jan 22, 2024

github-actions bot commented Jan 23, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

gennadyFauna commented Jan 24, 2024 •

edited

Loading

gennadyFauna commented Feb 12, 2024 •

edited

Loading

grst left a comment

grst Feb 13, 2024

gennadyFauna Feb 13, 2024

grst Feb 13, 2024

gennadyFauna Feb 27, 2024

gennadyFauna Mar 4, 2024

grst commented Mar 4, 2024

grst left a comment

grst commented Mar 5, 2024

grst commented Mar 7, 2024

		'https://depot.galaxyproject.org/singularity/kb-python:0.27.2--pyhdfd78af_0' :
		'biocontainers/kb-python:0.27.2--pyhdfd78af_0' }"

kallisto|bustools 0.28.2 update #294

kallisto|bustools 0.28.2 update #294

Conversation

gennadyFauna commented Jan 22, 2024

PR checklist

github-actions bot commented Jan 23, 2024 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

gennadyFauna commented Jan 24, 2024 • edited Loading

gennadyFauna commented Feb 12, 2024 • edited Loading

grst left a comment

Choose a reason for hiding this comment

grst Feb 13, 2024

Choose a reason for hiding this comment

gennadyFauna Feb 13, 2024

Choose a reason for hiding this comment

grst Feb 13, 2024

Choose a reason for hiding this comment

gennadyFauna Feb 27, 2024

Choose a reason for hiding this comment

gennadyFauna Mar 4, 2024

Choose a reason for hiding this comment

grst commented Mar 4, 2024

grst left a comment

Choose a reason for hiding this comment

grst commented Mar 5, 2024

grst commented Mar 7, 2024

github-actions bot commented Jan 23, 2024 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️

gennadyFauna commented Jan 24, 2024 •

edited

Loading

gennadyFauna commented Feb 12, 2024 •

edited

Loading