Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update SPAdes module to official nf-core and version bump #666

Merged
merged 26 commits into from
Oct 22, 2024
Merged

Update SPAdes module to official nf-core and version bump #666

merged 26 commits into from
Oct 22, 2024

Conversation

jfy133
Copy link
Member

@jfy133 jfy133 commented Sep 9, 2024 •
edited
Loading

TODO:

⚠️ JAMES: MAKE SURE TO CHECK AFTER TEMPLATE MERGE TO DOUBLE CHECK THE CHEKC_MEGAHIT_CPUS ETC FUNCTIONS ⚠️

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • If necessary, also make a PR on the nf-core/mag branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@jfy133 jfy133 linked an issue Sep 9, 2024 that may be closed by this pull request
Update metaspades to 4.0.0 #640
Open
@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 9, 2024 •
edited
Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 338a552

+| ✅ 319 tests passed       |+
#| ❔   2 tests were ignored |#
!| ❗   5 tests had warnings |!

❗ Test warnings:

  • pipeline_todos - TODO string in main.nf: Remove this line if you don't need a FASTA file [TODO: try and test using for --host_fasta and --host_genome]
  • pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
  • pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
  • pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline

❔ Tests ignored:

✅ Tests passed:

Run details

  • nf-core/tools version 3.0.2
  • Run at 2024-10-22 09:44:17

@jfy133
Copy link
Member Author

jfy133 commented Sep 10, 2024

Need to check other downstreeam files to make sure if there is an issue, however it seems that the original SPADES(HYBIRD) modules exported the the main scaffolds as uncompressed for downstream.( even though it also exported a version as compressed).

I think I will need to add an additional GUNZIP module after the assembly for safety...

@jfy133
Copy link
Member Author

jfy133 commented Sep 12, 2024 •
edited
Loading

Added the GUNZIP module actually does somthing similar to pre-assemblies, but then local MEGAHIT module also exports uncompressed files... so I guess time to update this too

But I need to udpate the MEGAHIT module to support co-assemblies (basically reads.join()

And also test the fix cpus thing

@jfy133
Copy link
Member Author

jfy133 commented Sep 20, 2024

@nf-core-bot fix linting

@jfy133
Copy link
Member Author

jfy133 commented Sep 20, 2024

Final thing: reimplement the CPU fixing

@jfy133 jfy133 marked this pull request as draft September 20, 2024 14:00
@jfy133 jfy133 marked this pull request as ready for review October 3, 2024 14:43
skrakau
skrakau reviewed Oct 8, 2024
View reviewed changes
Copy link
Member

@skrakau skrakau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great to see :D

conf/base.config Outdated
cpus = { check_megahit_cpus (8, task.attempt ) }
memory = { check_max (40.GB * task.attempt, 'memory' ) }
time = { check_max (16.h * task.attempt, 'time' ) }
cpus = { params.megahit_fix_cpu_1 ? 1 : check_max(8 * task.attempt, 'cpus') }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does that work? I remember vaguely that it didn't work to access params here, that's why I think I shifted the corresponding functionality to nextflow.config ... but maybe that changed?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if yes, then this is course much cleaner :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should do if it's in a closure { }. That's why e.g. task.attempt works.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you test?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but I will double test again 😬

Copy link
Member Author

@jfy133 jfy133 Oct 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hrm wait, this can't work as is as check_max should be gone, do further testing!

@jfy133
Copy link
Member Author

jfy133 commented Oct 17, 2024

⚠️ JAMES: MAKE SURE TO CHECK AFTER TEMPLATE MERGE TO DOUBLE CHECK THE CHEKC_MEGAHIT_CPUS ETC FUNCTIONS ⚠️

@jfy133
Copy link
Member Author

jfy133 commented Oct 22, 2024

For future reference, it does work! In the base.config I set the memory of SPADES and MEGAHIT to 7.MB to force retries, and then fix the CPUs :

nextflow run ../main.nf -profile test,docker --outdir ./results --spades_fix_cpus 2 --megahit_fix_cpu_1

image

image

When fixing the CPUs, as above, you see in the screenshots above that memory retries but CPUs stay at the values fixed by the parameters (the CPUs in the test.config is 12).

Without specifying the fixed CPUs, retrying indeed increases the CPUs

nextflow run ../main.nf -profile test,docker --outdir ./results

image

@jfy133 jfy133 merged commit 42d444f into dev Oct 22, 2024
16 checks passed
@jfy133 jfy133 deleted the bump-metaspades branch October 22, 2024 10:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@skrakau skrakau skrakau approved these changes

@d4straub d4straub Awaiting requested review from d4straub

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Update metaspades to 4.0.0
2 participants
@jfy133 @skrakau