Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: lower required threads for annotation rules #1528

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from

Conversation

mathiasbio
Copy link
Collaborator

@mathiasbio mathiasbio commented Feb 7, 2025

Description

The cluster has been quite clogged in this period in early 2025 and @Karl-Svard in prodbioinfo notified me about a job in balsamic cadd_annotate_somaticINDEL_research which was hardly using any threads or memory on the node on ganglia but it required 36 cores and booked up the whole node. It seems that this rule doesn't at all require this amount of resources and the cluster could be free:d up a bit if we lowered it.

I then looked at other similar rules and saw a few bcftools commands that was also run on the whole node. I don't think this should be necessary at all since the sizes of the VCFs are rarely even in the scale of 1gb.

On top of this the benchmark files specified in the rules also had the same name and should overwrite each other, meaning that we cannot track the benchmark of these rules.

Link issue: #1529

Changed

  • lowered threads for bcftools and CADD rules

Fixed

  • changed name of benchmark files for annotation rules to avoid name conflicts

Documentation

  • N/A
  • Updated Balsamic documentation to reflect the changes as needed for this PR.
    • [Document Name]

Tests

Feature Tests

Run WGS TN case and check the benchmark files for the rule to see if the threads are matching the memory requirement

  • The threads seem reasonable given the memory usage

Pipeline Integrity Tests

  • Report deliver (generation of the .hk file)
    • N/A
    • Verified
  • TGA T/O Workflow
    • N/A
    • Verified
  • TGA T/N Workflow
    • N/A
    • Verified
  • UMI T/O Workflow
    • N/A
    • Verified
  • UMI T/N Workflow
    • N/A
    • Verified
  • WGS T/O Workflow
    • N/A
    • Verified
  • WGS T/N Workflow
    • N/A
    • Verified
  • QC Workflow
    • N/A
    • Verified
  • PON Workflow
    • N/A
    • Verified

Clinical Genomics Stockholm

Documentation

  • Atlas documentation
    • N/A
    • Updated: [Link]
  • Web portal for Clinical Genomics
    • N/A
    • Updated: [Link]

Panel of Normal specific criteria

User Changes

  • N/A
  • This PR affects the output files or results.
    • User feedback is considered unnecessary because [Justification].
    • Affected users have been included in the development process and given a chance to provide feedback.

Infrastructure Changes

  • Stored files in Housekeeper
    • N/A
    • Updated: [Link]
  • CG (CLI and delivered/uploaded files)
    • N/A
    • Updated: [Link]
  • Servers (configuration files on Hasta)
    • N/A
    • Updated: [Link]
  • Scout interface
    • N/A
    • Updated: [Link]

Validation criteria

Validation criteria to be added to validation report PR: [LINK-TO-VALIDATION-REPORT-PR from the validations repository]

Version specific criteria

  • Text here or N/A

Important

One of the below checkboxes for validation need to be checked

  • Added version specific validation criteria to validation report
  • Changes validated in standard sections: [validation-section]
  • Validation criteria not necessary

Checklist

Important

Ensure that all checkboxes below are ticked before merging.

For Developers

  • PR Description
    • Provided a comprehensive description of the PR.
    • Linked relevant user stories or issues to the PR.
  • Documentation
    • Verified and updated documentation if necessary.
  • Validation criteria
    • Completed the validation criteria section of the template.
  • Tests
    • Described and tested the functionality addressed in the PR.
    • Ensured integration of the new code with existing workflows.
    • Confirmed that meaningful unit tests were added for the changes introduced.
    • Checked that the PR has successfully passed all relevant code smells and coverage checks.
  • Review
    • Addressed and resolved all the feedback provided during the code review process.
    • Obtained final approval from designated reviewers.

For Reviewers

  • Code
    • Code implements the intended features or fixes the reported issue.
    • Code follows the project's coding standards and style guide.
  • Documentation
    • Pipeline changes are well-documented in the CHANGELOG and relevant documentation.
  • Validation criteria
    • The author has completed the validation criteria section of the template
  • Tests
    • The author provided a description of their manual testing, including consideration of edge cases and boundary
      conditions where applicable, with satisfactory results.
  • Review
    • Confirmed that the developer has addressed all the comments during the code review.

@mathiasbio mathiasbio changed the base branch from master to develop February 7, 2025 15:01
Copy link

codecov bot commented Feb 7, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.50%. Comparing base (7d529e6) to head (4fb80e7).
Report is 39 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1528      +/-   ##
===========================================
+ Coverage    99.48%   99.50%   +0.02%     
===========================================
  Files           40       40              
  Lines         1932     2036     +104     
===========================================
+ Hits          1922     2026     +104     
  Misses          10       10              
Flag Coverage Δ
unittests 99.50% <ø> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mathiasbio mathiasbio linked an issue Feb 7, 2025 that may be closed by this pull request
3 tasks
@mathiasbio mathiasbio self-assigned this Feb 7, 2025
@mathiasbio mathiasbio added this to the Release 17 milestone Feb 7, 2025
Copy link

sonarqubecloud bot commented Feb 7, 2025

@mathiasbio mathiasbio marked this pull request as ready for review February 7, 2025 15:45
@mathiasbio mathiasbio requested a review from a team as a code owner February 7, 2025 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

[Maintenance] update threads and benchmark files for annotation rules
1 participant