HPC/Biowulf memory overallocation

[samarth8392]

Issue

I ran >100 whole exome seq samples sequenced at average 300-400x through the XAVIER pipeline and I received a warning from HPC/Biowulf about memory overallocation.

Coverage histogram from MultiQC

Message from HPC:

We have noticed that many of your jobs on Biowulf ALLOCATED
considerably MORE MEMORY than they used during a recent time period:

JOBS:         13345


Resources:    Needed: 15,608 GBh   Alloc: 232,901 GBh  Efficiency: 6.7%
From:         2024-04-12 10:02:08
Until:        2024-04-13 10:02:09

GBh = Gigabyte hours (i.e. GB * runtime in hours)

Please check on the resource usage of types of jobs you run regularly
and adjust memory allocations to reduce idle memory.

Memory that is allocated by a job but not used reduces cluster
efficiency and prevents the jobs of your NIH colleagues from using
this memory. In addition, allocating more resources than needed
will reduce the priority of your future jobs resulting in longer queue
wait times.

I looked at the runtime and max memory used for all the rules and noticed that all rules were only using 5-10GB of max memory and most memory was used by gatk_recal, LearnReadOrientationModel, and a few others.

Default memory:

rule	alloc_mem (gb)
fc_lane	64
fastq_screen	64
gatk_recal	72
bam_check	64
bam2fastq	48
haplotypecaller	48
samtools_flagstats	64
fastqc_bam	64
pileup_single	64
mutect2_single	64
varscan_single	64
vardict_single	64
mutect_single	64
somatic_mafs	64
LearnReadOrientationModel	64
somatic_merge_callers	64
mergegvcfs	48
genotype	96

Proposed solution

Reduce memory allocation for non-GATK and other rules that don't need a whole lot of memory.