Memory Specification Appears Broken for SLURM

[HarryMWinters]

Environment:

• aws-parallelcluster-2.6.0

[aws]
aws_region_name = us-east-1

[cluster default]
key_name = <ENTER-VALUE>
vpc_settings = vpc-public-east-1
master_instance_type = m4.2xlarge
compute_instance_type = m4.xlarge
fsx_settings = fs
# must be at least as big as the size on the AMI, which was created at 100GB
master_root_volume_size = 100
compute_root_volume_size = 100
scheduler = slurm
initial_queue_size = 1
maintain_initial_size = true
max_queue_size = 15000

# alinux2, py3.6.8, custom AMI
custom_ami=ami-0ac26e5989580566f

[vpc vpc-public-west-2]
vpc_id = vpc-<our-vpc-id>
master_subnet_id = subnet-<our-subnet>

[vpc vpc-public-east-1]
vpc_id = vpc-<a-different-subnet>

[global]
cluster_template = default
update_check = true
# if not set to false, AWS (bug) calls `ec2 run-instance --dryrun` to validate
# the ec2 account limits...but since pcluster uses an autoscaling group (ASG), 
# no point in checking these limits
sanity_check = false

[aliases]
ssh = ssh {CFN_USER}@{MASTER_IP} {ARGS}

[fsx fs]
shared_dir = /shared
storage_capacity = 3600
# sunday at midnight
weekly_maintenance_start_time = 7:00:00

Bug description and how to reproduce:
When specifying the memory option on the cluster the task fail to submit and the following error returns:

sbatch: error: Memory specification can not be satisfied
sbatch: error: Batch job submission failed: Requested node configuration is not available

To recreate make a sh file:

#!/bin/bash 
#SBATCH -p compute 
#SBATCH -n 1 
#SBATCH -N 1 
#SBATCH --mem=64g 
#SBATCH -o log.txt
sleep 30
echo Hello World

Then sbatch test.sh

Additional context:
This might be a duplicate of #1517 . If so, is supporting scheduling with memory options with slurm planned?

[rexcsn]

Hi @HarryMWinters,

I would consider this a duplicate of #1517.
Please experiment with the workaround outlined there and see if it will solve the issue.

Unfortunately, there is currently no plan to support memory option with slurm natively.

Please let us know if you have open question/concern. Thank you!

[jerowe]

The workaround mentioned in #1517 worked for me.

This wasn't an issue in parallelcluster 2.4.1. I'm not sure what changed. I compared the slurm.conf files from each and couldn't find anything that would point to this.

[lukeseawalker]

Hi,
Slurm was upgraded from v18.08 (ParallelCluster 2.4.1) to v19.05 (ParallelCluster 2.5.0)

We believe that Slurm 18 used to be able to setup memory specification automatically for a node, but since Slurm 19 user is required to specify memory manually.

The Slurm documentation reports that default value for RealMemory is 1 (https://slurm.schedmd.com/slurm.conf.html):

RealMemory
Size of real memory on the node in megabytes (e.g. "2048"). The default value is 1. Lowering RealMemory with the goal of setting aside some amount for the OS and not available for job allocations will not work as intended if Memory is not set as a consumable resource in SelectTypeParameters. So one of the *_Memory options need to be enabled for that goal to be accomplished. Also see MemSpecLimit.

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[tdudgeon]

I don't think this should be closed - the ability to specify memory with Slurm without ugly workarounds should be fixed

[demartinofra]

request for enhancement is being tracked in #2198