[BUG] Cannot submit jobs from a GPU Node

[ziw-liu]

When submitting jobs on a GPU node, the generated batch job requests GRES that is not valid:

srun: error: Unable to create step for job 16413634: Invalid generic resource (gres) specification

Submitting the same job on a login node succeeds.

[ieivanov]

Here is the error I got with similar submision:

srun: fatal: SLURM_MEM_PER_CPU, SLURM_MEM_PER_GPU, and SLURM_MEM_PER_NODE are mutually exclusive.

[edyoshikun]

I haven't been able to reproduce this bug. I used a reservation done via

 sbatch --job-name=nomachine --constraint=nomachine --partition=interactive --mem-per-cpu=8G --cpus-per-task=16 --gpus=1 --time=5-0:00:00 --wrap "sleep 120h" --output=$HOME/logs/sbatch.out --nodelist=gpu-sm01-02

submitted jobs for deskew and reconstruction without a problem

[talonchandler]

@tayllatheodoro, can you add your notes here? You said that you found a workaround by commenting monitor_jobs?

@amitabhverma, @ieivanov gave me a verbal report that he ran the recOrder GUI reconstruction from a gpu node and found an error that he figured was related to this issue.

@amitabhverma can you remind which HPC node you've been testing with? A GPU node or a login node?

[amitabhverma]

@talonchandler I don't think my issue was the same and never encountered the OP error. I could not run the reconstruction via a login node but it runs fine via a GPU/nomachine node. I use the Bruno interactive to setup my session.