Is it possible for SLURM auto submit to work on DP?

[philip30]

In my experience, it never works. I looked to the trainer code and saw that the code managing this works only on DDP.

def configure_slurm_ddp(self, num_gpu_nodes):
    self.is_slurm_managing_tasks = False

    ### !!HERE!!
    if self.use_ddp:
        self.num_requested_gpus = self.num_gpus * num_gpu_nodes
        self.num_slurm_tasks = 0
        try:
            self.num_slurm_tasks = int(os.environ['SLURM_NTASKS'])
            self.is_slurm_managing_tasks = self.num_slurm_tasks == self.num_requested_gpus

            # in interactive mode we don't manage tasks
            job_name = os.environ['SLURM_JOB_NAME']
            if job_name == 'bash':
                self.is_slurm_managing_tasks = False

        except Exception:
            # likely not on slurm, so set the slurm managed flag to false
            self.is_slurm_managing_tasks = False

However, sometimes we are not using the distributed computing on slurm (only DP). It would be nice to have the auto resubmit feature still working in this situation.

• OS: Linux
• Packaging conda
• Version 16

[williamFalcon]

good point. i think auto resubmit should work no matter how training is happening so long as it detects slurm.

mind submitting a PR?

[philip30]

I'll try to work on it!

[williamFalcon]

actually, lightning supports slurm no matter what backend you use...

    def register_slurm_signal_handlers(self):
        # see if we're using slurm (not interactive)
        on_slurm = False
        try:
            job_name = os.environ['SLURM_JOB_NAME']
            if job_name != 'bash':
                on_slurm = True
        except Exception as e:
            pass

        if on_slurm:
            log.info('Set SLURM handle signals.')
            signal.signal(signal.SIGUSR1, self.sig_handler)
            signal.signal(signal.SIGTERM, self.term_handler)