How to disable automatic SLURM detection / signal handling?

[jbohnslav]

❓ Questions and Help

What is your question?

I'm running single-GPU jobs on a SLURM cluster. PyTorch Lightning uses environment variables to detect that I'm on SLURM, and automatically interrupts SIGTERM signals. However, when I'm debugging, I don't want the SIGTERM to be bypassed-- I need to know where the signal is originating.

I can't seem to tell PytorchLightning to not use the Slurm handler, because it's automatically detected using environment variables. Is there any way to not use PL's default SLURM connector / SIGTERM bypass function?

• OS: Linux
• Packaging pip
• Version 1.1.1

[tchaton]

Hey @williamFalcon. Any idea on this one ?

[awaelchli]

@jbohnslav a quick hack could be to delete the slurm env variables like so:

import os
del os.environ["SLURM_NTASKS"]
del os.environ["SLURM_JOB_NAME"]

at the beginning of the script, then lightning will not detect it as slurm.

Another way could be to grab the original signal handler before trainer init,
and then restore it on fit start (overwriting the one trainer has set):

import signal
# before training init
original_handler = signal.getsignal(signal.SIGTERM)

# in on_fit_start hook
signal.signal(signal.SIGTERM, original_handler)

(A while back I had a feature PR #3632 that added a configureable way to register signals)

[YannDubs]

I just had a similar issue of wanting to deactivate SLURM detection (I use another library to deal with that and pytorch lightning is only one small component of my code).

Given that wanting to deactivate SLURM detection seems a recurrent usecase (see also #6204 #6389 ) I think there should really just be a flag to the trainer (e.g. disable_hpc_detection as suggested by #6389 (comment) )

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!