Pausing Job Submissions/Workers

[SWeav02]

I have a group of workers running that I set up to just stay running until I cancelled them. We're going to need to temporarily shut down our cluster for a scheduled power outage and I'd like to pause the workers or the cloud database queue after they finish their next workflow so no jobs are stopped in the middle. Is there a nice way of doing this?

[jacksund]

For now, you can pause a cluster but can't pause a worker.

If you submit a worker, it's going to run for as long as you told it too -- unless you kill the worker manually. So running simmate engine start-worker isn't the best idea. It's always better to do something like simmate engine start-worker --nitems-max 10 --close-on-empty-queue.

And to make it so you don't have to keep resubmitting shorter-lived workers, you should use the simmate engine start-cluster --type slurm --continuous command which will maintain a certain amount of short-lived workers submitting/running for you. Then whenever you need to pause things, you can kill the start-cluster process and the short-lived workers will wrap up and close on their own time.

(note: the start-cluster is in early testing as of v0.13.2 so there isn't really documentation on it yet)

[jacksund]

⚠️ only use this fix if ALL of your workers are stopped This is a fix if all of your workers crash.

Since you can't pause the workers there, you should actually just let the workers run until the power outage happens. When warwulf shuts down, the running jobs/workers won't ever get the chance to tell the database that they failed. So when you check the database, they'll still be in the R (running) state -- aka they'll become zombies. So when warwulf comes back online, you can grab all of the jobs in the R state (since you know ALL of them will be zombies) and switch them back to P (pending). Then resubmit your workers and you're good to go.

To switch the status of a run:

from simmate.database import connect
from simmate.engine.execution import WorkItem

# If ALL workers are stopped, anything still left with 
# the R state is actually a zombie.
zombies = WorkItem.objects.filter(
    status="R",
    tags=["exampletag"],  # optional extra filters
).all()

# convert each job's status back to Pending
for zombie in zombies:
    zombie.status = "P"
    zombie.save()

** In your case, if there are old jobs that you canceled the worker on mid-run, these will be flagged as zombies too. Just a heads up