Everest not running on compute cluster

[oyvindeide]

What happened? (You can include a screenshot if it helps explain)

Running Everest does not seem to work when running on compute cluster. The server starts on a node, but no jobs are submitted. After I while an error showed up in the terminal:

ERROR:everest_main:Everest run failed with: Traceback (most recent call last):
  File "/path/to/lib64/python3.11/site-packages/everest/detached/jobs/everserver.py", line 327, in main
    status, message = _get_optimization_status(run_model.exit_code, shared_data)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/lib64/python3.11/site-packages/everest/detached/jobs/everserver.py", line 391, in _get_optimization_status
    messages = _failed_realizations_messages(shared_data)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/lib64/python3.11/site-packages/everest/detached/jobs/everserver.py", line 401, in _failed_realizations_messages
    failed = shared_data[SIM_PROGRESS_ENDPOINT]["status"]["failed"]
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
KeyError: 'status'

Traceback (most recent call last):
  File "/path/to/lib64/python3.11/site-packages/everest/detached/jobs/everserver.py", line 327, in main
    status, message = _get_optimization_status(run_model.exit_code, shared_data)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/lib64/python3.11/site-packages/everest/detached/jobs/everserver.py", line 391, in _get_optimization_status
    messages = _failed_realizations_messages(shared_data)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/lib64/python3.11/site-packages/everest/detached/jobs/everserver.py", line 401, in _failed_realizations_messages
    failed = shared_data[SIM_PROGRESS_ENDPOINT]["status"]["failed"]
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
KeyError: 'status'

This was on lsf, but was also reported for slurm. Not sure if the error message was the same for slurm. Everest could run on the same config with local.

What did you expect to happen?

No response

steps to reproduce

Run math_func with:

simulator:
  queue_system: lsf

Environment where bug has been observed

• python 3.11
• python 3.12
• macosx
• rhel7
• rhel8
• local queue
• lsf queue
• slurm queue
• openPBS queue

[verveerpj]

I observed the same problem recently on our SLURM cluster and did some investigation. I here is what I already reported on slack:

1. When running everest on our slurm cluster, the detached server runs on a node, but no forward models are submitted, and everest terminates. There is an error, but that is just the database failing to read results. Nothing pertaining job submission in errors or logs.
2. When running everserver instead on the master node (so no detached server, no submission to the queue), the optimization runs successfully, submitting the forward models to the cluster. Hence, the problem is not the detachted server.
3. When running the succesfull everserver via sbatch , i.e. submitting it to a node, or by running it from an interactive shell running on a node (via srun), the optimization fails again, no errors.
4. I tested if our cluster can submit jobs from a node via a simple script, which worked fine.
5. I reverted the changes that were make recently to run an activate script on the cluster: no effect, still going wrong.
6. I reverted the recent PR (Use ensemble evaluator in everest in place of BatchSimulator #9342) that switched from the old job submission code to using the newer code in the base run model: things worked again.
7. This was all tested on a SLURM cluster and since the old code was working fine it seems the driver is working.