Not all forward model steps reported as finished despite all realizations were succesfull

[xjules]

What happened? (You can include a screenshot if it helps explain)

When running big poly (on LOCAL QUEUE) with 500 and 400 realizations respectively, when setting max running to 500 in the first case all forward steps were reported as completed.

In the second case (400 reals) I set max running to 200 and there were missing events despite all realizations were labeled as successful.

What did you expect to happen?

All forward model steps should be labeled as finished in the gui.

steps to reproduce

Environment where bug has been observed

• python 3.11
• python 3.12
• macosx
• rhel7
• rhel8
• local queue
• lsf queue
• slurm queue
• openPBS queue

[xjules]

Update: was doing more experiments and it looks like MAX_RUNNING 200 is magic number 😄 As setting MAX_RUNNING 100, 300 and 400 didn't cause any issues, just 200 was.

[jonathan-eq]

This issue would be closed by #9432, but we should try to fix the underlying issue instead. If this occurs reliably, it would be way easier to actually find the root cause.