Internal queues: time window as well as active task limit?

[hjoliver]

Internal queues currently release tasks according to the limit setting, which limits the number of tasks that can be active at once. Would it be useful to give queues an optional time window too, to allow (say) long-running suites that only submit jobs overnight when the host system is not busy?

Ref: https://groups.google.com/forum/#!topic/cylc/jB8PVSKfEW4

[oliver-sanders]

This seems like a good idea, should be fairly simple to implement.

To mix this in with the larger issue of resource scheduling, possible queue configurations:

• Time window
  • Should this take into account execution time limit if provided (submission time window and execution time window)?
  • e.g. to run overnight when a machine is quiet
  • e.g. to run when electricity is cheap (may be an irregular time window e.g. if prices differ at the weekend)
• Queue length
  • For particular platforms or platform groups using the same batch scheduler
  • e.g. to submit jobs when resource idle - PT0M
• Available resource
  • e.g. for background submission
• User defined command / function

[hjoliver]

@oliver-sanders - are you suggesting dynamic evaluation of queue release criteria? (I think you are, but just checking!)

[hjoliver]

(good ideas above, by the way)

[oliver-sanders]

More outlining than suggesting but it would be required, yes.

[trwhitcomb]

We ran into this a while back, and this is definitely a useful thing to have. However, when looking into this (particularly when interfacing with batch schedulers), it seemed like Cylc queues weren't necessarily the right place to solve this, but the scheduling system was, largely because the problem of preemption is linked to the job start time and duration rather than the submit time (which is the only thing that Cylc can control).

For the case that's raised, one could imagine if the desired behavior was on a system with PBS, the implementation could submit the task right away, but with additional directives that specify the time at which the job would be eligible to execute, even though it would be submitted directly to the queue.

[hjoliver]

@trwhitcomb - I agree, the batch system is the proper place to handle this sort of thing. But even so, Cylc could help here, e.g. if you don't have sufficient control over batch system configuration, or are using background jobs. Not a high priority at the moment though.