Support a new display status to distinguish "QUEUED" vs. "RUNNING"

[mbookman]

For historical reasons (initial support from the Pipelines API v1alpha2), dstat provides no distinction between tasks that are QUEUED vs. actually RUNNING. Today they are all listed as RUNNING.

We should surface this distinction in places where the status is displayed. It can be misleading to new users when they submit a large number of tasks, it may appear that some are taking a very long time to run when they are actually queued and blocked by other running tasks that are consuming Compute Engine Quota (such as CPU, Persistent Disk, or In-Use IP Addresses).

[mbookman]

Until this is surfaced in dstat output, users can use one of:

• gcloud compute instances list
• Cloud Console

to see the number of concurrently running VMs.

[sjgosai]

Clarification question: Are jobs that are QUEUED due to resource quotas, but listed as RUNNING (i.e. Status: VM starting (awaiting worker checkin)) constrained by the same timeout restriction as jobs that are actually running?

[wnojopra]

Clarification question: Are jobs that are QUEUED due to resource quotas, but listed as RUNNING (i.e. Status: VM starting (awaiting worker checkin)) constrained by the same timeout restriction as jobs that are actually running?

Yes, the timeout includes the time spent waiting for a worker to be allocated. The default timeout is seven days which can be changed with the --timeout flag. See the provider specific parameters section in provider docs