exp show: display running/queued state for experiments #6174

pmrowla · 2021-06-15T08:56:44Z

❗ I have followed the Contributing to DVC checklist.
📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

Will close #5965

For the DVC UI, this PR adds State column that will currently be Running, Queued or empty
- If the table has no running or queued experiments the column will not be displayed
- If State is Running, an additional Executor column will be displayed noting where the experiment is being run, currently this is limited to local (workspace) or local (background)
For --show-json (VSCode), experiment dicts now have a running key that will be true or false, and an executor key which will be an optional string executor name
Fixes bug where main repo would be locked during --temp or --queue/--run-all runs (related to exp show: Allow running while DVC is locked #5739) which prevented exp show from being usable

Regular experiments example:

Checkpoints example:

currently only includes PID and git fetch URL

pmrowla · 2021-06-15T11:31:35Z

Regarding docs, exp show usage examples need to be updated throughout the docs. I'm planning on adding a PR to display user-specified names for queued exps and will do a single docs PR w/the updated examples after that

efiop · 2021-06-15T11:50:52Z

dvc/command/experiments.py

+        if exp.get("running"):
+            state = "Run"
+        elif exp.get("queued"):
+            state = "Queue"


Should it be Running/Queued? Just feels a bit odd. Maybe this has been discussed somewhere. Just wondering.

Running/Queued makes more sense but I'm not really sure what's best here given that table width is a concern

Agree that Running/Queued is better, unless someone can come up with something shorter that's still clear. We could include a --no-status option to keep it narrower. Experiment and Created take up a lot of space, so we could also look into ways to narrow those if needed.

dberenbaum

It looks good to me for non-checkpoint experiments.

For checkpoints experiments, it's a little confusing because:

The Run status shows next to the already completed checkpoint. Maybe we should be showing the experiment name on a separate line from individual checkpoints? That would also narrow the first column.
For workspace runs, the status is Run for both the workspace row and the experiment row (which only exists after the first checkpoint). So it can look like there are two experiments running, one of which got added after the first checkpoint. Is there a way to indicate that the workspace is the same as one of the experiments, at least while the experiment is running?

dberenbaum · 2021-06-16T14:35:05Z

dvc/command/experiments.py

+        if exp.get("running"):
+            state = "Run"
+        elif exp.get("queued"):
+            state = "Queue"


Agree that Running/Queued is better, unless someone can come up with something shorter that's still clear. We could include a --no-status option to keep it narrower. Experiment and Created take up a lot of space, so we could also look into ways to narrow those if needed.

pmrowla · 2021-06-17T00:31:31Z

@dberenbaum

The Run status shows next to the already completed checkpoint.

This makes sense to me because when you are resuming/continuing a checkpoint run, you are starting from the last completed checkpoint (rather than starting from the queued row that only has params and no metrics/outputs)

Maybe we should be showing the experiment name on a separate line from individual checkpoints? That would also narrow the first column.

What would we do here for regular experiments? When #6050 is done we will have names in the rows for these queued/running experiments, and will have the same issue. I don't think it makes sense to take up 2 rows for a single experiment.

For workspace runs, the status is Run for both the workspace row and the experiment row (which only exists after the first checkpoint). So it can look like there are two experiments running, one of which got added after the first checkpoint. Is there a way to indicate that the workspace is the same as one of the experiments, at least while the experiment is running?

It seems like what we are going to eventually need is to display information about where an experiment is running (for remote executors) but I'm not sure where to indicate that right now. We could omit Run entirely from the workspace row, and display the state for experiment rows as something like Running (workspace)/ Running (background) but this is going to keep making the table wider

dberenbaum · 2021-06-17T18:46:39Z

This makes sense to me because when you are resuming/continuing a checkpoint run, you are starting from the last completed checkpoint (rather than starting from the queued row that only has params and no metrics/outputs)

Extracted to #6194.

It seems like what we are going to eventually need is to display information about where an experiment is running (for remote executors) but I'm not sure where to indicate that right now. We could omit Run entirely from the workspace row, and display the state for experiment rows as something like Running (workspace)/ Running (background) but this is going to keep making the table wider

Yes, we need a column like executor or location or something. Why not add it now? Maybe we can autohide it and the state column when it's empty (nothing is running)?

skshetry · 2021-06-18T05:23:42Z

dvc/command/experiments.py

@@ -304,7 +309,7 @@ def experiments_table(

    from dvc.compare import TabularData

-    headers = ["Experiment", "rev", "queued", "typ", "Created", "parent"]
+    headers = ["Experiment", "rev", "typ", "Created", "parent", "State"]


I am a bit confused about what it should be: State or Status. Both seem to make sense to me. Is State most appropriate here?

I think either term works here. I just used state because status has some other connotations in DVC

pmrowla · 2021-06-29T06:52:40Z

@dberenbaum I've added a separate executor column that will display where the experiment is running, and both state and executor are only shown if there is actually any running/queued experiments in the table. (see the 2 updated screen recordings in the top post)

dvc/command/experiments.py

skshetry · 2021-06-29T14:27:52Z

I think that we are mixing two things in the exp show - list and metrics/params show (which is collapsed in non-pager mode). Would be great to revisit the table formats for 3.0 (especially considering remote executors might add more columns here). cc @dberenbaum

dberenbaum · 2021-06-30T14:06:34Z

I think that we are mixing two things in the exp show - list and metrics/params show (which is collapsed in non-pager mode). Would be great to revisit the table formats for 3.0 (especially considering remote executors might add more columns here). cc @dberenbaum

Do you mean that columns like status and executor should be shown in a separate command and not in dvc exp show?

dberenbaum · 2021-06-30T14:54:52Z

@pmrowla Looks great!

One very minor comment: can we shorten local (workspace) and local (background) text to just workspace and background or temp (since that's the flag name) or similar? local seems redundant now, and even when we have remote executors, we can specify them by name.

dberenbaum · 2021-06-30T14:57:15Z

dvc/command/experiments.py

@@ -381,16 +395,22 @@ def show_experiments(
    if no_timestamp:
        td.drop("Created")

+    for col in ("State", "Executor"):


Is it possible for other columns to be empty? If so, would we want to hide them, too?

There shouldn't be any other columns in the table right now that can be empty

pmrowla added 8 commits June 15, 2021 16:48

exp: allow fetching from arbitrary executor by git URL

bd27002

exp: write info for running executors to .dvc/tmp/exp/run

d89cc6e

currently only includes PID and git fetch URL

exp: refactor stash/queue cleanup

4b3b1a4

exp: add ability to collect dict of currently running experiments

daf3f7c

exp show: display run/queue state in the table

8625ec8

exp run: unlock main repo while running tmpdir executors

1fbcdf1

add tests for new exp show behavior

5851343

fix workspace checkpoint run handling

3605759

pmrowla self-assigned this Jun 15, 2021

pmrowla added A: experiments Related to dvc exp enhancement Enhances DVC ui user interface / interaction labels Jun 15, 2021

fix issue where running stash revs were shown as queued

4e72659

pmrowla requested a review from dberenbaum June 15, 2021 11:30

pmrowla changed the title ~~[WIP] exp show: display running/queued state for experiments~~ exp show: display running/queued state for experiments Jun 15, 2021

pmrowla marked this pull request as ready for review June 15, 2021 11:31

pmrowla mentioned this pull request Jun 15, 2021

exp show: Allow running while DVC is locked #5739

Closed

efiop reviewed Jun 15, 2021

View reviewed changes

dberenbaum reviewed Jun 16, 2021

View reviewed changes

use "running"/"queued"

5fae2e0

dberenbaum mentioned this pull request Jun 17, 2021

exp show: checkpoint experiment summary #6194

Closed

skshetry reviewed Jun 18, 2021

View reviewed changes

add executor (location) column

3517b31

pmrowla requested a review from a team as a code owner June 29, 2021 06:29

pmrowla requested a review from dberenbaum June 29, 2021 06:51

skshetry reviewed Jun 29, 2021

View reviewed changes

dvc/command/experiments.py Outdated Show resolved Hide resolved

pmrowla added 2 commits June 30, 2021 17:09

compare: add TabularData.is_empty

97b6872

use td.is_empty to drop state/executor

9e04b2e

skshetry mentioned this pull request Jun 30, 2021

push params in the experiments table to the right #6254

Merged

dberenbaum reviewed Jun 30, 2021

View reviewed changes

use "workspace" and "temp" as executor names

e856d2d

pmrowla requested a review from dberenbaum July 12, 2021 08:39

mattseddon mentioned this pull request Jul 12, 2021

Feed data into experiments runs view iterative/vscode-dvc#636

Merged

dberenbaum approved these changes Jul 12, 2021

View reviewed changes

pmrowla merged commit 1271db6 into iterative:master Jul 12, 2021

pmrowla deleted the exp-show-running branch July 12, 2021 15:20

This was referenced Jul 13, 2021

Extend experiments runs view to include running experiments iterative/vscode-dvc#637

Closed

Add running experiment(s) to runs view iterative/vscode-dvc#645

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

exp show: display running/queued state for experiments #6174

exp show: display running/queued state for experiments #6174

pmrowla commented Jun 15, 2021 •

edited

Loading

pmrowla commented Jun 15, 2021

efiop Jun 15, 2021

pmrowla Jun 15, 2021

dberenbaum Jun 16, 2021

dberenbaum left a comment

dberenbaum Jun 16, 2021

pmrowla commented Jun 17, 2021 •

edited

Loading

dberenbaum commented Jun 17, 2021

skshetry Jun 18, 2021 •

edited

Loading

pmrowla Jun 18, 2021

pmrowla commented Jun 29, 2021

skshetry commented Jun 29, 2021

dberenbaum commented Jun 30, 2021

dberenbaum commented Jun 30, 2021

dberenbaum Jun 30, 2021

pmrowla Jul 1, 2021

exp show: display running/queued state for experiments #6174

exp show: display running/queued state for experiments #6174

Conversation

pmrowla commented Jun 15, 2021 • edited Loading

pmrowla commented Jun 15, 2021

efiop Jun 15, 2021

Choose a reason for hiding this comment

pmrowla Jun 15, 2021

Choose a reason for hiding this comment

dberenbaum Jun 16, 2021

Choose a reason for hiding this comment

dberenbaum left a comment

Choose a reason for hiding this comment

dberenbaum Jun 16, 2021

Choose a reason for hiding this comment

pmrowla commented Jun 17, 2021 • edited Loading

dberenbaum commented Jun 17, 2021

skshetry Jun 18, 2021 • edited Loading

Choose a reason for hiding this comment

pmrowla Jun 18, 2021

Choose a reason for hiding this comment

pmrowla commented Jun 29, 2021

skshetry commented Jun 29, 2021

dberenbaum commented Jun 30, 2021

dberenbaum commented Jun 30, 2021

dberenbaum Jun 30, 2021

Choose a reason for hiding this comment

pmrowla Jul 1, 2021

Choose a reason for hiding this comment

pmrowla commented Jun 15, 2021 •

edited

Loading

pmrowla commented Jun 17, 2021 •

edited

Loading

skshetry Jun 18, 2021 •

edited

Loading