Collect live plots for experiments running in the queue

[dberenbaum]

Related: iterative/vscode-dvc#3676

cc @mattseddon

VS Code needs to show live updates to plots for experiments running in the queue. They should be able to use dvc plots diff or some other interface to pass the revisions they want to see and the order in which they want the spec to be merged (see this comment for the order in which VS Code merges) and get all the info they need to generate plots (datapoints and specs) without having to navigate to each queued temp dir to collect.

Nice to have:

• Minimize the refactoring needed by VS Code (they currently use dvc plots diff exp1 exp2 ... -o .dvc/tmp/plots --split --json)
• Enable dvc plots diff exp1 exp2 to similarly show the live results of those experiments running in the queue.

[dberenbaum]

The y-axis value for the plots is set to dvc_inferred_y_value if there are multiple y fields in the spec. That field depends on the spec, so I think if we want to consider the data/datapoints static, we would need to append this field when we merge revisions and determine the final spec.

[pmrowla]

Just to clarify, is the only requirement here just that dvc plots diff exp-name ... includes the live/current datapoints if exp-name is a running experiment (and that all other existing dvc plots diff behavior regarding config merging and so on is to be left alone for now)?

@mattseddon @dberenbaum

[dberenbaum]

Correct, it looks like the current merging behavior in DVC and VS Code is good enough for now.