Story: Experiments workflow

[shcheklein]

Motivation

Ability to run/stop/resume/queue experiments as reflected by the dvc exp is considered one of the key features we'd like to release as part of the initial version. Motivation behind the corresponding DVC functionality can be found in docs or briefly in this blog.

To play with the experiments online: https://katacoda.com/dvc/courses/get-started/experiments

Tickets

New tickets and scope might come as we have some initial design input from @yalozhkin .

• Add extension Commands:

  • Stop an experiment(s) Experiments: Add an extension Command to stop a running experiment #230
  • Apply, branch, remove Experiments: Add Extension Commands for apply, branch, and remove #227
  • Queue, run all, gc Experiments: add extension Commands for gc and exp --queue/--run-all #229
  • exp run --reset Add "Run and Reset Experiment" Command #304

• Add UI/UX for these actions:

  • Apply, branch, remove Experiments: UI for apply, branch, remove an experiment #228 Add mechanism for launching per-row actions from the table webview #1267
  • Stop Experiments: UI for stopping experiments #231
  • Reset: Experiments: Reset a partial experiment #233

• Checkpoint experiments handling:

  • Detect checkpoint (partial) experiment Experiment: Detect and indicate a partial experiment #232
  • Reset (UX and command): Experiments: Reset a partial experiment #233

Details

As we discussed we start the following scope:

• Run an experiment in the workspace - this is default and it is done to some extent. But might require changes from the design perspective.

• Queue experiments run --queue and run them all --run-all

• an action to stop and experiment(s) by sending a SIGTERM

• Reset to run from "scratch": run --reset

• Apply (get) the experiment - apply

• Branch (get as a branch) the experiment - branch

• Remove an experiment(s) - remove

Only after that when we have more clarity, probably after the initial release we can do "remote" commands

• push, pull, list

---

Design

The running experiments workflow includes:

• The Runs panel with toolbar
• The Experiments table with toolbar

A running experiment preview:

More details and states in the Figma spec.

cc @yalozhkin

[rogermparent]

We should now have all the commands necessary for the first implementation of manipulating local experiments

• exp run
  • --queue
  • --run-all
  • --reset
• exp remove
• exp gc
  • -w
  • -a
  • -T
  • --queued
• exp apply
• exp branch
• exp remove
• Stop Experiment (not a DVC command, but still a VSCode Command)

From here, we'll want to implement these commands in more friendly places of the UI. All the functionality is in, but different minor changes may be required depending on where in the VSCode UI we stick these commands.

[shcheklein]

@rogermparent excellent, thanks. I'll start reviewing this.

[yalozhkin]

@mattseddon @rogermparent @shcheklein updated the issue/spec.

[mattseddon]

@shcheklein I think we can close this one as we are tracking the remaining point under another ticket WDYT?

[shcheklein]

If you don't mind, guys, let's keep it. This is useful for me to track the product progress (by filtering the Story label). It's not precise, but it feels good to keep for me around for now as a reference what has been done there, what else is left.

[mattseddon]

First iteration done.