Removing queued experiments

[dberenbaum]

There are some features I'd like when working with queued experiments:

• Removal of an experiment from the queue (dvc exp remove [queued_experiment_hash]).
• Clearing the entire experiment queue (dvc exp remove --queue).

Also, I'm unclear what DVC: Queue Experiment from Existing does.

[mattseddon]

DVC: Queue Experiment from Existing is something you asked for 😅.

It should give users the ability to queue an experiment using an existing experiment as a base. They will be given the choice to select which parameters they want to vary and will then be given the original param value to work from for each of those parameters.

LMK if that doesn't make sense.

Got any suggestions for a better name?

[dberenbaum]

Okay, I get it now, thanks. It's from #1120 (comment).

Got any suggestions for a better name?

Ideas:

• DVC: Set Experiment Parameters
• DVC: Set Parameters and Queue
• DVC: Modify Existing Experiment

Not sure any of those are great, but I think the main focus should be on setting parameters rather than on queuing or on using existing experiments.

Feedback on that feature now that I understand it better:

• It's nice for modifying parameters without modifying the workspace.
• The interface for modifying parameters is a little clunky, but not sure if it's possible to do much better in VS Code. Something closer to a spreadsheet with the existing values filled in would be ideal.
• It's doing a lot (selecting an existing experiment, selecting parameters, modifying parameters, queuing a new experiment), and it's not totally clear what's happening until trying it out, which makes it a little scary and confusing. Maybe there's a way to either make the workflow more explicit up front, or to take out some steps (for example, always use the workspace experiment as the starting point).

[mattseddon]

@dberenbaum I addressed your last comment in #1456

Thought it would be best to add some comments/reasoning here:

Ideas:

• DVC: Set Experiment Parameters
• DVC: Set Parameters and Queue
• DVC: Modify Existing Experiment

Not sure any of those are great, but I think the main focus should be on setting parameters rather than on queuing or on using existing experiments.

I went with DVC: Set Experiment Params and Queue as we have only ever referred to Parameters as Params in the UI (if you think we should always use Parameters then please LMK).

• The interface for modifying parameters is a little clunky, but not sure if it's possible to do much better in VS Code. Something closer to a spreadsheet with the existing values filled in would be ideal.

This is the "native" experience. I am also not a massive fan but this is what is expected from VS Code extensions (and there are way more complicated flows out there). If we get a lot of complaints then I would consider changing it but I feel it is good enough for now.

• It's doing a lot (selecting an existing experiment, selecting parameters, modifying parameters, queuing a new experiment), and it's not totally clear what's happening until trying it out, which makes it a little scary and confusing. Maybe there's a way to either make the workflow more explicit up front, or to take out some steps (for example, always use the workspace experiment as the starting point).

I have altered a couple of the quick pick titles to try and make the flow more explicit. The user will now experience this:

1. User selects "DVC: Set Experiment Params and Queue" from the command palette.
2. User is shown a quick pick with "Select an Experiment to Use as a Base" as the title and the full list of experiments detailed below.
3. User is shown a second quick pick with "Select Param(s) to Vary" and the full list of params shown below.
4. User is shown a series of quick inputs (one for each of the selected params) with "Enter a Value for ${path}" as the title and the original value shown as a placeholder.

Shown on the screen:

Screen.Recording.2022-03-17.at.1.46.20.pm.mov

[dberenbaum]

That all looks good! It seems more straightforward, but I'm biased now that I've spent a lot of time looking at the feature.

My biggest fear for this feature is spending time on a feature that might not get used that much, so IMO you definitely shouldn't spend more time on it at this point.

[mattseddon]

Thanks, @dberenbaum.

My biggest fear for this feature is spending time on a feature that might not get used that much, so IMO you definitely shouldn't spend more time on it at this point.

We should be able to tell from the analytics whether or not it is used and we'll also be able to iterate on things as the mechanics of queueing changes.

Will close this now. Thanks again for your feedback.