Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rough table implementation #4

Closed
rogermparent opened this issue Oct 15, 2020 · 9 comments
Closed

Rough table implementation #4

rogermparent opened this issue Oct 15, 2020 · 9 comments
Labels
priority-p2 Future feature, less priority for now question Further information is requested

Comments

@rogermparent
Copy link
Contributor

rogermparent commented Oct 15, 2020 •
edited
Loading

This is the current graphical representation of a planned webview

Webview image

From what I understand, this page shows a table and plots that both represent the output of dvc exp show --show-json.

Since --in/exclude-metrics and --in/exclude-params options exist, this view would likely want to emulate the behavior of configurable outputs and other features derived from them like sorting.
@rogermparent rogermparent added priority-p2 Future feature, less priority for now question Further information is requested webview labels Oct 15, 2020
@shcheklein
Copy link
Member

If there is no penalty, there's a case for always grabbing every field and always manipulating client-side.

@pmrowla could you please clarify what exactly does exp show --json returns by default - all commits within a branch? all experiments for every of those commits?

Also, what's the most convenient way to get all metrics/params? Is it the exclude flags without any arguments? We'll probably want that to be a special option that's always available.

@rogermparent not sure I follow, could you clarify please? W/o exclude option specified I think it should return everything. cc @pmrowla ...

@pmrowla
Copy link

pmrowla commented Oct 16, 2020

By default exp show will return all experiments derived from the current commit in the parent workspace, so if you have master checked out, and the tip commit is abc123, it would return any experiments which were generated from abc123.

Regarding params & metrics, by default they will all be returned. using the include/exclude options only filters out specific columns at the UI level, and does not actually save performance internally.

@rogermparent
Copy link
Contributor Author

rogermparent commented Oct 16, 2020 •
edited
Loading

Thanks! That's useful info going forward.

not sure I follow, could you clarify please? W/o exclude option specified I think it should return everything.

I believe you understood, I wasn't sure what the default behavior of dvc exp show was and made an incorrect inference from the docs.

@yalozhkin
Copy link

Hello, everyone! I have a question about running experiments. How user runs it? Will there be a CLI area in the VSCode or it happens in an external CLI?

@fabiosantoscode
Copy link

Hey, I'm just dropping by to say that looks sweet. Thanks.

Best of luck @rogermparent

@jorgeorpinel

This comment has been minimized.

Sign in to view
@rogermparent
Copy link
Contributor Author

rogermparent commented Oct 29, 2020 •
edited
Loading

Hey everyone! I've been working on ways to store, process, and display Experiment data, and I believe I've found the way to go.

First, we start with the output of dvc exp show --show-json, which outputs experiments in a big nested JSON format. While this isn't a bad way to share data, we can't store it exactly like this if we want to sort. From here there's quite a few shapes that we can process the data into, but I ultimately settled on processing the JSON provided by DVC into a flat array of objects with added commitId and experimentId fields to represent the relations. Not only does this simplify iteration, but I'm willing to bet we will want to be able to sort experiments ungrouped to get insight on data across all commits.

Flat arrays are also the data input of react-table, a popular React lib for exactly this kind of thing. It has:

  • support for sorting, grouping, and nesting, which handles most of the table state logic.
  • no enforced opinions on UI
  • lots of feature overrides and customization
  • a 15.3kb minzipped size

@pared
Copy link

pared commented Oct 30, 2020

@rogermparent worth noting that this data structure is consistent with what we are using in dvc plots - ultimately, any plots file ends up as a list of data points with added rev (revision) field and gets injected into the plot.

@rogermparent rogermparent mentioned this issue Oct 30, 2020
Merged
5 tasks
@shcheklein
Copy link
Member

@rogermparent let's move plots to a separate ticket ? And close this one as a "rough table implementation"?

@rogermparent rogermparent changed the title Implement table+plots Webview Rough table implementation Dec 2, 2020
@shcheklein shcheklein mentioned this issue Dec 30, 2020
Closed
6 tasks
rogermparent added a commit that referenced this issue Apr 14, 2021
@rogermparent
Experiment Commands #4: Addressing comments from 1-3 (#271)
70c6b33 
* Change execCommand to resolve to stdout and simplify its consumers

* runDvcCommand => runCommand

* Reorganize command enum to distinguish exp commands with a prefix

* DvcGc => Gc
rogermparent added a commit that referenced this issue Apr 14, 2021
@rogermparent
Experiment Commands #3: Add command for exp gc with selectable flags (
5b93e3f 
#269)

* Add command for GC

* Change gc to exp gc, and add leading `--` to enum flags

* Add GC command tests and export GC QuickPickItem interface

* Rename exp gc enum entry and reader command

* Change gcExperiments to experimentGarbageCollect

* Replace test() with it() for consistency

* Experiment Commands #4: Addressing comments from 1-3 (#271)

* Change execCommand to resolve to stdout and simplify its consumers

* runDvcCommand => runCommand

* Reorganize command enum to distinguish exp commands with a prefix

* DvcGc => Gc
rogermparent added a commit that referenced this issue Apr 14, 2021
@rogermparent
Experiments Commands #2: Add "Run All Queued Experiments" Command (#268)
dc04ea1 
* Add "Run Queued Experiments" command

* Add test for "runQueuedExperiments"

* Experiment Commands #3: Add command for `exp gc` with selectable flags (#269)

* Add command for GC

* Change gc to exp gc, and add leading `--` to enum flags

* Add GC command tests and export GC QuickPickItem interface

* Rename exp gc enum entry and reader command

* Change gcExperiments to experimentGarbageCollect

* Replace test() with it() for consistency

* Experiment Commands #4: Addressing comments from 1-3 (#271)

* Change execCommand to resolve to stdout and simplify its consumers

* runDvcCommand => runCommand

* Reorganize command enum to distinguish exp commands with a prefix

* DvcGc => Gc
rogermparent added a commit that referenced this issue Apr 14, 2021
@rogermparent @mattseddon
Add Commands for exp queue, exp run --run-all, and exp gc (#251)
1f7c783 
* Add a "Queue Experiments" command

No tests yet, but there is info message functionality which seems to work well.

* Try to make a test for the new queue command

* Re-add failing test

* Move individual test file into index test file

* Use commands enum for queue experiment

* Rename `queue_experiment` enum to follow new allcaps convention

* Experiments Commands #2: Add "Run All Queued Experiments" Command (#268)

* Add "Run Queued Experiments" command

* Add test for "runQueuedExperiments"

* Experiment Commands #3: Add command for `exp gc` with selectable flags (#269)

* Add command for GC

* Change gc to exp gc, and add leading `--` to enum flags

* Add GC command tests and export GC QuickPickItem interface

* Rename exp gc enum entry and reader command

* Change gcExperiments to experimentGarbageCollect

* Replace test() with it() for consistency

* Experiment Commands #4: Addressing comments from 1-3 (#271)

* Change execCommand to resolve to stdout and simplify its consumers

* runDvcCommand => runCommand

* Reorganize command enum to distinguish exp commands with a prefix

* DvcGc => Gc

Co-authored-by: mattseddon <37993418+mattseddon@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
priority-p2 Future feature, less priority for now question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@yalozhkin @pmrowla @jorgeorpinel @fabiosantoscode @shcheklein @rogermparent @pared