exp show: Include data files.

[daavoo]

If an user modifies data files in between dvc exp run calls, the change is not reflected in the table generated by dvc exp show.

In Studio users can view information of the data files associated to each experiment. I think it would be nice to include similar information in the table generated by dvc exp show.

Ideally, the feature would have the same options available for metrics and params (i.e. --include-files/--exclude-files).

[shcheklein]

+1 on this. I'll be important for VS Code. On a separate note. We haven't touched source files in the Studio yet, but it would be great to have signal in the table from which you can see that there are changes to the source files between experiments.

[pmrowla]

What should the table actually show? I don't think actually including a list of changed files in the CLI table is feasible due to the size constraints.

So should it just be a true/false column to note that there is a change to data files (so users know they should check the changes with dvc diff? Or maybe columns for added/modified/removed with counts that would match the summary totals you get from dvc diff?

[karajan1001]

But if we use added/modified/removed, there comes another question. These notions only represent the status compare to default/workspace version. Unlike with the numbers it is hard to know the differences between two modified exps.

[pmrowla]

For experiments in the table it would have to be the diff between the experiment and the original baseline git commit (not the diff of one experiment to another experiment)

To get the difference between two experiments you would just have to use dvc diff exp1 exp2

[daavoo]

+1 on this. I'll be important for VS Code. On a separate note. We haven't touched source files in the Studio yet, but it would be great to have signal in the table from which you can see that there are changes to the source files between experiments.

Any source files or just source files that are deps of a stage?

[dberenbaum]

What is meaningful info to show about data files? Just whether they have changed from the baseline commit? Size? Number of files? Maybe there is some tag/label/metadata that would be helpful?

Also, what files should be included (dependencies, outputs, dvc-tracked files, all files)?

Ideally, the feature would have the same options available for metrics and params (i.e. --include-files/--exclude-files).

If we add data files to the dvc exp show table, it will become even more important to provide easy ways to narrow the table. Adding more info is not universally desirable since the table can become too busy. See:

• dvc exp show with target #6271
• exp show: show only changed parameters #5966
• show: revisiting table format? #6451

[shcheklein]

Any source files or just source files that are deps of a stage?

I would say just a binary signal that some sources have changed is a good start.

Also, what files should be included (dependencies, outputs, dvc-tracked files, all files)?

My take (from Studio experience) - inputs (datasets) is the most important part. Les so intermediate, etc. We can start with datasets only if want to keep the table sane.

What is meaningful info to show about data files? Just whether they have changed from the baseline commit? Size? Number of files? Maybe there is some tag/label/metadata that would be helpful?

Good question. I think some signal that data has changes is enough (we highlight values in Studio with a special dot icon or something like this). Values - we could do size to start. In Studio users can switch between those.

[karajan1001]

Show the same thing in dvc status is the first step.

[dberenbaum]

Another idea here is to encourage use of the desc field and show that in the table.

[daavoo]

Another idea here is to encourage use of the desc field and show that in the table.

In addition, we could show the rev (or other selectable property, line in Studio iterative/studio-support#16)

[daavoo]

Another idea here is to encourage use of the desc field and show that in the table.

Related #6548

[karajan1001]

Another idea here is to encourage use of the desc field and show that in the table.

But the problem is that as we didn't change the desc after the .dvc file had been generated, it stays the same in all of the revisions. Cann't be taken any comparison.

[dberenbaum]

I was thinking that users might update the description when they make changes, like cats -> cats and dogs. Maybe it would be better to track the last modified commit and show the commit message 🤷 .

[karajan1001]

I was thinking that users might update the description when they make changes, like cats -> cats and dogs. Maybe it would be better to track the last modified commit and show the commit message 🤷 .

Only support for .dvc files not for pipeline generated ones? That's make sense, Only those dvc add generated data need a name, those generated via pipeline ones can be represented by the parameters from stages.