exp: show: Include additional info in --json.

[daavoo]

Use relpath as keys in deps and outs.
Add use_cache and is_data_source for outs.

This allows differentiating git-tracked dependencies and
intermediate outputs (pseudo-code bellow).

Closes #7575
Closes #7790

---

Example code on how to use the new fields:

for dep_name, dep in res["deps"].items():
    for out_name, out in res["outs"].items():
   
        # dep and out are the same artifact
        if dep_name == out_name:

             if not out["is_data_source"]:
                 # This is an intermediate artifact
                 
             if not out["use_cache"]:
                 # This is git-tracked artifact
                 
        else:
            # This is a git-tracked dependency

[pmrowla]

LGTM assuming this works for the vscode side

[mattseddon]

[Q] Do I have everything I need to work this out:

for dep in res[“deps”].values():
    for out in res[“outs”].values():
        if dep[“fs_path”] == out[“fs_path”]:
            # THIS IS AN INTERMEDIATE ARTIFACT
            dep["intermediate"] = True
            out["intermediate"] = True
            if out[“use_cache”]:
                # DEP IS DVC TRACKED
                dep["use_cache"] = True
            else:
                # DEP IS GIT-TRACKED 
                dep["use_cache"] = False

The answer is yes

Given that the rules about deciding what files should be included appear to be currently unclear and product-dependent I would prefer not to include this logic in DVC.

This does make sense to me. However, there will be much less chance of providing consistency across dependencies if we push this logic out to the consumers.

[shcheklein]

@mattseddon where does this snippet come from?

I see that is_data_source is included which is what we need I believe, @daavoo could you remind please what is the definition of the out.stage.is_data_source?

[mattseddon]

#7575 (comment)

[mattseddon]

Correct code is in the description?

[mattseddon]

This will be enough for us right now as we're working on implementing:

The first step is to get all of the deps from exp show shown in the columns tree and experiments webview.

as per iterative/vscode-dvc#1183 (comment)

I think we should revisit when we get to:

Once extra information has been added to the DVC output we will provide it to the user (in some way to be determined) so they can easily filter out what they need.

and/or when we redesign the data that the extension needs

[shcheklein]

@daavoo thanks for addressing this!

Example code on how to use the new fields:

I'm not sure that example is 100% correct. What happens if I have only a few .dvc (dvc add) in the project? There will be no deps at all, right? But there will be outs that should be marked as "data sources".

[shcheklein]

Please review the comment regarding the .dvc files ... at least we should fix an example.

In general - why would I need that double cycle at all btw, why can't I just take all data source outs (I hope it includes imports) ... and for final outputs - take outs that don't have deps.

[daavoo]

I'm not sure that example is 100% correct. What happens if I have only a few .dvc (dvc add) in the project? There will be no deps at all, right? But there will be outs that should be marked as "data sources".

The example is from the current DVC perspective where deps is what is shown in the table and artifacts not connected to the pipeline are not shown.

VSCode has access to both deps and outs and can process and decide differently.

Please review the comment regarding the .dvc files ... at least we should fix an example.

Reverse the loop. Iterate on outs:

for out_name, out in res["out"].items():
    if not out["is_data_source"]:
        if out_name in res["deps"]:
            # This is an intermediate artifact
        else:
            # This is a "final output"

why can't I just take all data source outs (I hope it includes imports)

You can, as per the example above.
Inclusion of imports is tested here:

dvc/tests/func/experiments/test_show.py

Lines 691 to 716 in 48fe304

	dvc.imp(os.fspath(erepo_dir), "out", "out_imported")
	outs = dvc.experiments.show()["workspace"]["baseline"]["data"]["outs"]
	assert outs == {
	"out": {
	"hash": ANY,
	"size": ANY,
	"nfiles": None,
	"use_cache": True,
	"is_data_source": False,
	},
	"out_add": {
	"hash": ANY,
	"size": ANY,
	"nfiles": None,
	"use_cache": True,
	"is_data_source": True,
	},
	"out_imported": {
	"hash": ANY,
	"size": ANY,
	"nfiles": None,
	"use_cache": True,
	"is_data_source": True,
	},
	}

why would I need that double cycle at all btw,

You kind of will still double looping because of and for final outputs - take outs that don't have deps (the in is still a loop).

And without also iterating on deps, you won't be able to have a signal for updates on git-tracked dependencies.

[shcheklein]

Got it, thanks @daavoo !