Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support yaml as metrics file #3572

Closed
1 task
dmpetrov opened this issue Apr 1, 2020 · 7 comments · Fixed by #3658
Closed
1 task

Support yaml as metrics file #3572

dmpetrov opened this issue Apr 1, 2020 · 7 comments · Fixed by #3658
Assignees
Labels
feature request Requesting a new feature p1-important Important, aka current backlog of things to do product: VSCode Integration with VSCode extension

Comments

@dmpetrov
Copy link
Member

dmpetrov commented Apr 1, 2020

Yaml is one of the most used file format. We need to support it for metrics.
More details: #3409 (comment)

ToDo:

  • update docs
@triage-new-issues triage-new-issues bot added the triage Needs to be triaged label Apr 1, 2020
@dmpetrov dmpetrov added feature request Requesting a new feature product: VSCode Integration with VSCode extension labels Apr 1, 2020
@triage-new-issues triage-new-issues bot removed the triage Needs to be triaged label Apr 1, 2020
@efiop efiop added the p1-important Important, aka current backlog of things to do label Apr 5, 2020
@dmpetrov
Copy link
Member Author

dmpetrov commented Apr 7, 2020

Ideally, as part of this issue a few other steps should be implemented from the link above:
[2] Add support for YAML since it is an often-used format - #3393 (comment)
[3] With two simple json/yaml formats, do we really need raw format? Let's get rid of raw metrics format.
[4] Do we really need xpath in dvc-files? I see a clear use case in metrics show as a filter but not in other commands. Let's consider removing XPath (from metrics files).
[5] Metrics type was mostly used for distinguishing xpath algorithms between hierarchical json and flat csv/tsv/hcsv/htsv. It looks like it is not needed if only json/yaml left. Let's remove metrics type.

@efiop efiop self-assigned this Apr 8, 2020
@efiop
Copy link
Contributor

efiop commented Apr 8, 2020

@dmpetrov So maybe let's get rid of raw, *sv support, drop explicit XPath and type, support only yaml (could leave json, but is there really a point?) and pretty much do the same thing that we do with params? E.g.

dvc run -m all.AUC ... # will assume metrics.yaml, similar to how we assume `params.yaml`
dvc run -m custom_name.yam:all.AUC ...

?

And -m won't even mean that metrics.yaml is considered an official output cached by dvc. -M will be simply gone in that case. So a usual workflow would be to keep that metrics file in git or not commit it at all, even. These are just things from the top of my head, not sure if this is the way to go.

@dmpetrov
Copy link
Member Author

dmpetrov commented Apr 9, 2020

@efiop agree re the format support (but I'd keep json). The good news is - both yaml and json are hierarchical and we can use the same patterns for filters (like path).

Granular metrics addressation (like -m all.AUC) seems a bit too complicated scenario. It has one advantage - multiple stages can append metrics to the same metrics file. But this metrics appending is not easy to implement. We can think about this feature when we have more demand for such scenarios.

efiop pushed a commit that referenced this issue Apr 18, 2020
* add: -f arg name from FILE to <filename>
to match dvc.org cmd ref

* dvc: cmd arg names (metavars) to match docs
for #3626

* metrics: update help output to indicate supported types
per #3409
and for #3572

* dvc: reorder subcommands to match docs
for #3608
efiop added a commit to efiop/dvc that referenced this issue Apr 21, 2020
Dropping support for all types except json and yaml. No more xpath
support.

Fixes iterative#3572
efiop added a commit that referenced this issue Apr 22, 2020
Dropping support for all types except json and yaml. No more xpath
support.

Fixes #3572
@jorgeorpinel
Copy link
Contributor

jorgeorpinel commented Apr 22, 2020

update docs

@dmpetrov will you be sending a draft for this or should I include in my regular updates? Seems likt it can be a pretty quick update, just add "YAML" after "JSON" in metrics subcommand references. Unless you want to include some special examples

@efiop
Copy link
Contributor

efiop commented Apr 22, 2020

@jorgeorpinel Dmitry promised to send a PR 🙏 There are some other changes in the way metrics work, so it won't be as easy as adding YAML to the docs.

@shcheklein
Copy link
Member

@jorgeorpinel let's create a ticket!

@jorgeorpinel
Copy link
Contributor

OK: iterative/dvc.org/issues/1175

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request Requesting a new feature p1-important Important, aka current backlog of things to do product: VSCode Integration with VSCode extension
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants