plots/metrics/params/experiments show: stop throwing exceptions #5984

pared · 2021-05-07T14:55:40Z

❗ I have followed the Contributing to DVC checklist.
📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

pared · 2021-05-12T16:20:07Z

In this change I try to provide a new way of handling errors.
Lets take a look at sample repositry, with broken dvc.yaml on workspace and one of the experiments commits breaking out.yaml metric.

Providing test code for reproduction puproses:

@pytest.fixture
def custom_exp_stage(tmp_dir, scm, dvc):
    dump_yaml(tmp_dir / "params.yaml", {"foo": 111, "bar": 222})
    dump_yaml(tmp_dir / "other.yaml", {"a": 333, "b": 999})
    CODE = dedent(
        """
        from ruamel.yaml import YAML
        ruyaml=YAML()
        with open('params.yaml', 'r') as fd:
          data=ruyaml.load(fd)
        with open('other.yaml', 'r') as fd:
          other=ruyaml.load(fd)
        
        def to_metric(d):
          res = {}
          for n,v in d.items():
            res[n + "_met"] = v/10
          return res

        with open('out.yaml', 'w') as fd:
          ruyaml.dump(to_metric(data), fd)
        with open('other_out.yaml', 'w') as fd:
          ruyaml.dump(to_metric(other), fd)"""
    )
    tmp_dir.gen("code.py", CODE)
    scm.add(["params.yaml", "other.yaml", "code.py"])
    scm.commit("init")

    stage = dvc.run(
        name="train",
        params=["foo,bar", "other.yaml:a,b"],
        metrics_no_cache=["out.yaml", "other_out.yaml"],
        cmd="python code.py",
        deps=["code.py"],
    )
    scm.add(
        ["dvc.yaml", "dvc.lock", ".gitignore", "out.yaml", "other_out.yaml"]
    )
    scm.commit("initial run done")
    return stage


def test_show_broken(tmp_dir, scm, dvc, custom_exp_stage):
    init_rev = scm.get_rev()
    dvc.experiments.run(
        custom_exp_stage.addressing, params=["foo=2"], name="good_experiment"
    )

    scm.gitpython.git.reset(init_rev, hard=True)
    BREAK_METRIC_CODE = dedent(
        """
    with open("out.yaml", "a") as fd:
        fd.write("break the yaml!")
    """
    )

    with open("code.py", "a") as fd:
        fd.write(BREAK_METRIC_CODE)

    dvc.experiments.run(custom_exp_stage.addressing, name="break_metric")
    scm.gitpython.git.reset(init_rev, hard=True)
    dvc.checkout()

    with open("dvc.yaml", "a") as fd:
        fd.write("break the yaml")

    assert main(["exp", "show", "--no-pager"]) == 0

In current master DVC version (lets assume 2.1.0) Upon running dvc exp show we will get:

ERROR: failed to show experiments - unable to read: 'dvc.yaml', YAML file structure is corrupted: while scanning a simple key
  in "<unicode string>", line 17, column 1
could not find expected ':'
  in "<unicode string>", line 18, column 1

The result for this change:

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Experiment                    ┃ Created  ┃ other_out.yaml:a_met ┃ other_out.yaml:b_met ┃ out.yaml:foo_met ┃ out.yaml:bar_met ┃ params.yaml:foo ┃ params.yaml:bar ┃ other.yaml:a ┃ other.yaml:b ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ workspace                     │ -        │ ERR_BRANCH           │ ERR_BRANCH           │ ERR_BRANCH       │ ERR_BRANCH       │ 111             │ 222             │ ERR_BRANCH   │ ERR_BRANCH   │
│ master                        │ 06:11 PM │ 33.3                 │ 99.9                 │ 11.1             │ 22.2             │ 111             │ 222             │ 333          │ 999          │
│ ├── bbfde7f [good_experiment] │ 06:11 PM │ 33.3                 │ 99.9                 │ 0.2              │ 22.2             │ 2               │ 222             │ 333          │ 999          │
│ └── a6b1a41 [break_metric]    │ 06:11 PM │ 33.3                 │ 99.9                 │ ERR_FILE         │ ERR_FILE         │ 111             │ 222             │ 333          │ 999          │
└───────────────────────────────┴──────────┴──────────────────────┴──────────────────────┴──────────────────┴──────────────────┴─────────────────┴─────────────────┴──────────────┴──────────────┘

skshetry · 2021-05-14T07:29:39Z

Hi, @pared. Could you please provide me the issue tickets and other contexts when and why we started to suppress those error messages? I have been trying to understand the issue on generalizing this thing as a guideline, on why it's okay to suppress exceptions for some commands and not on others.

But I am having issues understanding why we don't print errors to the terminal saying that the "view" or "table" might not be "complete" because of such and such reasons and proceed with the rendering not-so complete table and then exit with 1 error code. This is not related to this PR, but the status quo.

Btw, have you looked into #5038? I feel like tolerating errors is a more general issue than the metrics/params/plots/experiments, and we need a general framework to do this.

Regarding the metrics/params/plots/experiments, with #5324, these commands started fetching files from remote in case it didn't exist before. You might have noticed that remote errors are making these commands fail now as we are only catching DvcException whereas the remotes don't have a standardized exceptions.
So this makes me think that we won't ever be able to catch every errors and even we do, it might not be a good solution.

Take an example of a recently cloned dvc repo with no cache but remote configured with credentials that are invalid. We'd have just failed before, because there's no way to proceed further. But now with this PR, we'd just be hammering the remote trying to find all the files in the remote. So, I don't feel good about catch-all here in the PR.

But I also don't think catching a set of exceptions is the answer. Earlier [in a test repo I had], metrics show was not showing all information because it was raising OutputIsIgnoredError, which I'd have never thought to whitelist to that set. This is because we have too many places where the errors can be raised (and, RepoFS is too complicated).

So, I'd prefer we spend some time on having a general framework around this and try to harmonize exceptions. Regarding UI, I'd prefer we print error messages to stderr and then render the table, and maybe return the exit code.

pared · 2021-05-14T13:29:56Z

Could you please provide me the issue tickets and other contexts when and why we started to suppress those error messages? I have been trying to understand the issue on generalizing this thing as a guideline, on why it's okay to suppress exceptions for some commands and not on others.

There is none, the discussion regarding this behaviour is conducted under linked issues, from first message in this PR.
The guideline would be a good idea, i guess we need to fromalize this and inspect whether other commands could benefit from such behaviour. Lets start a discussion on that?

But I am having issues understanding why we don't print errors to the terminal saying that the "view" or "table" might not be "complete" because of such and such reasons and proceed with the rendering not-so complete table and then exit with 1 error code. This is not related to this PR, but the status quo.

This might be my reluctance to use logger speaking, but we used to have a lot of issues which point was that logs generated during command execution were occluding the command result. After first use with failed revision, WARNING or INFO is informative. User checks what is wrong, understands. Yet, repo history does not get revritten, so now every time the user uses dvc * show/diff, the warning is still there. And if user runs the command yet again (with some other failed revision) there just no results for failed revs. Current apporach gives granularity, telling the user whether particular file and revision failed to be obtained or whether the file does not exists. I don't say that ERR_FILE and ERR_REV is proper naming, but it lets one quickly understand what is going on.

Btw, have you looked into #5038? I feel like tolerating errors is a more general issue than the metrics/params/plots/experiments, and we need a general framework to do this.

I guess its related to first question.

Regarding the metrics/params/plots/experiments, with #5324, these commands started fetching files from remote in case it didn't exist before. You might have noticed that remote errors are making these commands fail now as we are only catching DvcException whereas the remotes don't have a standardized exceptions.
So this makes me think that we won't ever be able to catch every errors and even we do, it might not be a good solution.

Yes, in current master state we are not able to catch them all. This change however makes handling those errors possible. The intercep_error context manager was created in order to handle unexpected errors on revision. We know that in exp/params/metrics/plots commands we have 2 "levels" of potential failure: revision (remote fetch failed, wrongly formatted dvc.yaml) and path (wrongly saved metric, for example, or not saved at all). So we can intercept most of the errors (probably beside brancher ones). As we don't know what is the error, we have to assume that we might be able to recover on different branch. It might not be the best solution but the alternative is implementing handling for all types of errors, which seems to bother users. They have to create issue because whole command fails, while they just wanted to check the metrics progress (#5845)

Take an example of a recently cloned dvc repo with no cache but remote configured with credentials that are invalid. We'd have just failed before, because there's no way to proceed further. But now with this PR, we'd just be hammering the remote trying to find all the files in the remote. So, I don't feel good about catch-all here in the PR.

Yes this is unfortunate, but this could be handled later. Instead of erroring on every exception, suppress all and implement special handling for the ones we know how to handle. In this case we could implement onerror that raises exception when there is authorization problem, and ignores others.

So, I'd prefer we spend some time on having a general framework around this and try to harmonize exceptions. Regarding UI, I'd prefer we print error messages to stderr and then render the table, and maybe return the exit code.

Lets start the discussion then?

My main point in this change is that i believe we need a shift in how we handle errors for let's name it read only commands. They are safe, they will not break the repo, and their biggest value is providing even smallest piece of information about repository. In other places, we catch only particular errors, because its safer to break the execution than let it continue after unknown exception. Here it is different - unknown exception will not harm the repo, so its better to supress and do as much as possible.

dberenbaum · 2021-05-14T13:55:15Z

@skshetry What should the table look like in your mind? Should the a6b1a41 line be missing? Should that line exist but the ERR_FILE cells are empty?

I'd prefer we print error messages to stderr and then render the table, and maybe return the exit code

If we are still going to continue and possibly encounter multiple errors, how do we decide which exit code to show?

@pared Is there a way to see the individual error messages?

pared · 2021-05-14T14:07:19Z

@dberenbaum yes, they are put into log on debug level. I still consider putting them into trace, but no matter what we decide, it is still possible to see them.

As to exit code, it seems to me that status with failed revisions should still be 0. Other should be reserved to unexpected errors. But this is another subject for a more general discussion how do we handle excepitons in such cases.

skshetry · 2021-05-14T15:40:12Z

@skshetry What should the table look like in your mind? Should the a6b1a41 line be missing? Should that line exist but the ERR_FILE cells are empty?

I am not convinced that we need to indicate in the table itself, but I am not opposed to it.

If we are still going to continue and possibly encounter multiple errors

I am not saying that we print all the logs. I feel that there should be at least a message printed in stderr indicating that the table is not complete to the user.

how do we decide which exit code to show?

If we decide to fail in all cases, I'd say we return 1 if it is minor failures and 2 if it is serious (similar to ls).
If we decide not to fail in all cases, we could return 1 if it's serious/unhandled failures, else it'll be 0.

pared · 2021-05-17T10:29:29Z

If we decide to fail in all cases, I'd say we return 1 if it is minor failures and 2 if it is serious

This looks like a valid solution for this case. I guess we would need to analyze other commands whether they also need such granularity.

EDIT:
Or rather, apply this approach to all commands. But we need to remember that it would be a breaking change.

pared · 2021-05-19T15:50:39Z

@skshetry I started a discussion regarding brancher: #6039

#6039 will deal with how to approach onerror and brancher.

With the problem how we should inform user about some problems, we should probably deal here.

So, during the demo i proposed 3 solutions:

Input error data into the table

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Experiment                    ┃ Created  ┃ other_out.yaml:a_met ┃ other_out.yaml:b_met ┃ out.yaml:foo_met ┃ out.yaml:bar_met ┃ params.yaml:foo ┃ params.yaml:bar ┃ other.yaml:a ┃ other.yaml:b ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ workspace                     │ -        │ ERR_BRANCH           │ ERR_BRANCH           │ ERR_BRANCH       │ ERR_BRANCH       │ 111             │ 222             │ ERR_BRANCH   │ ERR_BRANCH   │
│ master                        │ 05:36 PM │ 33.3                 │ 99.9                 │ 11.1             │ 22.2             │ 111             │ 222             │ 333          │ 999          │
│ ├── f36320f [break_metric]    │ 05:36 PM │ 33.3                 │ 99.9                 │ ERR_FILE         │ ERR_FILE         │ 111             │ 222             │ 333          │ 999          │
│ └── 3b77c91 [good_experiment] │ 05:36 PM │ 33.3                 │ 99.9                 │ 0.2              │ 22.2             │ 2               │ 222             │ 333          │ 999          │
└───────────────────────────────┴──────────┴──────────────────────┴──────────────────────┴──────────────────┴──────────────────┴─────────────────┴─────────────────┴──────────────┴──────────────┘

Log warning when some errors occured

WARNING: Loading files for some revisions failed (`workspace, f36320f4f9f65d3a856ba621274b05e602fe12d7`) Use `-v` to get more info.
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Experiment                    ┃ Created  ┃ other_out.yaml:a_met ┃ other_out.yaml:b_met ┃ out.yaml:foo_met ┃ out.yaml:bar_met ┃ params.yaml:foo ┃ params.yaml:bar ┃ other.yaml:a ┃ other.yaml:b ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ workspace                     │ -        │ !                    │ !                    │ !                │ !                │ 111             │ 222             │ !            │ !            │
│ master                        │ 05:36 PM │ 33.3                 │ 99.9                 │ 11.1             │ 22.2             │ 111             │ 222             │ 333          │ 999          │
│ ├── 3b77c91 [good_experiment] │ 05:36 PM │ 33.3                 │ 99.9                 │ 0.2              │ 22.2             │ 2               │ 222             │ 333          │ 999          │
│ └── f36320f [break_metric]    │ 05:36 PM │ 33.3                 │ 99.9                 │ !                │ !                │ 111             │ 222             │ 333          │ 999          │
└───────────────────────────────┴──────────┴──────────────────────┴──────────────────────┴──────────────────┴──────────────────┴─────────────────┴─────────────────┴──────────────┴──────────────┘

Mark failed files with (for example) ! and provide some context with, for example a legend under the table:

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Experiment                    ┃ Created  ┃ other_out.yaml:a_met ┃ other_out.yaml:b_met ┃ out.yaml:foo_met ┃ out.yaml:bar_met ┃ params.yaml:foo ┃ params.yaml:bar ┃ other.yaml:a ┃ other.yaml:b ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ workspace                     │ -        │ !                    │ !                    │ !                │ !                │ 111             │ 222             │ !            │ !            │
│ master                        │ 05:36 PM │ 33.3                 │ 99.9                 │ 11.1             │ 22.2             │ 111             │ 222             │ 333          │ 999          │
│ ├── 3b77c91 [good_experiment] │ 05:36 PM │ 33.3                 │ 99.9                 │ 0.2              │ 22.2             │ 2               │ 222             │ 333          │ 999          │
│ └── f36320f [break_metric]    │ 05:36 PM │ 33.3                 │ 99.9                 │ !                │ !                │ 111             │ 222             │ 333          │ 999          │
└───────────────────────────────┴──────────┴──────────────────────┴──────────────────────┴──────────────────┴──────────────────┴─────────────────┴─────────────────┴──────────────┴──────────────┘
! means some problem occured. Use `-v` to get more info.

I think we should agree which way we will go and keep it consistient for params/metrics/plots diff too.
My own preference would be 3.

cannot contain too much information due to lack of space
we have already been here - ususally users get frustrated with warnings that they are aware of and cannot disable
Seems to me like reasonable compromise. Though I am not sure its possible to implement it in pager. In the example i used --no-pager.

@skshetry @pmrowla @dberenbaum which one do you prefer?

pmrowla · 2021-05-20T01:21:20Z

With 3, your issue with the pager is just that currently we can only paginate the table itself in ui, right?

@skshetry It seems like our pager handling should be a level above that, so that we can paginate any UI output at all (and not just a single table). Then for 3 we would just write the table and then the error information and it would all show up in the pager.

This would also allow us to do things like writing a second (optional?) table that could potentially contain rows of commits we failed to parse and the full string error message for each row

skshetry · 2021-05-20T10:37:21Z

@skshetry It seems like our pager handling should be a level above that

@pmrowla, I have been thinking about --paginate and --no-pager like flags that git has. Unfortunately, for this to work, we should only be using rich, and I haven't been able to make the decision yet.

My preference would be to not show any indication on the table at all. If we must, I'd prefer ! at the side of the Experiment.

And, regarding errors, I believe they should go to stderr and:

We should filter some errors, and figure out the weight/importance of the errors, and should print severe ones.
(example: I like your error message "for some revisions failed". There's no way users will be able to fix the issues in their history, so we could just report the error in superficial level).
We should not be overwhelming users with error messages. If there are already some important error messages, it's okay to hide the smaller ones. They could be related to other major issues that we have already reported or might just get fix along the way.
And, I'd prefer we don't print Use -v for more info.

pared · 2021-05-20T11:15:41Z

@skshetry

But when would we be displaying the error messages? Before showing the table or after? I like @pmrowla's idea for creating second table with errors, because it puts the errors into background - we keep the ability to show whatever works, and there is a way to show errors.

We should not be overwhelming users with error messages.

The question here is how do we differentiate what is the important error. I think this is a good idea, though this assumes we create logic reacting to different errors. But what do we do with unknown errors?

And, I'd prefer we don't print Use -v for more info.

Well, if one is aware that -v he/she will probably use it anyway, so we can just remove it.

pared · 2021-05-28T10:06:56Z

dvc/command/plots.py

+        else:
+            from dvc.ui import ui
+
+            ui.warn(


We used to create file with no actual plots. After this change we will not create it if there are no plots.

@pared, we haven't yet decided on colors yet, so we are using ui.error_write in other places. It does look good, so maybe that's okay, but just wanted to let you know.

pared · 2021-05-28T14:32:03Z

Migth stil need to reiterate on type checks, though the implementation should be finished.

Some remarks:

For displaying failed metrics/params in experiments, we are showing only ! for now, as it needs solving on its own (see discussion above)
We no longer produce empty plots file if no plots has been gathered during collection/rendering.
Introduced new class Onerror which is gathering data on errors encountered during processing our data. This class has errors dict that is supposed to store info on failed revisions and particular paths. If the whole revision failed (eg malformed dvc.yaml, errors['{rev}'] will contain the exception raised. If only particular file failed, {rev}.{path} path will contain the error.

shcheklein · 2021-05-28T14:37:18Z

@pared good stuff! For the sake of the VS Code team, could you please summarize how does it affect --show-json? do we see these granular errors there now and how?

pared · 2021-05-31T16:14:39Z

@shcheklein In current state there is no change in --show-json - I think we need to discuss how can we support retriveing the errors too.

Suor · 2021-07-13T10:06:52Z

@pared I guess will handle that while updating dvc.

skshetry

Looks good @pared. I have some concerns about "data"/"error" being nested (like an onion? 🧅), but I also don't see how it could be kept in one place. I have inlined a few comments but look good overall. 👍🏼

skshetry · 2021-07-15T08:11:42Z

tests/unit/test_compare.py

+        metrics={
+            "branch_1": {
+                "data": {
+                    "metrics.json": {"data": {"b": {"ad": 1, "bc": 2}, "c": 4}}
+                }
+            },
+            "branch_2": {
+                "data": {
+                    "metrics.json": {"data": {"a": 1, "b": {"ad": 3, "bc": 4}}}
+                }
+            },
+        },
+        all_branches=True,
    )
    out, _ = capsys.readouterr()
    assert out == textwrap.dedent(
        """\
-        Path    diff    new    old
-        x.b     -       6      5
-        a.d.e   1       4      3
-        a.b.c   1       2      1
+        Revision    Path          a    b.ad    b.bc    c
+        branch_1    metrics.json  -    1       2       4
+        branch_2    metrics.json  1    3       4       -
        """
    )


 def test_metrics_show_md(capsys):
    show_metrics(
-        {
-            "metrics.yaml": {
-                "x.b": {"old": 5, "new": 6},
-                "a.d.e": {"old": 3, "new": 4, "diff": 1},
-                "a.b.c": {"old": 1, "new": 2, "diff": 1},
-            }
+        metrics={
+            "branch_1": {
+                "data": {
+                    "metrics.json": {"data": {"b": {"ad": 1, "bc": 2}, "c": 4}}
+                }
+            },
+            "branch_2": {
+                "data": {
+                    "metrics.json": {"data": {"a": 1, "b": {"ad": 3, "bc": 4}}}
+                }
+            },
        },
+        all_branches=True,
        markdown=True,
    )
    out, _ = capsys.readouterr()
    assert out == textwrap.dedent(
        """\
-        | Path   | diff   | new   | old   |
-        |--------|--------|-------|-------|
-        | x.b    | -      | 6     | 5     |
-        | a.d.e  | 1      | 4     | 3     |
-        | a.b.c  | 1      | 2     | 1     |
+        | Revision   | Path         | a   | b.ad   | b.bc   | c   |
+        |------------|--------------|-----|--------|--------|-----|
+        | branch_1   | metrics.json | -   | 1      | 2      | 4   |
+        | branch_2   | metrics.json | 1   | 3      | 4      | -   |


Do we need a test to check that they don't trip over new field "error"? Maybe, try sprinkling them on a few tests?

skshetry · 2021-07-15T08:34:44Z

dvc/command/metrics.py

-            # When `metrics` contains a `None` key, it means that some files
-            # specified as `targets` in `repo.metrics.show` didn't contain any
-            # metrics.
-            missing = metrics.pop(None, None)
-            if missing:
-                raise BadMetricError(missing)
-


So, we don't raise any error now?

Yes, I had 2 reasons for removing that:

I was unable to retrigger this behavior.

New approach assumes we do not throw exceptions but rather gather them. Lack of metrics will be represented as empty files content.

Should/can we at least provide a warning or something that some metrics are missing?

dvc/command/plots.py

tests/func/experiments/test_show.py

pared · 2021-07-16T13:40:07Z

@Suor @rogermparent

The change is on the master now.

pared force-pushed the unify_show_error_handling branch 3 times, most recently from 7650d03 to 9501cf2 Compare May 11, 2021 17:17

pared force-pushed the unify_show_error_handling branch 2 times, most recently from fc8bd38 to 90cfc5a Compare May 18, 2021 11:38

Suor mentioned this pull request May 21, 2021

Confine collection errors at lower level possible #6042

Closed

pared force-pushed the unify_show_error_handling branch 8 times, most recently from 1508b6a to f078c5d Compare May 28, 2021 09:12

pared commented May 28, 2021

View reviewed changes

pared changed the title ~~[WIP] plots/metrics/params/experiments show: stop throwing exceptions~~ plots/metrics/params/experiments show: stop throwing exceptions May 28, 2021

pared added 7 commits July 13, 2021 11:52

fixups

88624b8

plots/metrics/params/experiments: temporarily remove error warnings

934c35a

plots: extract parsing out of class method

5ac9a36

metrics/experiments: provide exception serializer for show-json

39b3d77

params/plots/metrics: signal errors

0a8e9bc

rebase fixups

936624e

fix TODOs

325c160

pared force-pushed the unify_show_error_handling branch from 07b0865 to 325c160 Compare July 13, 2021 09:52

fixup

cd4c95b

dberenbaum approved these changes Jul 13, 2021

View reviewed changes

pmrowla approved these changes Jul 14, 2021

View reviewed changes

skshetry approved these changes Jul 15, 2021

View reviewed changes

skshetry's review refactor

fe78054

pared merged commit 21af743 into iterative:master Jul 16, 2021

efiop added bugfix fixes bug enhancement Enhances DVC labels Jul 19, 2021

mattseddon mentioned this pull request Jul 21, 2021

Placeholder: Needs design iterative/vscode-dvc#175

Closed

13 tasks

pared mentioned this pull request Jul 22, 2021

plots: cleanup data extraction #6355

Merged

2 tasks

pared mentioned this pull request Aug 19, 2021

plots: document no-error policy iterative/dvc.org#2734

Closed

pared deleted the unify_show_error_handling branch January 4, 2022 10:13

dberenbaum mentioned this pull request Feb 18, 2022

allow skipping certain git revisions/commits in DVC #5066

Closed

dberenbaum mentioned this pull request Apr 21, 2022

gc: requires loading parameters files #7585

Open

maxagin mentioned this pull request May 1, 2022

DVC errors that may occur during or after running experiment iterative/vscode-dvc#1636

Closed

4 tasks

dberenbaum mentioned this pull request Feb 14, 2023

plots diff: don't make one plot break all the others #9025

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

plots/metrics/params/experiments show: stop throwing exceptions #5984

plots/metrics/params/experiments show: stop throwing exceptions #5984

pared commented May 7, 2021 •

edited

Loading

pared commented May 12, 2021

skshetry commented May 14, 2021 •

edited

Loading

pared commented May 14, 2021

dberenbaum commented May 14, 2021

pared commented May 14, 2021

skshetry commented May 14, 2021 •

edited

Loading

pared commented May 17, 2021 •

edited

Loading

pared commented May 19, 2021 •

edited

Loading

pmrowla commented May 20, 2021

skshetry commented May 20, 2021

pared commented May 20, 2021

pared May 28, 2021

skshetry Jun 17, 2021

pared commented May 28, 2021

shcheklein commented May 28, 2021

pared commented May 31, 2021

Suor commented Jul 13, 2021

skshetry left a comment

skshetry Jul 15, 2021

skshetry Jul 15, 2021

pared Jul 15, 2021 •

edited

Loading

dberenbaum Aug 20, 2021

pared commented Jul 16, 2021

plots/metrics/params/experiments show: stop throwing exceptions #5984

plots/metrics/params/experiments show: stop throwing exceptions #5984

Conversation

pared commented May 7, 2021 • edited Loading

pared commented May 12, 2021

skshetry commented May 14, 2021 • edited Loading

pared commented May 14, 2021

dberenbaum commented May 14, 2021

pared commented May 14, 2021

skshetry commented May 14, 2021 • edited Loading

pared commented May 17, 2021 • edited Loading

pared commented May 19, 2021 • edited Loading

pmrowla commented May 20, 2021

skshetry commented May 20, 2021

pared commented May 20, 2021

pared May 28, 2021

Choose a reason for hiding this comment

skshetry Jun 17, 2021

Choose a reason for hiding this comment

pared commented May 28, 2021

shcheklein commented May 28, 2021

pared commented May 31, 2021

Suor commented Jul 13, 2021

skshetry left a comment

Choose a reason for hiding this comment

skshetry Jul 15, 2021

Choose a reason for hiding this comment

skshetry Jul 15, 2021

Choose a reason for hiding this comment

pared Jul 15, 2021 • edited Loading

Choose a reason for hiding this comment

dberenbaum Aug 20, 2021

Choose a reason for hiding this comment

pared commented Jul 16, 2021

pared commented May 7, 2021 •

edited

Loading

skshetry commented May 14, 2021 •

edited

Loading

skshetry commented May 14, 2021 •

edited

Loading

pared commented May 17, 2021 •

edited

Loading

pared commented May 19, 2021 •

edited

Loading

pared Jul 15, 2021 •

edited

Loading