DVC errors that may occur during or after running experiment

[maxagin]

Find solution for displaying errors when the experiment failed, broken parameters, metrics and more.

TODO

EXPs table

• Error at the row level
• On a particular column within a row
• A hint with title and error content

Figma specs

Plots

• Plots: handle errors gracefully Handle DVC errors gracefully in plots #1649

Figma specs

---

To consider

• The hint includes Error and How To Fix content example
• Hover style and active symbol style

Helpful links

• plots/metrics/params/experiments show: stop throwing exceptions dvc#5984
• https://github.com/iterative/dvc/blob/8cdcf9b8bb31a5badf7459efc3ec308d6d3ed62e/dvc/commands/experiments/show.py#L24-L25

[maxagin]

@shcheklein let me know what do you think.

• Error at the row level
• Color justification and variants

• Error at a particular column within a row
• A hint with title and error content
• Active symbol style example

The hint including Error and How To Fix content example

[shcheklein]

@maxagin we need to see a bit more realistic scenarios - e.g. the whole params file / metrics file could not be parsed (so many errors at once), or if the whole experiment failed - what do exactly we see instead of metrics / params / etc.

It might change the requirements and perception of the design.

[maxagin]

@shcheklein here are a few examples. Let me know what you think about it (The modal opens on click)

The whole metrics file could not be parsed (many errors at once)

If the whole experiment failed

[shcheklein]

Thanks, Max. It makes sense, but happens if we have 10+ values + 10 experiments?

Think I'm trying to see here is how noisy it can become if we are using these full color icons.

[maxagin]

Please check this out. This is example with light background

[maxagin]

Example using dark theme.

[maxagin]

Tried to reflect on our earlier discussion. Please comment.

[shcheklein]

@maxagin that's good (not sure if it's easy or not to implement those sticky signs at the edge though).

A few questions / comments:

• alignment - will those symbols be aligned to the right? If not, we need to try some realistic scenario, when a column is wider than its content and content is aligned to the right while symbols not.
• symbol itself - what symbol does DVC CLI use? should we use the same? (if't an exclamation mark - make it in the same style as in this screenshot?)

[maxagin]

@shcheklein

not sure if it's easy or not to implement those sticky signs at the edge though

This needs to be tested with the current design patterns. I just wanted to introduce the concept quickly. However this is a webview container, so we have a lot of freedom I think.

when a column is wider than its content and content is aligned to the right while symbols not

Let me create a better visualisation of the concept, because I feel it is not clear.

…what symbol does DVC CLI use? should we use the same? …

DVC and Studio use ‘!’ for errors. I will create an example. However the symbol may have advantages, not just because it is different then font or has two states (no fill, with fill), but also because it stands out more, but not too much. Let’s test the ‘!’.

[maxagin]

when a column is wider than its content and content is aligned to the right while symbols not . @shcheklein

Concept of showing row's errors that are shown at the table, but does not currently at the viewport (horizontal scroll)

Responsive behaviour

Just a stylized better example

---

Behavior of the right panel elements on scroll

When the last error of the row showed up the side indicator will be removed. Same if we scroll left - as soon as the error disappears under the left window age, the indicator will appear at the right panel.

If there are no hidden errors the right panel won't appear

[maxagin]

…what symbol does DVC CLI use? should we use the same? … @shcheklein

Using exclamation mark instead of rhombus for the errors symbol.

[shcheklein]

Questions that we still have:

• on the screenshots table is not aligned in its usual way (hard to see the final result). Can we do it please? Do we have the design system in Figma to iterate on that faster? (if not, let's discuss this - we need this I think)
• do we need a state when the whole experiment failed?
• we need to wrap it up and create a spec in Figma to hand off

[mattseddon]

Please put the final design into #662 when it is ready and close this issue.

[shcheklein]

What is the status of this, @maxagin ?

[maxagin]

@shcheklein I am preparing the Figma components. This is next on my list. Probably Monday will be done (first phase). The description has been updated.

[maxagin]

Hello. I have finalized the specs in Figma. Please review the doc and let me know if you have any comments/suggestions.

@shcheklein I especially added the Overview section so we can see the final UI and have a reference before jumping to the specs details.

Folks, let me know if this makes sense. Thanks

Figma

The ticket description has also been updated.

[shcheklein]

Looks good to me @maxagin , left a few small comments in Figma.

[maxagin]

make Error red? how do these notifications look in the VS Code itself?

@shcheklein, sharing the notifications examples below.

However here we are talking about hints and our hints are using ‘gray’ colors. I would suggest sticking to the existing pattern. wdyt?

[shcheklein]

However here we are talking about hints and our hints are using ‘gray’ colors. I would suggest sticking to the existing pattern. wdyt?

they are grey now, because they are indeed "hints" / neutral, and they don't even have any titles, right? I would take exactly same style for errors as VC Code has, wdyt? Except in our case symlol can be !.

[maxagin]

Should we always colorize errors with red? (green can confuse in this case). Red might overlap with one of the experiments but I think that should be fine?

My logic.

1. We want to minimize the contrasts. For this we have agreed to use the same color as the text. In this case gray.
2. The case when all exp has an error is different because the cell is not empty and the symbol appears beside the exp title so, since every exp has a color I found it makes sense to use the same color for the error. Did the same here, take a look and let me know wdyt please.

[shcheklein]

Sorry, may be it was not clear. I was asking only about tooltips, not about the table itself, and only about it's title "Error" (or we could use a small icon). I'm not sure we even had an intention to minimize the contrasts on them (but also, dark red won't break it tbh anyways).

so, since every exp has a color... I found it makes sense to use the same color for the error.

Yes, I find it confusing. It's the case where certain style is expected. Error can't be green or yellow. It's either make it neutral or make it all some dark red- everything else won't work very well I think.

[maxagin]

Example no. 1

Example no. 2

@shcheklein let me know wdyt please

[maxagin]

Another option would be to have it all white everywhere.

[shcheklein]

sorry, maybe I didn't communicate this well. I was totally fine to have them white in the table (and I think it's even better), I specifically meant only those colorful in Figma - only in the first column.

I like the tooltip (icon is not fully aligned?, but otherwise it looks good).

[maxagin]

I was totally fine to have them white in the table

Good 👍 . Will share the update soon.

icon is not fully aligned?

I am using the library component. I will have to compare it with other sources, maybe something has been messed up.

[maxagin]

Hmmm… 1px difference actually exists. For now will fix the icon placement only in this spec, but will keep the comment to change the library component. The font size also looks smaller than it should be.

[shcheklein]

Same question for the custom ! icon, why not use the native one that has been provided?

It's not an icon I believe, right? It's just a char.

And we are using it for consistency (DVC CLI). Also, heavy icons in the table will create a lot of noise.

[shcheklein]

In the tooltip it's fine to use the built-one, it won't be consistent with the table though. Your call.

[mattseddon]

Updated prototype:

Based on the spec.

[maxagin]

@mattseddon I have updated the specs taking into consideration our conversations. Please let me know if you find anything to be improved.

If exp is broken we may want to use "..." for the cells to indicate - no information. This is not the error, but just no info indicator. WDYT?

[maxagin]

Custom ! icon, why not use the native one that has been provided? @mattseddon

I think that the native icons you mentioned are related to the Notifications, which are placed at the specific right/button corner and have actions to resolve the issue + specific color.

This is why I have created a special symbol (which actually was a part of the codicon set before update #1636 (comment))
Let me know if this is good.

[mattseddon]

Re: "special symbol" - From https://code.visualstudio.com/api/references/icons-in-labels:

This seems very close to our use case. I would suspect that the icons was removed from the codicons or replaced for good reason. We should not bring it back.

From the design:

• How with the user know that there is a partial error for an experiment when these errors (signified by !) are offscreen or the columns are hidden?
• What is the reason for showing ! for partial errors but not for full errors?
• How can we simplify the design to make things very obvious to the user and not have to document what is going on?

[maxagin]

How with the user know that there is a partial error for an experiment when these errors (signified by !) are offscreen or the columns are hidden?

This is planned to be fixed in the second phase.

What is the reason for showing ! for partial errors but not for full errors?

If exp is broken we may want to use "..." for the cells to indicate - no information. This is not the error, but just no info indicator at the cell level.

How can we simplify the design to make things very obvious to the user and not have to document what is going on?

Please explain. TMO what we have here it is prity strate forward.

[mattseddon]

How with the user know that there is a partial error for an experiment when these errors (signified by !) are offscreen or the columns are hidden?

This is planned to be fixed in the second phase.

What is the reason for showing ! for partial errors but not for full errors?

If exp is broken we may want to use "..." for the cells to indicate - no information. This is not the error, but just no info indicator at the cell level.

The same can be said for either situation. We have no data to show. The data from the CLI does not contain !s we have to add them for both partial and full errors.

How can we simplify the design to make things very obvious to the user and not have to document what is going on?

Please explain. TMO what we have here it is prity strate forward.

Here are the rules as I see them

If an experiment has a record level error:

1. Experiment name is highlighted in red in webview.
2. Experiment name is highlighted in red in experiments tree.
3. Plots selection bullet is removed (therefore cannot select experiment and display it in plots webview).
4. The data in the record is left blank.
5. Hovering on the experiment will show the experiments error in a popup.
6. Hovering over the missing data with show nothing.

If an experiment has a file level error:

1. Experiment name is not highlighted in webview.
2. Experiment name is not highlighted in tree (how do I know there is something wrong with this record).
3. Plots selection bullet is unchanged (therefore can select experiment and display it in plots webview).
4. The missing data from the record is filled with !s.
5. Hovering on the experiment will not show the experiments error in a popup.
6. Hovering over the missing data with show the error for that particular file.
7. When errors columns are hidden a magic sticky column will appear that shows a ! (with what error if there are multiple).

As a user just looking at the table, I would have to look in different places to confirm that there are no errors for a particular record. Seems overcomplicated when we already have a sticky column that can contain all high-level information relating to the experiment.

Tieing in with #1649 (comment)

1. If the record is a full error then why would we let a user try to display it in the plots webivew? There is nothing to display.
2. If the record is a partial error then which of the possibly multiple errors would we show on hover of the experiment in the ribbon?
3. If the record is a partial error then why is it highlighted in red inside of the plots webview and not inside of the experiments webview or tree?

[maxagin]

Thumbs up @mattseddon for the details, this is partly completed already, however some needs to be addressed. Let me reflect on it and I will share the updated specs soon.

[maxagin]

If an experiment has a record level error: 4. The data in the record is left blank.

Without “!” or “...” in the cells? Right?

[mattseddon]

If an experiment has a record level error: 4. The data in the record is left blank.

Without “!” or “...” in the cells? Right?

... is what we currently show when there is no data for any cell in the table. This is true for error records or metrics on queued experiments.

[maxagin]

Hey. This is an updated overview including every detail. Please check it out and let me know if we are good here so I can create a full spec document. Thank you @mattseddon

[mattseddon]

Hey. This is an updated overview including every detail. Please check it out and let me know if we are good here so I can create a full spec document. Thank you @mattseddon

This doesn't answer a lot of my questions/concerns from above. We can discuss tomorrow in the planning meeting once the release is done.

[maxagin]

Very good @mattseddon !

[mattseddon]

@maxagin I would like to start working on this again, I looked at the spec and everything has been moved back into WIP. LMK when the next iteration is ready.

[maxagin]

Thank you @mattseddon for the heads up. I will try to finish it at the earliest convenience - there is still some work needed.

[dberenbaum]

@pmrowla Will there be any circumstance in which the whole experiment fails and still shows in dvc exp show results after the queuing changes are merged?

That might no longer be relevant in this view (although it might be shown if VS Code adds a view of the queue).

[maxagin]

Will there be any circumstance in which the whole experiment fails and still shows in dvc exp show results after the queuing changes are merged? #1636 (comment)

@mattseddon Does it mean we are not expecting to have any error at the row-level anymore?

[pmrowla]

By default failed experiments will not show in the table after the queueing changes, but we have discussed adding flags to exp show to control what gets displayed (i.e dvc exp show --failed) since it may still be useful for users to be able to see what params & deps were used in the experiment that failed

[dberenbaum]

See iterative/dvc#7592 (comment)

[maxagin]

Hey folks! The new specs version was published. Please use it and let me know if you have any concerns.

[mattseddon]

Provided feedback in the spec 👍🏻.

[maxagin]

@mattseddon the specs have been updated. Please review it and comment. Thanks!

[mattseddon]

The first iteration has been done. Let's close this issue and come back to it when we work on plots errors (the two specs are together).

[maxagin]

Good. Thank you @mattseddon !