Story: Experiments panel in the sidebar

[yalozhkin]

Story

The experiments panel lets users see their experiments when they code or focus on the plots. Also, it helps to navigate through all the experiments when the table is filtered (filters don't apply to the experiments panel) or too long (in case of expanded checkpoints).

Design

Preview:

More details and spec in Figma.

[rogermparent]

Relevant links:

• TreeItem API
• TreeView sample extension (node_modules tree browser)

I'm pretty sure we can only have one icon to the left, like the show/hide eye to the left of the dots (the arrow doesn't count, that's built into the view for anything with children, it being a TreeView and all.). Any workaround like using combined icons would likely end up hacky and wouldn't have to ability for different actions assigned to these elements.

I believe we can, however, put control icons like this on the right side of the menu like any other item via package.json contributes, seen as the pencil-shaped "Edit" button on each item in the sample extension.

[mattseddon]

Circling back to this original issue from #175 (comment).

As previously stated it is not going to be possible to match the existing design for this side panel tree view. Here is a screenshot of what we currently have:

In order to close this ticket we need to research all of the potential workarounds and implement a prototype solution for @yalozhkin to take a look at. We need to keep in mind that we need a way for users to quickly identify which experiments in the tree view are on display and which records they relate to in the real time plots / static images. Here is a screenshot of the expected UI (see design for more details):

I have already asked in the VS Code dev slack about possible options, here is the response that I got:

I initially thought that simply colouring the eyes would be a possible workaround but on more careful consideration I am thinking that we should research alternatives.

Two obvious options would be using the file decoration provider api to provide the appropriate decoration to each experiment (along with the open or closed eye as the single icon) and another would be a webview view in the sidebar. We do have an example of a file decoration provider already in the code base at extension/src/repository/decorationProvider.ts.

Things to consider:

1. The file decoration API takes a theme color and I am not sure if we will be able to get the hex color out so that we can pass it to any webview clients (perhaps we can have access to those colors already through css in the webview - see $row-bg-color: var(--vscode-editor-background) in webview/src/components/Table/styles.module.scss & this).
2. How are we going to store our list of colors to populate both the existing table webview, experiments tree view and future plots webview?
3. How do we generate the initial list of colors?

[sangxxh]

1. The file decoration API takes a theme color and I am not sure if we will be able to get the hex color out so that we can pass it to any webview clients (perhaps we can have access to those colors already through css in the webview - see $row-bg-color: var(--vscode-editor-background) in webview/src/components/Table/styles.module.scss & this).
2. How are we going to store our list of colors to populate both the existing table webview, experiments tree view and future plots webview?
3. How do we generate the initial list of colors?

Since the native VSCode API elements can't take dynamic hex color values instead of fixed theme colors, it can be a limitation. So vscode-dvc's own custom theme colors will need to be hard-coded in the package.json manifest.
There can be an unknown large number of experiments and we can't hard-code the colors to cover all possible experiments.

I think one way to resolve that is to repeat the colors:

• Provide a manageable number of custom theme colors in package.json, say 15 for example
• Repeat colors for every 15 experiments

It may not be too bad to repeat the colors? If we have too many colors, it can be hard for the user's eyes to focus and identify them; especially when they're looking at the colored icons across the tree view, table Webview and the future plots Webview.

This is how matplotlib provides the default legend colors as well, they're repeated for every 10 legends:

Regarding coordinating the colors between the sidebar and the webviews, @mattseddon, the Webview documentation that you linked to says that Webviews can access theme colors via CSS variables so we should be fine.

And with the icons, since there can be only 1 icon, we can consider keeping the dot icons instead of the eyes icons? Keeping the dot icons on the sidebar make it visually consistent with the dots on the table and plots Webviews. This should make it easier for the user to find experiments across the views.

Let me know your thoughts.

---

Regarding Webview, it does provide the flexibility to achieve the original design. The vscode Code Time extension's sidebar is a good example with a lot of customisation done using Webview.
However, @mattseddon has pointed out from our discussion that:

• There's too much overhead in setting up the Webview for the little customisation that we need.

• Unnecessary effort to simulate the native Tree View experience

• Difficulty with integration tests and all of the data that gets sent to them goes via the global window object

• And more from https://code.visualstudio.com/api/extension-guides/webview#should-i-use-a-webview doc:

  

There are too many trade-offs with Webview that I think we can agree to try to avoid using it.

[rogermparent]

perhaps we can have access to those colors already through css in the webview.

We do! See test-vscode-styles.scss for an informal list of CSS variables available in the Webview, It's only one theme, though so we can't really guarantee any consistency between themes, though I'm sure there'll be some consistent patterns to ensure that the colors are at least visible against the background.

How are we going to store our list of colors to populate both the existing table webview, experiments tree view and future plots webview?

In my opinion, the standout choice for keeping the feature functional and implementation simple is to simply assign colors based on index, as the purpose of the legend is to be able to match up experiments at a glance while looking at multiple views, which still works even if the colors "move" when experiments do.

How do we generate the initial list of colors?

Should we go with index-based color assignment, we can simply have a predefined list of theme colors that will repeat along the experiments, which will repeat when necessary.

Since the native VSCode API elements can't take dynamic hex color values instead of fixed theme colors, it can be a limitation. So vscode-dvc's own custom theme colors will need to be hard-coded in the package.json manifest.

Maybe, but we'll have to ensure that these custom defined colors get passed down to the Webview- otherwise, we're stuck reusing the ones VSCode provides us.

There can be an unknown large number of experiments and we can't hard-code the colors to cover all possible experiments.
I think one way to resolve that is to repeat the colors:
* Provide a manageable number of custom theme colors in package.json, say 15 for example
* Repeat colors for every 15 experiments
It may not be too bad to repeat the colors? If we have too many colors, it can be hard for the user's eyes to focus and identify them; especially when they're looking at the colored icons across the tree view, table Webview and the future plots Webview.

Absolutely! There's no way every Experiment will be able to get its own color. We could even go as low as 10 or 5 IMO, which would be enough to be used alongside names to distinguish Experiments at a glance.

And with the icons, since there can be only 1 icon, we can consider keeping the dot icons instead of the eyes icons? Keeping the dot icons on the sidebar make it visually consistent with the dots on the table and plots Webviews. This should make it easier for the user to find experiments across the views.
There are too many trade-offs with Webview that I think we can agree to try to avoid using it.

This has come up before, the design isn't an absolute so if we need to change it to work with the native TreeView API and avoid a Webview, that's totally okay.

Nice research so far!

[sangxxh]

perhaps we can have access to those colors already through css in the webview.

We do! See test-vscode-styles.scss for an informal list of CSS variables available in the Webview, It's only one theme, though so we can't really guarantee any consistency between themes, though I'm sure there'll be some consistent patterns to ensure that the colors are at least visible against the background.

If we use custom theme colors with contributes.colors, the user has the choice to customize the experiment colors to make them stand out more from other elements if needed. Useful in the case where the user has some funky vscode theme going on 😄

In my opinion, the standout choice for keeping the feature functional and implementation simple is to simply assign colors based on index, as the purpose of the legend is to be able to match up experiments at a glance while looking at multiple views, which still works even if the colors "move" when experiments do.

Should we go with index-based color assignment, we can simply have a predefined list of theme colors that will repeat along the experiments, which will repeat when necessary.

👍
This approach is simple to set up and convenient for any experiment view components to consume and extract the colors.

Maybe, but we'll have to ensure that these custom defined colors get passed down to the Webview- otherwise, we're stuck reusing the ones VSCode provides us.

I had a quick play around:

• Added a custom contributes.colors sample in package.json
• Applied the custom theme color via CSS variable in the table WebView

It works - we can use custom defined colors:

Absolutely! There's no way every Experiment will be able to get its own color. We could even go as low as 10 or 5 IMO, which would be enough to be used alongside names to distinguish Experiments at a glance.

Great! Agree with keeping it as low as 5 - 10 colors.

[shcheklein]

It may not be too bad to repeat the colors? If we have too many colors, it can be hard for the user's eyes to focus and identify them; especially when they're looking at the colored icons across the tree view, table Webview and the future plots Webview.

true. I have a concern though - how will users differentiate them on the plots dashboard (webview to right in the initial screenshot) if they select two experiments of the same color? Lines will look exactly the same, right?

[sangxxh]

true. I have a concern though - how will users differentiate them on the plots dashboard (webview to right in the initial screenshot) if they select two experiments of the same color? Lines will look exactly the same, right?

That's a good point. It is a trade-off with the repeating color approach. We're gonna have to come up with a way to mitigate it and help the user identify the labels. Eg. When hovering on the experiment on any view, highlight the plotline as well as the corresponding table row and the sidebar experiment label. What do you think?

Another alternative is to apply the index-based color assignment for active experiments only, but this can also look confusing to the user because when the user toggles different experiments on/off, the colors will keep "moving" and the user eyes may have trouble "registering" the new vs old color changes across the 3 views on vscode.

[shcheklein]

And with the icons, since there can be only 1 icon, we can consider keeping the dot icons instead of the eyes icons?

Could you please elaborate on this also? Eye and dot icons play a different role, so I'm not I understand the suggestion. Do you mean to keep only those colored dots?

the colors will keep "moving"

yep, agreed doesn't look good. Hovering could be pretty painful as well. Both options are suboptimal. Question here is - do we have some other options? (besides creating a webview). I don't know the answer, I'm just curious to see if there are other possibilities. We should try to check all the options.

How many total colors do we have?

[sangxxh]

Eye and dot icons play a different role, so I'm not I understand the suggestion. Do you mean to keep only those colored dots?

I was thinking of keeping the dot icons only on the left of the experiment title. For the eyes, one option is to squeeze them in the tree view action buttons into the existing action design:

The problem is the action items only show up on hover or when a tree item is selected.

Another thing I should look into though is how do VSCode's Source Control sidebar manage to get the "M", "U" characters below to appear to the right of the tree item. If we can have either the dot or the eyes icon there, it could be a way to solve the problem:

Let me research it.

the colors will keep "moving"

yep, agreed doesn't look good. Hovering could be pretty painful as well. Both options are suboptimal. Question here is - do we have some other options? (besides creating a webview). I don't know the answer, I'm just curious to see if there are other possibilities. We should try to check all the options.

I think the Source Control example I mentioned above is worth looking deeper into for now. I'll look into it first then circle back on this if I come up with anything else. At the high level, I believe we've tried to consider all VSCode APIs though.

How many total colors do we have?

We're looking at having 5 - 10 colors. I'm going to try with 5. The mock-up Figma is showing 4 colors as an example btw.

[shcheklein]

We're looking at having 5 - 10 colors. I'm going to try with 5. The mock-up Figma is showing 4 colors as an example btw.

Sorry, my question was confusing. I meant, what is the maximum number of colors we could utilize. I'm curious since it might affect the product decision here.

[sangxxh]

Sorry, my question was confusing. I meant, what is the maximum number of colors we could utilize. I'm curious since it might affect the product decision here.

Nw. There's no hard maximum, but we were looking at 5 - 10 colors since it's more manageable - because we have to declare these colors manually in the VSCode extension manifest. If we want to experiment with more colors - it's just a matter of hard-coding more theme colors to the vscode extension manifest (package.json).

[shcheklein]

So, how about we do something like this. Pick 100+ colors. Instead of an eye icon we could use an empty circle by default and one of the pre-assigned colors when user toggles it on. 100+ should be enough to reduce the probability of two experiments colliding to the minimum. It won't be noisy since we won't be showing them by default only when you select them. WDYT?

I think it's fine to start with something like this from the product perspective. We then should involve @yalozhkin into the discussion to see if we have some other ideas.

An alternative to try that would also work for us. Need to research if it's possible. Place the colorful circle to the right (instead of the letters) and an eye icon in place of the circle. Or other way around.

No matter how do we place icons (we essentially have quite a few possible combinations if we don't go with a custom webview -which we all don't like), we need more colors to my mind to avoid collisions and/or dynamic reassignments. Algorithm should be stable as new experiments come, old being deleted, etc.

[sangxxh]

So, how about we do something like this. Pick 100+ colors. Instead of an eye icon we could use an empty circle by default and one of the pre-assigned colors when user toggles it on. 100+ should be enough to reduce the probability of two experiments colliding to the minimum. It won't be noisy since we won't be showing them by default only when you select them. WDYT?

I think it's fine to start with something like this from the product perspective. We then should involve @yalozhkin into the discussion to see if we have some other ideas.

Yes, that sounds like a good start for a PoC and we can iterate further from there.

An alternative to try that would also work for us. Need to research if it's possible. Place the colorful circle to the right (instead of the letters) and an eye icon in place of the circle. Or other way around.

Whether it's the letter or the dot icon (if it's possible), there's a UI issue that they can be hidden when the sidebar panel is not wide enough:

I think this is a drawback that may annoy the user, unfortunately...

[sangxxh]

@mattseddon @rogermparent @shcheklein I saw a critical issue when adding 100 colors to vscode-dvc manifest file. It causes the extension to freeze the whole vscode's extension host with the warning UNRESPONSIVE extension host. Both DVC and other extensions just keep showing the loading animation:

The same error can be seen when running tests - it causes the tests to hang and never finishes the process as well (I had to kill the process in the screenshot but the test could be hanging forever):

For now, I reduce the manifest to include 50 colors and it works, but we should be cautious of this. May escalate an issue to vscode.

[mattseddon]

@hantatsang

@mattseddon @rogermparent @shcheklein I saw a critical issue when adding 100 colors to vscode-dvc manifest file. It causes the extension to freeze the whole vscode's extension host with the warning UNRESPONSIVE extension host. Both DVC and other extensions just keep showing the loading animation

There are theme colors already built in to VS code that we could utilise. For example we already rely on the git-colors in our existing sourceControlManagement view / decorationProvider as we want to match our SCM view to the native one. I think that we could lean on say this list so that we don't have to declare 50 of our own colors / crash the extension host (if 100 crashed the VMs extension host then 50 would probably crash a lesser machine's). We should definitely (at least) use the existing chart colors. I would say that we could comfortably use the git colors too (as we already have a dependency on them, that's ~20 that we don't need to hardcode. Obviously this suggestion will need more investigation.

Whether it's the letter or the dot icon (if it's possible), there's a UI issue that they can be hidden when the sidebar panel is not wide enough:

☝🏻 In terms of this issue I think the dot being hidden is ok AND the experiments tree is not nested like the file explorer one. Which means: 1. That it is less likely that the dot will be hidden as there is more real estate available & 2. As long as the element's text is colored and the user can visually link the element to the table / plot then I think that would be fine.

I am going to review your PR now. I had a quick look in and it looks like you haven't updated the storybook (screenshot at the bottom in case you don't have access). I'll mention it again when I go through the PR. I can't blame you as the CI wouldn't have run in your clone.

Cheers,

[mattseddon]

Closing as we may use this task for another candidate