Experiment Table UI: visually separate baseline from experiments for readability when sorting?

[bobertlo]

Would be possible to somehow visually separate the baseline commit from the experiments in the table just to help with readability when sorting, since the baseline is not sorted with the experiments?

My main observation is that while It is pretty clear currently how to read the table when sorting by a performance metric, it is less clear when sorting by an input parameter and trying to correlate performance metrics since the baseline commit is not sorted or visually separated.

Maybe there could just be a space like there is between the workspace and the baseline commit? I see this could interfere with with the drop down function though and I am sure there are a lot of other possibilities to consider.

[shcheklein]

@bobertlo could you please clarify, baseline commit - you mean the workspace, the very first one?

[bobertlo]

@shcheklein

No, sorry I was unclear I mean the checked out HEAD commit. It says baseline in the help message somewhere and I really love that concept. In this picture I am working on a branch named baseline1 running a sweep of the dropout parameter and sorting by that parameter:

It looks like a regular part of the table but does not sort. I have the table really wide here, but especially while figuring out how to use everything I would end up scrolling over and cutting off the left column with the labels. I feel really silly/picky explaining it now, but it hurts a weird little part of my brain 🙃

If there was a space separating it like the workspace line, or it sorted with the experiments or was highlighted it would make a lot more sense to me? Again it is a tiny thing.

It has been an overwhelmingly positive experience so far and I'm really excited for you guys. This is like the only feedback I have that is not super positive.

[shcheklein]

@bobertlo have you tried Studio, btw?

If feels we are coming to the (what we call in Studio) "delta-mode" with this :) Any experiment (by default its HEAD) can be designated as a "baseline" + you could switch the table into "delta" mode where each value in the table will be calculated as a difference to the baseline:

https://studio.iterative.ai/user/shcheklein/views/example-get-started-yj0q1nba3g?mode=delta

[bobertlo]

Yes! I really like how Studio and this extension compliment each other. There were some really good design decisions there. I was actually wishing there was a good way to do parameter sweeps in Studio (not that you can't) when I happened to see the extension.

Everything is there and working for me, but I'll often just reproduce the HEAD as an experiment because it's really important for me to have them in order in the table to be able to click through and toggle the traces on the plots. I thought this might be useful, but maybe it would mess up someone else's eccentric workflow 😉

[shcheklein]

Yes, I didn't mean that it should be unique to Studio, rather that we should probably consider implementing this here step-by-step - baseline highlight, ability to set any experiment as a baseline, delta mode, column values highlighting.

[bobertlo]

Yes, I didn't mean that it should be unique to Studio,

Oh great! I wasn't sure about the planned scope and when testing this it feels like a mature design.

baseline highlight, ability to set any experiment as a baseline, delta mode, column values highlighting.

Awesome! Those all sound great. Baseline and column highlighting sound especially high value for the effort. I wonder if they could somehow integrate with the plots to highlight the baseline trace and/or generate a legend with the selected columns?? It could be so cool if there is a way to present the second part intuitively.

[maxagin]

Would be possible to somehow visually separate the baseline commit from the experiments in the table just to help with readability when sorting, since the baseline is not sorted with the experiments?

Hey @bobertlo. Thank you for your feedback. Please, can you draw a use case? How the visual separation of the baseline will help for the sorting? Also if you can attach a short video of how you are currently using it, would be helpful.

[bobertlo]

@maxagin Hey! Sorry I haven't had as much bandwidth lately so it took me a while to get back.

To start out, I’m not convinced the visual separation would be that helpful anymore, mainly because the names are "sticky" and it was more confusing without those elements. Again I usually exp run the original parameter value again just to get it to sort in the table. Again, it is not a big deal but maybe if the “folder” that says the branch name was a blank column and there was a “HEAD” entry that sorted it would be better for my OCD but it’s really not a big deal.

@shcheklein mentioned maybe the information being inconvenient to consume from the table could just mean it would be useful to have it somewhere else. The idea of having being able to “star” a couple columns from the table (ie a hyper-parameter and a performance metric) and display those in the “ribbon” seems like it could be really nice (but it would be less helpful I would think if they were not sorted.)

I'm attaching a picture of my workspace since my internet really does not want to send a video. I selected the columns that I am looking at here for clarity, but I usually have everything enabled since when I select between different model directories it all gets reset anyway.

[maxagin]

Thank you very much @bobertlo ! This is helpful feedback.
Regarding the video file you have mentioned, I really would like to have it, would it help if I share with you my email - maginsky@iterative.ai? Thank you!

[shcheklein]

Addressing it here #2458 . We'll try to make all commits look a bit different in some way.