Shareable URLs for Kedro-Viz

[yetudada]

Description

This idea focuses on creating a way for users to share versions of their current Kedro-Viz pipeline visualisation for others to explore. We've often seen this request for interactive sharing of a pipeline visualisation arise from technical "translators" (people who are technical enough to run kedro-viz. This group wanted ways to share their pipeline visualisation with people who could not make a copy of the Kedro project, install Kedro-Viz and run Kedro-Viz to see the pipeline (normally other senior stakeholders).

This item is a high-priority issue because there is no other way to open up Kedro-Viz to non-technical users unless we make it possible for them to access Kedro-Viz without a technical or "translator" user.

Users currently solve the problem of sharing a version of their current Kedro-Viz pipeline by:

• Download a PNG version of their Kedro-Viz pipeline
• Screenshotting a version of their Kedro-Viz pipeline
• Creating a hosted Kedro-Viz (one user did this through a Dash app)
• And in the past, they used a link to Studio, an internal product, to see an interactive version of their plots

I've focused this issue on pipeline visualisation sharing but, we will also need to think about what this means for Collaborative Experiment Tracking users.

Possible user experience

Based on exporting something from kedro-viz

A suggestion for a possible user experience based on the CLI could be:

1. User sets up the pipeline in a Kedro project
2. User installs Kedro-Viz with pip install kedro-viz from the CLI
3. User runs kedro viz export
4. User receives a URL which can be shared with other team members

A suggestion for a possible user experience on the UI could be:

1. User sets up the pipeline in a Kedro project
2. User installs Kedro-Viz with pip install kedro-viz from the CLI
3. User runs kedro viz and opens up the UI
4. User clicks on an icon to generate the URL
5. A pop-up appears with the generated link
6. User receives a URL which can be shared with other team members

Based on interacting with the project GitHub repo

Another user experience could be that a user:

• Heads to a website
• Sees a form which asks for a link to a GitHub repo and what branch needs to be visualised
• User inserts link to GitHub repo and which branch
• User gets a generated link

I was mucking around on Powerpoint and made this, it's similar to the way that Streamlit works. Please do not take this as a design but let it inspire you. This design also has security features because it's behind a login.

Other user considerations

Additional research on this feature has shown additional requirements that users will have:

• The security of shared links will be an essential requirement; i.e. we'll need to think of a workflow where access to the visualisation is restricted to certain people
• We'll need to figure out how to host the URLs too; our users have struggled to do this on their own
• One user flagged a need to ship a certain branch in their git-workflow i.e. the team might have broken the pipeline in develop but the pipeline visualisation still works on main
• One user wanted an ability to see a changelog (or see what's changed between the pipeline) on Studio people used to be able to see timestamped versions of Kedro-Viz when a new version was submitted but not a delta between pipelines

And additionally, there are outstanding questions like:

• What happens if a user updates the pipeline? Does this mean they need to send a new link or will their current one update?
• What happens if Collaborative Experiment Tracking is enabled?
• Users have also spoken about being able to edit parameters and re-do the run. What would happen if we enabled Supporting the reporting use case on Kedro-Viz #1279?

[tynandebold]

Notes from meeting with @stephkaiser and @NeroOkwa:

Discussion

There are two high-level approaches we could take:

1. A static HTML approach (Viz as a React-component) where the user hosts the app
2. A platform approach, a la Streamlit, where we host the apps

Pros of number 1:

• Maintenance: offload all infrastructure decisions and upkeep to the user
• Flexibility: users are free to use their preferred hosting provider
• Security: management of risk and access is up to the user

Cons of number 1:

• The current iteration of Kedro-Viz as a React component doesn't work with experiment tracking

Pros of number 2:

• We control everything

Cons of number 2:

• We control everything
• The team becomes responsible for the platform and all maintenance
• Time to initial release is very long
• Large workload to get up and running
• Infrastructure costs: how do we pay for those?

Conclusion

We think number 1, a static HTML approach, is best. Furthermore, it's a validated solution in the wild, with Brix, AI4DQ, and CustomerOne all using the feature in their own way.

On experiment tracking, unless we build it in, option 1 doesn't account for it being present in the hosted version of the application. The deployed version of the app would only show the flowchart. For an initial release, we think this tradeoff is a good one to make for a few reasons:

• The vast majority of Kedro-Viz users don't use experiment tracking. Supporting research here
• From user interviews, those who have requested this feature only reference the flowchart
• We can more quickly get this feature out and into the hands of users
• It's easy to guide the user in such a way that makes them realize they're only able to share their flowchart. Using language like Deploy your Kedro-Viz flowchart now! is explicit and hopefully eliminates confusion.

Implementation

Two UX patterns were floated, taken from @yetudada's writing above:

1. A UX based on the CLI
   • Our Full-code user archetypes would be the primary users of this flow
2. A UX based on a button or panel in Kedro-Viz's UI
   • Our Low-code user archetypes would be primary users of this flow

The option of a hosted website where a user enters GitHub repo details into a form and submits a button to receive a deployed URL was discussed and wasn't favored. We'd need to design, build, deploy, and host that application somewhere, which doesn't seem necessary or needed at this stage. This could be done later, though.

Other considerations

For our third user archetype (No-code users), they're the ones who'll receive the deployed link from our other two archetypes. We need to consider this user's journey and what they can do when they receive the link, if anything.

Lastly, inspiration for similar UI/UX sharing and collaborating patterns can be found in products like Figma, Miro, and Google Docs/Sheets.

[rashidakanchwala]

The current iteration of Kedro-Viz as a React component doesn't work with experiment tracking

hey, I am not sure why it wouldn't work. We could try to do what we do in Flowchart -- we can export the experiments as json and put them in a folder. and in the code, we can say that apollo query reads from the graphql backend server but if theere are json files then it reads from the json.
Ofcourse that would mean for any new experiments we would have to deploy the latest version of the app again but maybe we can have a version 2.0 where circleCI does that for us --as @idanov suggestion sometime ago.

[rashidakanchwala]

I still do like the approach of doing it in phases, and focusing on just the flowchart first and then experiment tracking :) . Also we already do Collaborative Experiment Tracking that solves the problem of sharing experiments.

[tynandebold]

The current iteration of Kedro-Viz as a React component doesn't work with experiment tracking

hey, I am not sure why it wouldn't work. We could try to do what we do in Flowchart -- we can export the experiments as json and put them in a folder. and in the code, we can say that apollo query reads from the graphql backend server but if theere are json files then it reads from the json. Ofcourse that would mean for any new experiments we would have to deploy the latest version of the app again but maybe we can have a version 2.0 where circleCI does that for us --as @idanov suggestion sometime ago.

It wouldn't work without making changes. When I write, "The current iteration of Kedro-Viz as a React component..." I mean the current version we have at this moment in time, the one that's released.

[amandakys]

From the design perspective I had a few thoughts:

From @tynandebold's comments here it seems like there would be 2 entry points into the shareable URL feature.

1. CLI
2. UI

These 2 entry points would target different user groups and also provide a different level of customisability and/functionality.
For example, we might not expect the low-code users to be responsible for setting up hosting.

If we take the preferred approach where the user is responsible for hosting the app, then the exit point might require further clarification, as the initial idea of being able to supply the user with a working URL will not work if they haven't set up appropriate hosting. (I think I'm also missing some context on what it means to export Viz as a React Component)

From my perspective, it seems like the shareable URL feature would first export "something" then direct users to some documentation on how we'd recommend hosting it (or use their own preferred hosting method). Once the hosting is set up, we might want to:

• allow the user to provide the URL, then have it exposed via the UI i.e. "copy url here"
• provide the ability to push updates and/or auto-redeploy, but this seems optional especially since we probably can't cover all possible hosting methods/providers.

The follow up flow that would allow the project dev's to supply information via the code base or CLI to improve accessibility to the low-code users, so they can directly copy a functioning URL from the UI.

In order to build out a more comprehensive user journey, it would be good to clarify what the expected inputs and outputs are and what format they'd take.

[tynandebold]

From my perspective, it seems like the shareable URL feature would first export "something" then direct users to some documentation on how we'd recommend hosting it (or use their own preferred hosting method).

Given this puts a lot of the burden on the user, I don't think this is exactly what we want. I hope it's makes more sense after I give my take on what the inputs and outputs are:

Inputs

Authentication/CLI credentials for hosting a static website. To start, I'd say we focus on AWS S3, though it could really extend to things like Netlify, Vercel, and others.

The user journey for both the CLI and UI entry points would require these credentials as inputs, and ideally nothing else, since the burden of security/user access/etc would all fall to the user. Which leads to...

Outputs

A URL to the hosted version of Kedro-Viz. Simple as that.

I want us to try and do as much as we can in the background to make this like a one-click deploy.

Prerequisites

The user will need an AWS account with S3 access keys (or credentials to Netlify/Vercel/etc).

---

Let me know if that clears some of this up or prompts more questions.

[NeroOkwa]

Here are some additional user feedback regarding this pain point (LINK):

• “I feel that being able to share a link on the look of the pipeline will help me and my translator a lot in the process of explaining to the client and EMs what the current state is”.
  “Another example: one of my pipelines outputs a series of very fancy and beautiful plots, but since I'm skipping the management meetings my CST cant share those plots to the clients on demand only if I'm there and ready to present... this has been kinda bad because they are now creating counterpart plots on excel (not as fancy) but it feels wrong. If only they were able to have access to my Kedro-Viz to present, that would be a good value bringer for everyone".

• “The second challenge is the download of it. Now I don't actually know how we can solve this because for example, there's a huge pipeline. It downloads an SVG or a PNG, which so like, it's, it's like a huge image, which we can see probably on a big monitor. I'm really not sure how we can squeeze it into one small window, because if you squeezed, the readability becomes like very difficult. You cannot basically see what the feature groups are ...If we can do something around improving the downloadable options of that".

• “It would be nice to embed Kedro-Viz in a Sphinx documentation or something like that. So for example, in the current project we are building Sphinx documentation, having a nice hosting on GitLab documentation. It would be nice to have a page there where the user can go and just see and interact with the pipeline".

[tynandebold]

@amandakys do you think we should show the feature hint cards (aka visual walkthrough) on the deployed version of Viz? I'm not sure. Curious what you think.

cc @stephkaiser for when she's back.

[amandakys]

My first instinct is yes - it would be best to reduce how much we diverge between two versions.

If we think about the target users of the deployed viz, they are more likely to be new users, non-technical or even external users who are less familiar with viz, so the option to have a walkthrough or guidance in general would be valuable. Since we don't have a walkthrough tutorial at the moment though, do you specifically mean the tooltips prompting new features? These might be less immediately useful, but I stick with my point before about minimising divergence.

[stephkaiser]

Agree with Amandas comment above btw 👍

Linking to shareable URL designs here: https://www.figma.com/file/3kSpvIO1veLKfy9qHxuXwF/Kedro-WIP?type=design&node-id=171-124&mode=design

Shareable URL flows here - https://miro.com/app/board/uXjVM8oNLys=/?share_link_id=39391792422

Loading animation flow here: https://www.figma.com/proto/3kSpvIO1veLKfy9qHxuXwF/Kedro-WIP?page-id=171%3A124&type=design&node-id=360-16494&viewport=-782%2C347%2C0.15&t=1l49SE6C4kwnGkTG-9&scaling=min-zoom&starting-point-node-id=360%3A16494&show-proto-sidebar=1