Fixing the Experiment tracking set up docs

[noklam]

Hey viz team! I am not sure where is the best place to raise this so I just use GH issue, please let me know if there are preferred channel with these draft issue.

Currently we have 2 links mentioning experiment tracking. If you do a google search and start from the 1st link, it will led you to a broken experiment tracking set up, as it doesn't state the changes required in settings.py for SqlliteStore. Another common trap is users forgot the namespace in catalog and couldn't see the metrics in Viz. I don't know what's the best way to reorganize the docs yet but I sympathize with the users it is not easy to get it right.

1. https://kedro.readthedocs.io/en/stable/logging/experiment_tracking.html
2. https://kedro.readthedocs.io/en/latest/tutorial/set_up_experiment_tracking.html

Evidence
https://discord.com/channels/778216384475693066/846330075535769601/1015020113163976794

This is a chat I have on Discord, but I have answered a few questions similar before, so I think maybe we could do something about the doc. Or should we start consider making Sqlitestore the default?

[tynandebold]

Thanks for this, Nok.

I think we need to have a larger rethink about the general experiment tracking documentation. Perhaps this ticket can morph into that? I know that @AhdraMeraliQB is working on a couple of additions to experiment tracking docs right now, and it's getting to a place where we may need to streamline. Lastly, @AntonyMilneQB did have a thought about this in another ticket that I'll try and find and then add here.

References:

• Add documentation on how to visualise Matplotlib plots on Kedro-Viz kedro#1820
• Update docs on experiment tracking to add new plot feature kedro#1819

[antonymilne]

Definitely sympathise with the difficulties here also 👍

In terms of documentation, as per my comments in kedro-org/kedro#1820, I think kedro-viz documentation has outgrown the Tutorial section of our docs (which originally just covered the spaceflights tutorial really). We need to find a better place for it to live. Even better, we need a whole different way of documenting kedro-viz because expecting a user to read through text documentation to figure out how to use a point and click tool feels pretty bad. e.g. we should have tooltips on the kedro-viz UI that help users see what to click, or a little guided tour button or something. But that's a big job.

---

Not related to documentation, but on the SQLiteStore point:

• should we make SQLiteStore the default SESSION_STORE_CLASS? Not while it lives on kedro-viz. Should we make it the default commented-out suggestion? I think yes, but Update default suggestions in settings.py to ones that work kedro#1538 states otherwise, that we should wait until it lives on kedro core.
• should we move SQLiteStore to kedro? I believe this was always @limdauto's intention. It was only developed in kedro-viz for quick release and iteration and has not moved across because it's still "experimental" in some sense. Edit: I gather @idanov thinks SQLiteStore should not move to the kedro side since we don't want a sqlite dependency there.
• regardless of where it lives, SQLiteStore has existed for nearly a year now, and we need to figure out what happens with it... What its role is relates closely to questions of 1. what should happen to the tracking datasets (Ability to link plots to an experiment kedro#1626 (comment)), 2. how we might enable a search functionality (Richer search for experiment tracking #1039), 3. multi-user experience (Allow users to collaborate while using experiment tracking #1218). It feels like we're a bit stuck here currently and personally I would love it if we could resolve some of this sooner rather than later

Edit: split this off into a new issue #1217 so that this one can remain just on documentation.

[noklam]

In terms of documentation, as per my comments in kedro-org/kedro#1820, I think kedro-viz documentation has outgrown the Tutorial section of our docs (which originally just covered the spaceflights tutorial really). We need to find a better place for it to live.

I agree with this, I actually think spaceflights itself is quite big, if you get something wrong in the middle, often you have to start from Step 1 and read through everything.

I think users come to this documentation with 1 of these intentions

1. Trying to experiment with the experiment-tracking feature and play around with the UI
2. Adding experiment tracking to their existing project

In that case, some checkpoint/a new starter/demo repository that people can just clone and run is more preferable. It's much easier to have a working example and tweak it to see what changes/breaks than to have a broken repository and try to get it right.

The main concern of this is we have to maintain yet one more starter/repository.

[antonymilne]

The checkpoint idea would be neat to do as using a branch so you could do e.g. kedro new -s spaceflights -c experiment-tracking. But this is tricky to maintain because every time we update a starter we have to remember to update two branches, and it's already a bit of a burden to maintain the starters. Unless there's some cunning way to make some git reference experiment-tracking which somehow points to branch main + extra experiment-tracking commits so that it always keeps up to date automatically with main.

[noklam]

@AntonyMilneQB Slightly off-topic, I think this is more general than viz and apply to other tutorials as well. I agree that is a burden to maintain. However, I would argue that this is not an additional effort, because we are already doing it, but not in a smarter way.

Basically, the setup that we have has 6 starters and a lot of "checkpoints" that only exist in the documentation. So we are maintaining these "diff" already, but only inside the markdown file. I spend 20 minutes today just reading the docs and setting up the project to reproduce an error. So every time someone who needs to update these docs should have already manually go through this process (get latest tutorial + manually applying the diff suggested in the docs)

See kedro-org/kedro#1843, we are missing the import which would

[merelcht]

To do:

1. Instructions on https://kedro.readthedocs.io/en/stable/logging/experiment_tracking.html are incomplete. Make sure a user following these can actually get experiment tracking up and running. Need to add set up of SessionStore.
2. Double check https://kedro.readthedocs.io/en/latest/tutorial/set_up_experiment_tracking.html actually works by following all steps.

[tynandebold]

Closed via kedro-org/kedro#1893.