Visualize pipeline objects in notebook

[ravi-kumar-pilla]

Description

Resolves #1993

NOTE: The bundle URL will be updated once #2268 is merged

Development notes

• Created a class NotebookVisualizer and a method show responsible for visualizing Kedro-Viz using the esm bundle in notebook
• Added load_data_for_notebook_users and load_and_populate_data_for_notebook_users methods to kedro-viz -> integrations -> notebook -> data_loader.py
• Added initialize and reset methods in data_access_manager for reuse
• Added few utility functions and classes
• Update release note, tests and gcp load balancer doc link
• Removed few GraphQL checks from workflow as they were failing

QA notes

• All tests should pass
• For manual testing, open a jupyter notebook and try -

from kedro.pipeline import pipeline, node

def dummy(ds1):
   return ds1
    
n0 = node(dummy, 'flights', 'processed_flights')
dummy_pipe = pipeline([n0])

from kedro_viz.integrations.notebook import NotebookVisualizer
NotebookVisualizer(dummy_pipe).show()

• You can also test demo_project pipelines, try -

from kedro_viz.integrations.notebook import NotebookVisualizer
from demo_project.pipeline_registry import register_pipelines
demo_pipe = register_pipelines()

# Since globalNavigation depends on localStorage, the option is not working.
 
NotebookVisualizer(pipeline=demo_pipe, options={ "display": {
                "expandPipelinesBtn": False,
                "exportBtn": False,
                "labelBtn": False,
                "layerBtn": False,
                "metadataPanel": True,
                "miniMap": False,
                "sidebar": False,
                "zoomToolbar": False,
            },
            "expandAllPipelines": False,
            "behaviour": { 
                "reFocus": False,
            },
            "theme": "dark",
            "width": "100%",
            "height": "600px",   
            }).show()

Testing Results :

Jupyter lab:

Databricks:

Marimo:

VS Code

Checklist

• Read the contributing guidelines
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added new entries to the RELEASE.md file
• Added tests to cover my changes

[ravi-kumar-pilla]

@astrojuanlu for security reason (accessing localStorage), esm bundle does not allow globalNavigation to be True. Though for our use case this is not a blocker but if it is an issue, we can fix it later or go with umd. I restored globalNavigation and it should work now.

On databricks:

[astrojuanlu]

Last commit fixed things it seems.

VS Code

This is how it looks like for me, and I have a big screen (1512 x 982). Any chance we can not show the logs? And maybe make the area a bit smaller?

Jupyter Notebook

Same thing

And I'm seeing some warnings and errors in the terminal:

[E 2025-02-11 18:31:21.922 ServerApp] Uncaught exception in write_error
    Traceback (most recent call last):
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/tornado/web.py", line 1788, in _execute
        result = method(*self.path_args, **self.path_kwargs)
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/tornado/web.py", line 269, in _unimplemented_method
        raise HTTPError(405)
    tornado.web.HTTPError: HTTP 405: Method Not Allowed
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jupyter_server/extension/handler.py", line 29, in get_template
        template = cast(Template, self.settings[env].get_template(name))  # type:ignore[attr-defined]
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1016, in get_template
        return self._load_template(name, globals)
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 975, in _load_template
        template = self.loader.load(self, name, self.make_globals(globals))
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jinja2/loaders.py", line 126, in load
        source, filename, uptodate = self.get_source(environment, name)
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jinja2/loaders.py", line 209, in get_source
        raise TemplateNotFound(
    jinja2.exceptions.TemplateNotFound: '405.html' not found in search path: '/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/notebook/templates'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jupyter_server/base/handlers.py", line 740, in write_error
        html = self.render_template("%s.html" % status_code, **ns)
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jupyter_server/extension/handler.py", line 93, in render_template
        template = cast(Template, self.get_template(name))  # type:ignore[attr-defined]
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jupyter_server/extension/handler.py", line 32, in get_template
        return cast(Template, super().get_template(name))  # type:ignore[misc]
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jupyter_server/base/handlers.py", line 662, in get_template
        return self.settings["jinja2_env"].get_template(name)
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1016, in get_template
        return self._load_template(name, globals)
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 975, in _load_template
        template = self.loader.load(self, name, self.make_globals(globals))
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jinja2/loaders.py", line 126, in load
        source, filename, uptodate = self.get_source(environment, name)
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jinja2/loaders.py", line 209, in get_source
        raise TemplateNotFound(
    jinja2.exceptions.TemplateNotFound: '405.html' not found in search paths: '/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jupyter_server', '/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jupyter_server/templates'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/tornado/web.py", line 1298, in send_error
        self.write_error(status_code, **kwargs)
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jupyter_server/base/handlers.py", line 742, in write_error
        html = self.render_template("error.html", **ns)
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jupyter_server/extension/handler.py", line 98, in render_template
        return cast(str, template.render(**ns))
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1295, in render
        self.environment.handle_exception()
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 942, in handle_exception
        raise rewrite_traceback_stack(source=source)
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/notebook/templates/error.html", line 1, in top-level template code
        <!doctype html><html><head><meta charset="utf-8"><title>{% block title %}{{page_title | e}}{% endblock %}</title>{% block favicon %}<link rel="shortcut icon" type="image/x-icon" href="/static/favicons/favicon.ico">{% endblock %}<script defer="defer" src="{{page_config.fullStaticUrl}}/main.407246dd27aed8010549.js?v=407246dd27aed8010549"></script></head><body class="jp-ThemedContainer">{% block stylesheet %}<style>/* disable initial hide */
      File "/Users/juan_cano/Projects/QuantumBlackLabs/tmp/spaceflights/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 490, in getattr
        return getattr(obj, attribute)
    jinja2.exceptions.UndefinedError: 'page_config' is undefined
[W 2025-02-11 18:31:21.960 JupyterNotebookApp] 405 OPTIONS /notebooks/srcdoc/api/deploy-viz-metadata (@127.0.0.1) 76.06ms referer=None

marimo

This is where it looks best, interestingly enough. Still, maybe the area is too big.

[ravi-kumar-pilla]

Hi @astrojuanlu , Wow ! Thanks for the quick test and feedback. I can check if we can -

1. Customize the height and width accepting it as user options (pretty much doable)
2. Hiding logs (need to check on this)
3. The errors on the console - few seem to be from .venv (not sure it is related to the bundle) but there will be some security errors related to localStorage which seem unavoidable for the moment.

I will fix 1, 2 for now
Thank you

[rashidakanchwala]

Hi @ravi-kumar-pilla ,

Does it make sense to separate how we:

• Load Kedro-Viz from a Kedro project via a FastAPI server
• Load Kedro-Viz in a notebook by generating JSON and bundling it using ESM

Currently, notebook-related functions are in data_loader.py and server.py, making these files larger and somewhat out of place. Would it be better to create a new folder under integrations called notebooks and move the visualizer and loader files there for better separation?

Let me know your thoughts!

[astrojuanlu]

1 more thing @ravi-kumar-pilla : do you think having

def _repr_html_(self):
    from kedro_viz.integrations.notebook import NotebookVisualizer

    return NotebookVisualizer(self).generate_html()

would make

In [2]: pipeline
[Just show the HTML]

work with what's currently in this PR?

(That way the user doesn't even have to do any extra imports for this to work)

[ravi-kumar-pilla]

1 more thing @ravi-kumar-pilla : do you think having

def _repr_html_(self):
    from kedro_viz.integrations.notebook import NotebookVisualizer

    return NotebookVisualizer(self).generate_html()

would make

In [2]: pipeline
[Just show the HTML]

work with what's currently in this PR?

(That way the user doesn't even have to do any extra imports for this to work)

Love the idea. Sure we can do that !

[rashidakanchwala]

thanks @ravi-kumar-pilla , works amazing! I only tried it on jupyter notebooks, but did a lot of iterative building pipeline and it was great.
codewise it is in much better state. thankssss