Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLI command kedro viz build added #1697

Merged
merged 29 commits into from
Jan 16, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
66f177d
CLI command kedro viz build added
jitu5 Jan 4, 2024
cc7c144
Lint fix
jitu5 Jan 4, 2024
a389a83
lint fix
jitu5 Jan 4, 2024
6528d36
Lint fix
jitu5 Jan 4, 2024
5a9f2e1
add mypy ignore
jitu5 Jan 4, 2024
b86b578
Missing build file added
jitu5 Jan 8, 2024
40003f8
Lint error fix
jitu5 Jan 8, 2024
0c879a8
BaseDeployer class added
jitu5 Jan 9, 2024
5145ba4
Unused code removed
jitu5 Jan 9, 2024
26f6275
Fix lint issue
jitu5 Jan 10, 2024
c7ed79c
added base_deployer
rashidakanchwala Jan 10, 2024
11ac321
latest
rashidakanchwala Jan 10, 2024
42af3af
add deployer factory
rashidakanchwala Jan 11, 2024
7ad0905
Test and comments of deployers updated
jitu5 Jan 12, 2024
2a4e00f
fix lint
ravi-kumar-pilla Jan 15, 2024
ecb6a05
remove circular dependency
ravi-kumar-pilla Jan 15, 2024
90c285b
Tests and lint fixes
jitu5 Jan 15, 2024
eadb85c
Merge branch 'feature/cli-build' of https://github.com/kedro-org/kedr…
jitu5 Jan 15, 2024
d579bb7
Import fixes
jitu5 Jan 15, 2024
e8e25fa
Lint fixes
jitu5 Jan 15, 2024
0ad5301
Lint fixes
jitu5 Jan 15, 2024
98f11b2
Merge branch 'main' into feature/cli-build
jitu5 Jan 15, 2024
2ffc777
fix lint
rashidakanchwala Jan 15, 2024
4c977a6
fix return on deploy_get_url
rashidakanchwala Jan 15, 2024
d51634f
Exception and class method updated
jitu5 Jan 15, 2024
35b9775
Merge branch 'feature/cli-build' of https://github.com/kedro-org/kedr…
jitu5 Jan 15, 2024
0297a5f
User entered platform added in error
jitu5 Jan 16, 2024
b27cdaf
Fix test
jitu5 Jan 16, 2024
932e654
Merge branch 'main' into feature/cli-build
jitu5 Jan 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,12 @@ Options:
-h, --help Show this message and exit.
```

To create a build directory of your local Kedro-Viz instance with static data from the command line, use the following command from the root folder of your Kedro project:

```bash
kedro viz build
```

Comment on lines +151 to +155
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we have some docs explaining how to view this build?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, in another PR we will focus on the build docs

### Experiment Tracking usage

To enable [experiment tracking](https://docs.kedro.org/en/stable/experiment_tracking/index.html) in Kedro-Viz, you need to add the Kedro-Viz `SQLiteStore` to your Kedro project.
Expand Down
4 changes: 4 additions & 0 deletions RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,12 @@ Please follow the established format:
- Use present tense (e.g. 'Add new feature')
- Include the ID number for the related PR (or PRs) in parentheses
-->

# Upcoming Release

## Major features and improvements
- Add build command to the CLI using `kedro viz build` for creating build directory of local Kedro Viz instance with static data. (#1615)

## Bug fixes and other changes

- Set Kedro Viz start method to spawn process while launching it from Jupyter Notebook. (#1696)
Expand Down
2 changes: 2 additions & 0 deletions package/kedro_viz/integrations/deployment/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
"""`kedro_viz.integrations.deployment` provides interface to
integrate Kedro viz with deployers."""
110 changes: 110 additions & 0 deletions package/kedro_viz/integrations/deployment/base_deployer.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
"""`kedro_viz.integrations.deployment.base_deployer` defines
creation of Kedro-viz build"""

import abc
import json
import logging
import tempfile
from datetime import datetime
from pathlib import Path

from jinja2 import Environment, FileSystemLoader
from packaging.version import parse

from kedro_viz import __version__
from kedro_viz.api.rest.responses import save_api_responses_to_fs
from kedro_viz.integrations.kedro import telemetry as kedro_telemetry

_HTML_DIR = Path(__file__).parent.parent.parent.absolute() / "html"
_METADATA_PATH = "api/deploy-viz-metadata"

logger = logging.getLogger(__name__)


class BaseDeployer(abc.ABC):
"""A class to handle the creation of Kedro-viz build.

Attributes:
_path (str): build path name.
_fs (fsspec.filesystem): Filesystem for local/remote protocol.

Methods:
deploy_and_get_url(): Deploy Kedro-viz to cloud storage provider/local and return its URL.
"""

def __init__(self):
self._path = None
self._fs = None

def _upload_api_responses(self):
"""Write API responses to the build."""
save_api_responses_to_fs(self._path)

def _ingest_heap_analytics(self):
"""Ingest heap analytics to index file in the build."""
project_path = Path.cwd().absolute()
heap_app_id = kedro_telemetry.get_heap_app_id(project_path)
heap_user_identity = kedro_telemetry.get_heap_identity()
should_add_telemetry = bool(heap_app_id) and bool(heap_user_identity)
html_content = (_HTML_DIR / "index.html").read_text(encoding="utf-8")
injected_head_content = []

env = Environment(loader=FileSystemLoader(_HTML_DIR))

if should_add_telemetry:
logger.debug("Ingesting heap analytics.")
telemetry_content = env.get_template("telemetry.html").render(
heap_app_id=heap_app_id, heap_user_identity=heap_user_identity
)
injected_head_content.append(telemetry_content)

injected_head_content.append("</head>")
html_content = html_content.replace("</head>", "\n".join(injected_head_content))

with tempfile.TemporaryDirectory() as temp_dir:
temp_file_path = f"{temp_dir}/index.html"

with open(temp_file_path, "w", encoding="utf-8") as temp_index_file:
temp_index_file.write(html_content)

self._fs.put(temp_file_path, f"{self._path}/")
SajidAlamQB marked this conversation as resolved.
Show resolved Hide resolved

def _upload_static_files(self, html_dir: Path):
"""Upload static HTML files to Build."""
logger.debug("Uploading static html files to %s.", self._path)
try:
self._fs.put(f"{str(html_dir)}/*", str(self._path), recursive=True)
self._ingest_heap_analytics()
except Exception as exc: # pragma: no cover
logger.exception("Upload failed: %s ", exc)
raise exc

def _upload_deploy_viz_metadata_file(self):
"""Create and write metadta file to api folder"""

logger.debug(
"Creating and Uploading deploy viz metadata file to %s.",
self._path,
)

try:
metadata = {
"timestamp": datetime.utcnow().strftime("%d.%m.%Y %H:%M:%S"),
"version": str(parse(__version__)),
}
with self._fs.open(f"{self._path}/{_METADATA_PATH}", "w") as metadata_file:
metadata_file.write(json.dumps(metadata))
except Exception as exc: # pragma: no cover
logger.exception("Upload failed: %s ", exc)
raise exc

def deploy(self):
"""Create and deploy all static files to local/remote file system"""

self._upload_api_responses()
self._upload_static_files(_HTML_DIR)
self._upload_deploy_viz_metadata_file()

@abc.abstractmethod
def deploy_and_get_url(self):
"""Abstract method to deploy and return the URL."""
18 changes: 18 additions & 0 deletions package/kedro_viz/integrations/deployment/deployer_factory.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
"""`kedro_viz.integrations.deployment.deployer_factory` creates
Kedro-viz deployer class instances"""

from kedro_viz.integrations.deployment.local_deployer import LocalDeployer
from kedro_viz.integrations.deployment.s3_deployer import S3Deployer


class DeployerFactory:
"""A class to handle creation of deployer class instances."""

@staticmethod
def create_deployer(platform, region=None, bucket_name=None):
"""Instantiate Kedro-viz deployer classes"""
if platform == "s3":
return S3Deployer(region, bucket_name)
if platform == "local":
return LocalDeployer()
raise ValueError(f"Invalid platform '{platform}' specified")
38 changes: 38 additions & 0 deletions package/kedro_viz/integrations/deployment/local_deployer.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
"""`kedro_viz.integrations.deployment.base_deployer` defines
creation of Kedro-viz build"""

import logging
from pathlib import Path

import fsspec

from kedro_viz import __version__
from kedro_viz.integrations.deployment.base_deployer import BaseDeployer

_BUILD_PATH = "build"
_FILE_PROTOCOL = "file"

logger = logging.getLogger(__name__)


class LocalDeployer(BaseDeployer):
"""A class to handle the creation of Kedro-viz build folder.

Attributes:
_build_path (str): build path name.
_local_fs (fsspec.filesystem): Filesystem for local file protocol.

Methods:
deploy_and_get_url(): The creation of Kedro-viz build folder.
"""

def __init__(self):
super().__init__()
self._path = Path(_BUILD_PATH)
self._path.mkdir(parents=True, exist_ok=True)
self._fs = fsspec.filesystem(_FILE_PROTOCOL)

def deploy_and_get_url(self):
"""Copy Kedro-viz to local build folder and return its URL."""
self.deploy()
return self._path
87 changes: 6 additions & 81 deletions package/kedro_viz/integrations/deployment/s3_deployer.py
Original file line number Diff line number Diff line change
@@ -1,28 +1,19 @@
"""`kedro_viz.integrations.deployment.s3_deployer` defines
deployment class for AWS S3"""

import json
import logging
import tempfile
from datetime import datetime
from pathlib import Path

import fsspec
from jinja2 import Environment, FileSystemLoader
from packaging.version import parse

from kedro_viz import __version__
from kedro_viz.api.rest.responses import save_api_responses_to_fs
from kedro_viz.integrations.kedro import telemetry as kedro_telemetry
from kedro_viz.integrations.deployment.base_deployer import BaseDeployer

_HTML_DIR = Path(__file__).parent.parent.parent.absolute() / "html"
_METADATA_PATH = "api/deploy-viz-metadata"
_S3_PROTOCOL = "s3"

logger = logging.getLogger(__name__)


class S3Deployer:
class S3Deployer(BaseDeployer):
"""A class to handle the deployment of Kedro-viz to AWS S3.

Attributes:
Expand All @@ -42,79 +33,13 @@ def __init__(self, region, bucket_name):
region (str): AWS region to deploy to.
bucket_name (str): Name of the S3 bucket.
"""
super().__init__()
self._region = region
self._bucket_name = bucket_name
self._bucket_path = f"{_S3_PROTOCOL}://{bucket_name}"
self._remote_fs = fsspec.filesystem(_S3_PROTOCOL)

def _upload_api_responses(self):
"""Upload API responses to S3."""
save_api_responses_to_fs(self._bucket_path)

def _ingest_heap_analytics(self):
"""Ingest heap analytics to index file in the build folder."""
project_path = Path.cwd().absolute()
heap_app_id = kedro_telemetry.get_heap_app_id(project_path)
heap_user_identity = kedro_telemetry.get_heap_identity()
should_add_telemetry = bool(heap_app_id) and bool(heap_user_identity)
html_content = (_HTML_DIR / "index.html").read_text(encoding="utf-8")
injected_head_content = []

env = Environment(loader=FileSystemLoader(_HTML_DIR))

if should_add_telemetry:
logger.debug("Ingesting heap analytics.")
telemetry_content = env.get_template("telemetry.html").render(
heap_app_id=heap_app_id, heap_user_identity=heap_user_identity
)
injected_head_content.append(telemetry_content)

injected_head_content.append("</head>")
html_content = html_content.replace("</head>", "\n".join(injected_head_content))

with tempfile.TemporaryDirectory() as temp_dir:
temp_file_path = f"{temp_dir}/index.html"

with open(temp_file_path, "w", encoding="utf-8") as temp_index_file:
temp_index_file.write(html_content)

self._remote_fs.put(temp_file_path, f"{self._bucket_path}/")

def _upload_static_files(self, html_dir: Path):
"""Upload static HTML files to S3."""
logger.debug("Uploading static html files to %s.", self._bucket_path)
try:
self._remote_fs.put(f"{str(html_dir)}/*", self._bucket_path, recursive=True)
self._ingest_heap_analytics()
except Exception as exc: # pragma: no cover
logger.exception("Upload failed: %s ", exc)
raise exc

def _upload_deploy_viz_metadata_file(self):
logger.debug(
"Creating and Uploading deploy viz metadata file to %s.",
self._bucket_path,
)

try:
metadata = {
"timestamp": datetime.utcnow().strftime("%d.%m.%Y %H:%M:%S"),
"version": str(parse(__version__)),
}
with self._remote_fs.open(
f"{self._bucket_path}/{_METADATA_PATH}", "w"
) as metadata_file:
metadata_file.write(json.dumps(metadata))
except Exception as exc: # pragma: no cover
logger.exception("Upload failed: %s ", exc)
raise exc

def _deploy(self):
self._upload_api_responses()
self._upload_static_files(_HTML_DIR)
self._upload_deploy_viz_metadata_file()
self._path = f"{_S3_PROTOCOL}://{bucket_name}"
self._fs = fsspec.filesystem(_S3_PROTOCOL)

def deploy_and_get_url(self):
"""Deploy Kedro-viz to S3 and return its URL."""
self._deploy()
self.deploy()
return f"http://{self._bucket_name}.s3-website.{self._region}.amazonaws.com"
31 changes: 29 additions & 2 deletions package/kedro_viz/launchers/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@

from kedro_viz import __version__
from kedro_viz.constants import AWS_REGIONS, DEFAULT_HOST, DEFAULT_PORT
from kedro_viz.integrations.deployment.s3_deployer import S3Deployer
from kedro_viz.integrations.deployment.deployer_factory import DeployerFactory
from kedro_viz.integrations.pypi import get_latest_version, is_running_outdated_version
from kedro_viz.launchers.utils import (
_check_viz_up,
Expand Down Expand Up @@ -238,7 +238,7 @@ def deploy(region, bucket_name):
load_and_populate_data(Path.cwd(), ignore_plugins=True)

# Start the deployment
deployer = S3Deployer(region, bucket_name)
deployer = DeployerFactory.create_deployer("s3", region, bucket_name)
url = deployer.deploy_and_get_url()

click.echo(
Expand Down Expand Up @@ -270,3 +270,30 @@ def deploy(region, bucket_name):
)
finally:
viz_deploy_timer.terminate()


@viz.command(context_settings={"help_option_names": ["-h", "--help"]})
def build():
jitu5 marked this conversation as resolved.
Show resolved Hide resolved
"""Create build directory of local Kedro Viz instance with static data"""

try:
load_and_populate_data(Path.cwd(), ignore_plugins=True)
deployer = DeployerFactory.create_deployer("local")
url = deployer.deploy_and_get_url()

click.echo(
click.style(
"\u2728 Success! Kedro-Viz build files have been successfully added to the "
f"{url} directory.",
fg="green",
)
)

# pylint: disable=broad-exception-caught
except Exception as exc: # pragma: no cover
click.echo(
click.style(
f"ERROR: Failed to build Kedro-Viz : {exc} ",
fg="red",
)
)
Loading