-
Notifications
You must be signed in to change notification settings - Fork 113
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CLI command kedro viz build added #1697
Merged
Merged
Changes from all commits
Commits
Show all changes
29 commits
Select commit
Hold shift + click to select a range
66f177d
CLI command kedro viz build added
jitu5 cc7c144
Lint fix
jitu5 a389a83
lint fix
jitu5 6528d36
Lint fix
jitu5 5a9f2e1
add mypy ignore
jitu5 b86b578
Missing build file added
jitu5 40003f8
Lint error fix
jitu5 0c879a8
BaseDeployer class added
jitu5 5145ba4
Unused code removed
jitu5 26f6275
Fix lint issue
jitu5 c7ed79c
added base_deployer
rashidakanchwala 11ac321
latest
rashidakanchwala 42af3af
add deployer factory
rashidakanchwala 7ad0905
Test and comments of deployers updated
jitu5 2a4e00f
fix lint
ravi-kumar-pilla ecb6a05
remove circular dependency
ravi-kumar-pilla 90c285b
Tests and lint fixes
jitu5 eadb85c
Merge branch 'feature/cli-build' of https://github.com/kedro-org/kedr…
jitu5 d579bb7
Import fixes
jitu5 e8e25fa
Lint fixes
jitu5 0ad5301
Lint fixes
jitu5 98f11b2
Merge branch 'main' into feature/cli-build
jitu5 2ffc777
fix lint
rashidakanchwala 4c977a6
fix return on deploy_get_url
rashidakanchwala d51634f
Exception and class method updated
jitu5 35b9775
Merge branch 'feature/cli-build' of https://github.com/kedro-org/kedr…
jitu5 0297a5f
User entered platform added in error
jitu5 b27cdaf
Fix test
jitu5 932e654
Merge branch 'main' into feature/cli-build
jitu5 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
"""`kedro_viz.integrations.deployment` provides interface to | ||
integrate Kedro viz with deployers.""" |
110 changes: 110 additions & 0 deletions
110
package/kedro_viz/integrations/deployment/base_deployer.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,110 @@ | ||
"""`kedro_viz.integrations.deployment.base_deployer` defines | ||
creation of Kedro-viz build""" | ||
|
||
import abc | ||
import json | ||
import logging | ||
import tempfile | ||
from datetime import datetime | ||
from pathlib import Path | ||
|
||
from jinja2 import Environment, FileSystemLoader | ||
from packaging.version import parse | ||
|
||
from kedro_viz import __version__ | ||
from kedro_viz.api.rest.responses import save_api_responses_to_fs | ||
from kedro_viz.integrations.kedro import telemetry as kedro_telemetry | ||
|
||
_HTML_DIR = Path(__file__).parent.parent.parent.absolute() / "html" | ||
_METADATA_PATH = "api/deploy-viz-metadata" | ||
|
||
logger = logging.getLogger(__name__) | ||
|
||
|
||
class BaseDeployer(abc.ABC): | ||
"""A class to handle the creation of Kedro-viz build. | ||
|
||
Attributes: | ||
_path (str): build path name. | ||
_fs (fsspec.filesystem): Filesystem for local/remote protocol. | ||
|
||
Methods: | ||
deploy_and_get_url(): Deploy Kedro-viz to cloud storage provider/local and return its URL. | ||
""" | ||
|
||
def __init__(self): | ||
self._path = None | ||
self._fs = None | ||
|
||
def _upload_api_responses(self): | ||
"""Write API responses to the build.""" | ||
save_api_responses_to_fs(self._path) | ||
|
||
def _ingest_heap_analytics(self): | ||
"""Ingest heap analytics to index file in the build.""" | ||
project_path = Path.cwd().absolute() | ||
heap_app_id = kedro_telemetry.get_heap_app_id(project_path) | ||
heap_user_identity = kedro_telemetry.get_heap_identity() | ||
should_add_telemetry = bool(heap_app_id) and bool(heap_user_identity) | ||
html_content = (_HTML_DIR / "index.html").read_text(encoding="utf-8") | ||
injected_head_content = [] | ||
|
||
env = Environment(loader=FileSystemLoader(_HTML_DIR)) | ||
|
||
if should_add_telemetry: | ||
logger.debug("Ingesting heap analytics.") | ||
telemetry_content = env.get_template("telemetry.html").render( | ||
heap_app_id=heap_app_id, heap_user_identity=heap_user_identity | ||
) | ||
injected_head_content.append(telemetry_content) | ||
|
||
injected_head_content.append("</head>") | ||
html_content = html_content.replace("</head>", "\n".join(injected_head_content)) | ||
|
||
with tempfile.TemporaryDirectory() as temp_dir: | ||
temp_file_path = f"{temp_dir}/index.html" | ||
|
||
with open(temp_file_path, "w", encoding="utf-8") as temp_index_file: | ||
temp_index_file.write(html_content) | ||
|
||
self._fs.put(temp_file_path, f"{self._path}/") | ||
SajidAlamQB marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
def _upload_static_files(self, html_dir: Path): | ||
"""Upload static HTML files to Build.""" | ||
logger.debug("Uploading static html files to %s.", self._path) | ||
try: | ||
self._fs.put(f"{str(html_dir)}/*", str(self._path), recursive=True) | ||
self._ingest_heap_analytics() | ||
except Exception as exc: # pragma: no cover | ||
logger.exception("Upload failed: %s ", exc) | ||
raise exc | ||
|
||
def _upload_deploy_viz_metadata_file(self): | ||
"""Create and write metadta file to api folder""" | ||
|
||
logger.debug( | ||
"Creating and Uploading deploy viz metadata file to %s.", | ||
self._path, | ||
) | ||
|
||
try: | ||
metadata = { | ||
"timestamp": datetime.utcnow().strftime("%d.%m.%Y %H:%M:%S"), | ||
"version": str(parse(__version__)), | ||
} | ||
with self._fs.open(f"{self._path}/{_METADATA_PATH}", "w") as metadata_file: | ||
metadata_file.write(json.dumps(metadata)) | ||
except Exception as exc: # pragma: no cover | ||
logger.exception("Upload failed: %s ", exc) | ||
raise exc | ||
|
||
def deploy(self): | ||
"""Create and deploy all static files to local/remote file system""" | ||
|
||
self._upload_api_responses() | ||
self._upload_static_files(_HTML_DIR) | ||
self._upload_deploy_viz_metadata_file() | ||
|
||
@abc.abstractmethod | ||
def deploy_and_get_url(self): | ||
"""Abstract method to deploy and return the URL.""" |
18 changes: 18 additions & 0 deletions
18
package/kedro_viz/integrations/deployment/deployer_factory.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
"""`kedro_viz.integrations.deployment.deployer_factory` creates | ||
Kedro-viz deployer class instances""" | ||
|
||
from kedro_viz.integrations.deployment.local_deployer import LocalDeployer | ||
from kedro_viz.integrations.deployment.s3_deployer import S3Deployer | ||
|
||
|
||
class DeployerFactory: | ||
"""A class to handle creation of deployer class instances.""" | ||
|
||
@staticmethod | ||
def create_deployer(platform, region=None, bucket_name=None): | ||
"""Instantiate Kedro-viz deployer classes""" | ||
if platform == "s3": | ||
return S3Deployer(region, bucket_name) | ||
if platform == "local": | ||
return LocalDeployer() | ||
raise ValueError(f"Invalid platform '{platform}' specified") |
38 changes: 38 additions & 0 deletions
38
package/kedro_viz/integrations/deployment/local_deployer.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
"""`kedro_viz.integrations.deployment.base_deployer` defines | ||
creation of Kedro-viz build""" | ||
|
||
import logging | ||
from pathlib import Path | ||
|
||
import fsspec | ||
|
||
from kedro_viz import __version__ | ||
from kedro_viz.integrations.deployment.base_deployer import BaseDeployer | ||
|
||
_BUILD_PATH = "build" | ||
_FILE_PROTOCOL = "file" | ||
|
||
logger = logging.getLogger(__name__) | ||
|
||
|
||
class LocalDeployer(BaseDeployer): | ||
"""A class to handle the creation of Kedro-viz build folder. | ||
|
||
Attributes: | ||
_build_path (str): build path name. | ||
_local_fs (fsspec.filesystem): Filesystem for local file protocol. | ||
|
||
Methods: | ||
deploy_and_get_url(): The creation of Kedro-viz build folder. | ||
""" | ||
|
||
def __init__(self): | ||
super().__init__() | ||
self._path = Path(_BUILD_PATH) | ||
self._path.mkdir(parents=True, exist_ok=True) | ||
self._fs = fsspec.filesystem(_FILE_PROTOCOL) | ||
|
||
def deploy_and_get_url(self): | ||
"""Copy Kedro-viz to local build folder and return its URL.""" | ||
self.deploy() | ||
return self._path |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we have some docs explaining how to view this build?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, in another PR we will focus on the build docs