Update deployment diagram to include Dask 🧜‍♂️ (#1392)

* Delete deployments.png Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Update deployment diagram to include Dask 🧜‍♂️ Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Setup the markdown support in Sphinx via myst-parser Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Add sphinxcontrib-mermaid to setup.py Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Add myst-parser to docs requirements in setup.py Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Circumvent already-registered source suffix Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Remove recommonmrk (replaced by myst_parser) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Suppress non-consecutive header level warning treated as error Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Try updaating docs/source/contribution/developer_contributor_guidelines.md Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Move note contents to a new line to magically fix anchor generation Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Standardize note formatting to make it easier to script bulk changes Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Fix docs/source/contribution/documentation_contributor_guidelines.md note directives Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Bulk fix note directives find docs/source/ -type f -name '*.md' -exec perl -i -p0e 's/```eval_rst\n\.\. note:: */```{note}\n/g' {} + Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * update warning directives Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * We use H5... Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Use warning over caution Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * attention -> warning Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * change more eval rst nd attention blocks, and fix anchor Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * fix anchor Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * More eval_rst fixes, and change link to rst to without extension Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * fix link in pyspark.md Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Fix non-consecutive header levels instead of ignoring them Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * fix remaining eval_rst issues Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Revert "Delete deployments.png" This reverts commit 85bfade. Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Revert "Update deployment diagram to include Dask 🧜‍♂️" This reverts commit a1d0509. Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * remove recommonmark and mermid reqs and refs Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Apply suggestions from code review Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Restore `source_suffix` after removing recommonmark Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * remaining changes, like removing unnecessary requirements and using warning tag once more, and source suffixes Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Update note (and similar) block contents from reStructuredText to Markdown syntax Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Add missing changes from last commit Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Revert "Revert "Delete deployments.png"" This reverts commit 9cbdf46. Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Revert "Revert "Update deployment diagram to include Dask 🧜‍♂️"" This reverts commit eaabf5f. Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Update docs/conf.py to add mermaid extension Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Update setup.py to add mermaid extension Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Add unminified mermaid.js to pages w/ mermaid node Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Add Mermaid JS file, then remove where unnecessary Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Delete the bad "None" items from `mermaid_version` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Update RELEASE.md Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Co-authored-by: Antony Milne <49395058+AntonyMilneQB@users.noreply.github.com>
kedro-org · Jul 6, 2022 · fdc6be7 · fdc6be7
1 parent 71eb84f
commit fdc6be7
Show file tree

Hide file tree

Showing 5 changed files with 48 additions and 3 deletions.
diff --git a/RELEASE.md b/RELEASE.md
@@ -33,11 +33,11 @@
 * Required `cookiecutter>=2.1.1` to address a [known command injection vulnerability](https://security.snyk.io/vuln/SNYK-PYTHON-COOKIECUTTER-2414281).
 * The session store no longer fails if a username cannot be found with `getpass.getuser`.
 * Added generic typing for `AbstractDataSet` and `AbstractVersionedDataSet` as well as typing to all datasets.
+* Rendered the deployment guide flowchart as a Mermaid diagram, and added Dask.
 
 ## Minor breaking changes to the API
 * The module `kedro.config.default_logger` no longer exists; default logging configuration is now set automatically through `kedro.framework.project.LOGGING`. Unless you explicitly import `kedro.config.default_logger` you do not need to make any changes.
 
-
 ## Upcoming deprecations for Kedro 0.19.0
 * `kedro.extras.ColorHandler` will be removed in 0.19.0.
 

diff --git a/docs/conf.py b/docs/conf.py
@@ -20,12 +20,17 @@
 from distutils.dir_util import copy_tree
 from inspect import getmembers, isclass, isfunction
 from pathlib import Path
-from typing import List, Tuple
+from typing import Dict, List, Optional, Tuple
 
 from click import secho, style
+from docutils import nodes
+from sphinx.application import Sphinx
+from sphinxcontrib.mermaid import mermaid
 
 from kedro import __version__ as release
 
+MERMAID_JS_URL = "https://unpkg.com/mermaid/dist/mermaid.min.js"
+
 # -- Project information -----------------------------------------------------
 
 project = "Kedro"
@@ -56,6 +61,7 @@
     "sphinx.ext.viewcode",
     "nbsphinx",
     "sphinx_copybutton",
+    "sphinxcontrib.mermaid",
     "myst_parser",
 ]
 
@@ -517,6 +523,29 @@ def _add_jinja_filters(app):
         app.builder.templates.environment.filters["env_override"] = env_override
 
 
+def remove_unused_mermaid_script_file(
+    app: Sphinx,
+    pagename: str,
+    templatename: str,
+    context: Dict,
+    doctree: Optional[nodes.document],
+) -> None:
+    # The `doctree` arg is `None` when not created from a reST document.
+    if not doctree:
+        return
+
+    # Remove the Mermaid JavaScript from pages without Mermaid diagrams.
+    if not doctree.next_node(mermaid):
+        # Create a copy of `context["script_files"]`; modifying the list
+        # in place affects all pages, because they all use the same ref.
+        context["script_files"] = [
+            x for x in context["script_files"] if x != MERMAID_JS_URL
+        ]
+
+    # Remove "None" entries added when `mermaid_version` is set to `""`.
+    context["script_files"] = [x for x in context["script_files"] if x != "None"]
+
+
 def setup(app):
     app.connect("config-inited", _prepare_build_dir)
     app.connect("builder-inited", _add_jinja_filters)
@@ -525,6 +554,8 @@ def setup(app):
     # fix a bug with table wraps in Read the Docs Sphinx theme:
     # https://rackerlabs.github.io/docs-rackspace/tools/rtd-tables.html
     app.add_css_file("css/theme-overrides.css")
+    app.add_js_file(MERMAID_JS_URL)
+    app.connect("html-page-context", remove_unused_mermaid_script_file)
 
 
 # (regex, restructuredText link replacement, object) list
@@ -552,3 +583,5 @@ def setup(app):
 user_agent = "Mozilla/5.0 (X11; Linux x86_64; rv:99.0) Gecko/20100101 Firefox/99.0"
 
 myst_heading_anchors = 5
+
+mermaid_version = ""  # We add a minified Mermaid script file ourselves.
diff --git a/docs/source/deployment/deployment_guide.md b/docs/source/deployment/deployment_guide.md
@@ -21,4 +21,15 @@ We also provide information to help you deploy to the following:
 
 In addition, we also provide instructions on [how to integrate a Kedro project with Amazon SageMaker](aws_sagemaker.md).
 
-![](../meta/images/deployments.png)
+```{mermaid}
+flowchart TD
+    A{Can your Kedro pipeline run on a single machine?} -- YES --> B[Consult the single-machine deployment guide];
+    B --> C{Do you have Docker on your machine?};
+    C -- YES --> D[Use a container-based approach];
+    C -- NO --> E[Use the CLI or package mode];
+    A -- NO --> F[Consult the distributed deployment guide];
+    F --> G["What distributed platform are you using?<br/><br/>Check out the guides for:<br/><br/><li>Argo</li><li>Prefect</li><li>Kubeflow Pipelines</li><li>AWS Batch</li><li>Databricks</li><li>Dask</li>"];
+    style G text-align:left
+    H["Does (part of) your pipeline integrate with Amazon SageMaker?<br/><br/>Read the SageMaker integration guide"];
+    style H text-align:left
+```
diff --git a/docs/source/meta/images/deployments.png b/docs/source/meta/images/deployments.png
diff --git a/setup.py b/setup.py
@@ -112,6 +112,7 @@ def _collect_requirements(requires):
         "sphinx-autodoc-typehints==1.11.1",
         "sphinx_copybutton==0.3.1",
         "ipykernel>=5.3, <7.0",
+        "sphinxcontrib-mermaid~=0.7.1",
         "myst-parser~=0.17.2",
     ],
     "geopandas": _collect_requirements(geopandas_require),