Create a databricks-iris starter that enables packaged deployment on Databricks #129
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>
Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>
Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>
Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>
jmholzer
changed the title
Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>
Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>
Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>
noklam
reviewed
May 26, 2023
...ks-iris/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/databricks_run.py
Show resolved
Hide resolved
databricks-iris/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/hooks.py
Show resolved
Hide resolved
databricks-iris/{{ cookiecutter.repo_name }}/conf/base/logging.yml
Outdated
Show resolved
Hide resolved
databricks-iris/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/__main__.py
Outdated
Show resolved
Hide resolved
….yml Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>
…' of github.com:kedro-org/kedro-starters into feat/modify-pyspark-iris-databricks-packaged-deployment Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>
|
To test this: |
astrojuanlu
reviewed
May 29, 2023
databricks-iris/{{ cookiecutter.repo_name }}/conf/base/spark.yml
Outdated
Show resolved
Hide resolved
Thanks for figuring this out @astrojuanlu! |
jmholzer
deleted the
feat/modify-pyspark-iris-databricks-packaged-deployment
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation and Context
The guide on deploying packaged projects to Databricks proposed in kedro-org/kedro#2595 uses the
databricks-irisstarter. This PR adds this starter. Thedatabricks-irisstarter is a duplicate of thepyspark-irisstarter with a few changes.databricks_run.py: a module for running the project on Databricks, as Click causes us to be unable to run projects with the default entry point on Databricks.conf/base/logging.yml).conf/base/catalog.ymlare saved in/dbfs/FileStore.This PR has a large diff because it is a brand new starter, only the following files have been changed from
pyspark-iris:{{ cookiecutter.repo_name }}/src/setup.py: contains an entry point definitiondatabricks_run.{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/databricks_run.py: contains a script needed to run a packaged Kedro project on Databricks.{{ cookiecutter.repo_name }}/src/conf/base/logging.yml: config for writing logs to DBFS.{{ cookiecutter.repo_name }}/src/conf/base/catalog.yml: points to datasets on DBFS.How has this been tested?
Manually on Databricks in conjunction with the new guide.
Checklist