Add databricks asset bundles docs #4265
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
noklam
commented
Oct 28, 2024
| @@ -0,0 +1,272 @@ | |||
| # Use an IDE, dbx and Databricks Repos to develop a Kedro project | |||
There was a problem hiding this comment.
This is not new docs so no need to review, I move the original databricks_ide_developmenet.md page to a new file and use that file for DAB instead.
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
noklam
force-pushed
the
noklam/databricks-asset-bundles-docs
branch
from
8eec4ba
to
2733196
Compare
merelcht
reviewed
Nov 1, 2024
There was a problem hiding this comment.
Should we also update the index page https://docs.kedro.org/en/stable/deployment/databricks/index.html and mention asset bundles?
| The `dbx` package is deprecated by Databricks, and dbx workflow documentation is moved to a [new page](./databricks_dbx_workflow.md). | ||
| ``` | ||
|
|
||
| This guide demonstrates a workflow for developing a Kedro Project on Databricks using Databricks Asset Bundles. You will learn how to develop your project using a local environment, then use `kedro-databricks` and Databricks Asset Bundle to package your code for running pipelines on Databricks. |
There was a problem hiding this comment.
Maybe add a link here to https://www.databricks.com/resources/demos/tours/data-engineering/databricks-asset-bundles ?
|
|
||
| The next step is to use dbx to sync your project to your Repo. | ||
| `kedro-databricks` is a wrapper around the `databricks` CLI. It's the simplest way to get started without getting stuck with configuration. To learn more about Databricks Asset Bundles and customization, read [What are Databricks Asset Bundles](https://docs.databricks.com/en/dev-tools/bundles/index.html). |
There was a problem hiding this comment.
Nit: UK spelling
Suggested change
| `kedro-databricks` is a wrapper around the `databricks` CLI. It's the simplest way to get started without getting stuck with configuration. To learn more about Databricks Asset Bundles and customization, read [What are Databricks Asset Bundles](https://docs.databricks.com/en/dev-tools/bundles/index.html). | |
| `kedro-databricks` is a wrapper around the `databricks` CLI. It's the simplest way to get started without getting stuck with configuration. To learn more about Databricks Asset Bundles and customisation, read [What are Databricks Asset Bundles](https://docs.databricks.com/en/dev-tools/bundles/index.html). |
| ``` | ||
|
|
||
| This command will sync your local directory (`--source .`) with your Repo (`--dest-repo iris-databricks`) on Databricks. When started for the first time, `dbx sync` will write output similar to the following to your terminal: | ||
| 2. Initialize the Databricks configuration: |
There was a problem hiding this comment.
Suggested change
| 2. Initialize the Databricks configuration: | |
| 2. Initialise the Databricks configuration: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Partially #3360
Development notes
kedro-databricksDeveloper Certificate of Origin
We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a
Signed-off-byline in the commit message. See our wiki for guidance.If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.
Checklist
RELEASE.mdfile