Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation for deploying packaged Kedro projects on Databricks #2595

Merged
merged 72 commits into from
Jun 1, 2023
Merged
Changes from 1 commit
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
cf03914
Add deployment workflow page
jmholzer May 3, 2023
57e1d55
Add table of contents, entry point guide, data and conf upload guide
jmholzer May 16, 2023
66fd862
Add detailed instructions for creating a job on Databricks
jmholzer May 18, 2023
2b21797
Add images and automated deployment resources
jmholzer May 20, 2023
ea1142a
Merge branch 'main' into docs/add-databricks-deployment-workflow
jmholzer May 20, 2023
5c76bc5
Remove use of 'allows', add summary
jmholzer May 22, 2023
db5c404
Merge branch 'docs/add-databricks-deployment-workflow' of github.com:…
jmholzer May 22, 2023
0f20bb7
Remove link to missing image
jmholzer May 22, 2023
1baa5e2
Add deployment workflow to toctree
jmholzer May 22, 2023
ed73514
Lint and fix missing link
jmholzer May 22, 2023
6057994
Minor style, syntax and grammar improvements
jmholzer May 22, 2023
5eedb2b
Merge branch 'main' into docs/add-databricks-deployment-workflow
jmholzer May 22, 2023
3d22bd6
Fixes for correctness during validation
jmholzer May 23, 2023
384aebb
Merge branch 'docs/add-databricks-deployment-workflow' of github.com:…
jmholzer May 23, 2023
4d27dad
Add instructions for creating log output location
jmholzer May 23, 2023
8b246d1
Lint
jmholzer May 23, 2023
25da49d
Lint databricks_run
jmholzer May 23, 2023
c384f62
Merge branch 'main' into docs/add-databricks-deployment-workflow
jmholzer May 23, 2023
c83107b
Minor wording change in reference to logs
jmholzer May 23, 2023
df5e743
Merge branch 'docs/add-databricks-deployment-workflow' of github.com:…
jmholzer May 23, 2023
cfbe80b
Modify reference to Pyspark-Iris
jmholzer May 23, 2023
7e5406a
Update docs/source/integrations/databricks_deployment_workflow.md
jmholzer May 24, 2023
fdfe925
Fix linter errors to enable docs build for inspection
stichbury May 24, 2023
3ac02cc
Update build-docs.sh
stichbury May 24, 2023
04ca9c0
Fix broken link
stichbury May 24, 2023
93997e2
Update docs/source/integrations/databricks_deployment_workflow.md
jmholzer May 24, 2023
74ab283
Update docs/source/integrations/databricks_deployment_workflow.md
jmholzer May 24, 2023
caa9ecc
Update docs/source/integrations/databricks_deployment_workflow.md
jmholzer May 24, 2023
9901b40
Update docs/source/integrations/databricks_deployment_workflow.md
jmholzer May 24, 2023
dd9b4a0
Remove spurious word
jmholzer May 24, 2023
bd2ce14
Update docs/source/integrations/databricks_deployment_workflow.md
jmholzer May 24, 2023
f5cd5be
Update docs/source/integrations/databricks_deployment_workflow.md
jmholzer May 24, 2023
f3cf261
Update docs/source/integrations/databricks_deployment_workflow.md
jmholzer May 24, 2023
edbee92
Update docs/source/integrations/databricks_deployment_workflow.md
jmholzer May 24, 2023
f67d7f4
Update docs/source/integrations/databricks_deployment_workflow.md
jmholzer May 24, 2023
5c60a4b
Update docs/source/integrations/databricks_deployment_workflow.md
jmholzer May 24, 2023
1b45757
Update docs/source/integrations/databricks_deployment_workflow.md
jmholzer May 24, 2023
a7aacb5
Update docs/source/integrations/databricks_deployment_workflow.md
jmholzer May 24, 2023
a11624c
Update docs/source/integrations/databricks_deployment_workflow.md
jmholzer May 24, 2023
40c6e62
Update docs/source/integrations/databricks_deployment_workflow.md
jmholzer May 24, 2023
f37a8e8
Update docs/source/integrations/databricks_deployment_workflow.md
jmholzer May 24, 2023
dc8c9ed
Update docs/source/integrations/databricks_deployment_workflow.md
jmholzer May 24, 2023
28c7b87
Update docs/source/integrations/databricks_deployment_workflow.md
jmholzer May 24, 2023
f392625
Merge branch 'docs/add-databricks-deployment-workflow' of github.com:…
jmholzer May 24, 2023
1e9a01d
Add advantages subheading
jmholzer May 24, 2023
c2b406e
Update docs/source/integrations/databricks_deployment_workflow.md
jmholzer May 24, 2023
ee58551
Merge branch 'docs/add-databricks-deployment-workflow' of github.com:…
jmholzer May 24, 2023
0407991
Add alternative ways to upload data to DBFS
jmholzer May 24, 2023
2149bfd
Move note on unpackaged config and data
jmholzer May 24, 2023
672c3d0
Fix broken links
jmholzer May 24, 2023
d939fba
Move databricks back into deployment section
stichbury May 25, 2023
89760f1
Remove references to PySpark Iris (pyspark-iris) starter
jmholzer May 25, 2023
adbca32
Merge branch 'docs/add-databricks-deployment-workflow' of github.com:…
jmholzer May 25, 2023
a7dc9e8
Graphics links fixes, revise titles
stichbury May 25, 2023
2a43dca
Merge branch 'docs/add-databricks-deployment-workflow' of https://git…
stichbury May 25, 2023
acae7c3
Fix broken internal link
stichbury May 25, 2023
21a1097
Merge branch 'main' into docs/add-databricks-deployment-workflow
stichbury May 25, 2023
b24c062
Fix links broken by new folder
stichbury May 25, 2023
ba0c184
Remove logs directory
jmholzer May 26, 2023
10e7010
Merge branch 'docs/add-databricks-deployment-workflow' of github.com:…
jmholzer May 26, 2023
65ccd68
Update image of final job configuration
jmholzer May 26, 2023
e88612b
Add full stops in list.
jmholzer May 30, 2023
085e9e5
Fix conda environment name.
jmholzer May 30, 2023
d8bd527
Modify wording and image for creating a new job cluster
jmholzer May 30, 2023
1777daa
Modify wording in guide to create new job cluster
jmholzer May 30, 2023
62a4b9a
Merge branch 'docs/add-databricks-deployment-workflow' of github.com:…
jmholzer May 30, 2023
1af24e4
Remove --upgrade option
jmholzer May 30, 2023
7be8166
Add both ways of creating a new job
jmholzer May 30, 2023
593e418
Merge branch 'main' into docs/add-databricks-deployment-workflow
jmholzer May 30, 2023
0b3eb4e
Merge branch 'docs/add-databricks-deployment-workflow' of github.com:…
jmholzer May 30, 2023
6454959
Merge branch 'main' into docs/add-databricks-deployment-workflow
jmholzer May 31, 2023
221a77a
Merge branch 'main' into docs/add-databricks-deployment-workflow
astrojuanlu Jun 1, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update docs/source/integrations/databricks_deployment_workflow.md
Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>
  • Loading branch information
jmholzer and stichbury authored May 24, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 1b457577cbba3d1055e579e51f417e6a18d662cb
2 changes: 1 addition & 1 deletion docs/source/integrations/databricks_deployment_workflow.md
Original file line number Diff line number Diff line change
@@ -257,7 +257,7 @@ This page also shows an overview of all past runs of your job. As you only just

The following things happen when you run your job:

- The job cluster will be provisioned and started (job status: `Pending`).
- The job cluster is provisioned and started (job status: `Pending`).
- The packaged Kedro project and all its dependencies will be installed (job status: `Pending`)
jmholzer marked this conversation as resolved.
Show resolved Hide resolved
- The packaged Kedro project is run from the specified `databricks_run` entry point (job status: `In Progress`).
- The packaged code finishes executing and the job cluster is stopped (job status: `Succeeded`).