-
Notifications
You must be signed in to change notification settings - Fork 72
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make resource and artifact paths in bundle config relative to config folder #708
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 tasks
pietern
reviewed
Aug 31, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two thoughts on the selector/transformer approach:
- They are the same except for the directory relative to which the rewrite should happen. I think this doesn't warrant two separate structures. Perhaps the dir can be passed through to the functions that produce the selectors (and be made to return instances of a transformer)?
- The
rewriteFunc
already does afilepath.Rel
to determine how the path is relative to the given directory. The same is happening intranslateToBundleRootRelativePath
. Can these be consolidated? Right now we're mixing up what seems to be 1) making paths relative to the bundle root, and 2) rewriting them into remote paths. If we break this up into doing 1 first and 2 second, then I suspect things to be clearer.
|
pietern
reviewed
Sep 1, 2023
pietern
approved these changes
Sep 4, 2023
github-merge-queue bot
pushed a commit
that referenced
this pull request
Sep 5, 2023
~(this should be changed to target `main`)~ This reveals the template from #686 in CLI prompts for once #686 and #708 are merged. --------- Co-authored-by: Andrew Nester <andrew.nester@databricks.com> Co-authored-by: PaulCornellDB <paul.cornell@databricks.com> Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
Merged
pietern
added a commit
that referenced
this pull request
Sep 6, 2023
This release includes permission related commands for a subset of workspace services where they apply. These complement the `permissions` command and do not require specification of the object type to work with, as that is implied by the command they are nested under. CLI: * Group permission related commands ([#730](#730)). Bundles: * Fixed artifact file uploading on Windows and wheel execution on DBR 13.3 ([#722](#722)). * Make resource and artifact paths in bundle config relative to config folder ([#708](#708)). * Add support for ordering of input prompts ([#662](#662)). * Fix IsServicePrincipal() only working for workspace admins ([#732](#732)). * databricks bundle init template v1 ([#686](#686)). * databricks bundle init template v2: optional stubs, DLT support ([#700](#700)). * Show 'databricks bundle init' template in CLI prompt ([#725](#725)). * Include $PATH in set of environment variables to pass along. ([#736](#736)). Internal: * Update Go SDK to v0.19.0 ([#729](#729)). * Replace API call to test configuration with dummy authenticate call ([#728](#728)). API Changes: * Changed `databricks account storage-credentials create` command to return . * Changed `databricks account storage-credentials get` command to return . * Changed `databricks account storage-credentials list` command to return . * Changed `databricks account storage-credentials update` command to return . * Changed `databricks connections create` command with new required argument order. * Changed `databricks connections update` command with new required argument order. * Changed `databricks volumes create` command with new required argument order. * Added `databricks artifact-allowlists` command group. * Added `databricks model-versions` command group. * Added `databricks registered-models` command group. * Added `databricks cluster-policies get-permission-levels` command. * Added `databricks cluster-policies get-permissions` command. * Added `databricks cluster-policies set-permissions` command. * Added `databricks cluster-policies update-permissions` command. * Added `databricks clusters get-permission-levels` command. * Added `databricks clusters get-permissions` command. * Added `databricks clusters set-permissions` command. * Added `databricks clusters update-permissions` command. * Added `databricks instance-pools get-permission-levels` command. * Added `databricks instance-pools get-permissions` command. * Added `databricks instance-pools set-permissions` command. * Added `databricks instance-pools update-permissions` command. * Added `databricks files` command group. * Changed `databricks permissions set` command to start returning . * Changed `databricks permissions update` command to start returning . * Added `databricks users get-permission-levels` command. * Added `databricks users get-permissions` command. * Added `databricks users set-permissions` command. * Added `databricks users update-permissions` command. * Added `databricks jobs get-permission-levels` command. * Added `databricks jobs get-permissions` command. * Added `databricks jobs set-permissions` command. * Added `databricks jobs update-permissions` command. * Changed `databricks experiments get-by-name` command to return . * Changed `databricks experiments get-experiment` command to return . * Added `databricks experiments delete-runs` command. * Added `databricks experiments get-permission-levels` command. * Added `databricks experiments get-permissions` command. * Added `databricks experiments restore-runs` command. * Added `databricks experiments set-permissions` command. * Added `databricks experiments update-permissions` command. * Added `databricks model-registry get-permission-levels` command. * Added `databricks model-registry get-permissions` command. * Added `databricks model-registry set-permissions` command. * Added `databricks model-registry update-permissions` command. * Added `databricks pipelines get-permission-levels` command. * Added `databricks pipelines get-permissions` command. * Added `databricks pipelines set-permissions` command. * Added `databricks pipelines update-permissions` command. * Added `databricks serving-endpoints get-permission-levels` command. * Added `databricks serving-endpoints get-permissions` command. * Added `databricks serving-endpoints set-permissions` command. * Added `databricks serving-endpoints update-permissions` command. * Added `databricks token-management get-permission-levels` command. * Added `databricks token-management get-permissions` command. * Added `databricks token-management set-permissions` command. * Added `databricks token-management update-permissions` command. * Changed `databricks dashboards create` command with new required argument order. * Added `databricks warehouses get-permission-levels` command. * Added `databricks warehouses get-permissions` command. * Added `databricks warehouses set-permissions` command. * Added `databricks warehouses update-permissions` command. * Added `databricks dashboard-widgets` command group. * Added `databricks query-visualizations` command group. * Added `databricks repos get-permission-levels` command. * Added `databricks repos get-permissions` command. * Added `databricks repos set-permissions` command. * Added `databricks repos update-permissions` command. * Added `databricks secrets get-secret` command. * Added `databricks workspace get-permission-levels` command. * Added `databricks workspace get-permissions` command. * Added `databricks workspace set-permissions` command. * Added `databricks workspace update-permissions` command. OpenAPI commit 09a7fa63d9ae243e5407941f200960ca14d48b07 (2023-09-04)
github-merge-queue bot
pushed a commit
that referenced
this pull request
Sep 6, 2023
This release includes permission related commands for a subset of workspace services where they apply. These complement the `permissions` command and do not require specification of the object type to work with, as that is implied by the command they are nested under. CLI: * Group permission related commands ([#730](#730)). Bundles: * Fixed artifact file uploading on Windows and wheel execution on DBR 13.3 ([#722](#722)). * Make resource and artifact paths in bundle config relative to config folder ([#708](#708)). * Add support for ordering of input prompts ([#662](#662)). * Fix IsServicePrincipal() only working for workspace admins ([#732](#732)). * databricks bundle init template v1 ([#686](#686)). * databricks bundle init template v2: optional stubs, DLT support ([#700](#700)). * Show 'databricks bundle init' template in CLI prompt ([#725](#725)). * Include $PATH in set of environment variables to pass along. ([#736](#736)). Internal: * Update Go SDK to v0.19.0 ([#729](#729)). * Replace API call to test configuration with dummy authenticate call ([#728](#728)). API Changes: * Changed `databricks account storage-credentials create` command to return . * Changed `databricks account storage-credentials get` command to return . * Changed `databricks account storage-credentials list` command to return . * Changed `databricks account storage-credentials update` command to return . * Changed `databricks connections create` command with new required argument order. * Changed `databricks connections update` command with new required argument order. * Changed `databricks volumes create` command with new required argument order. * Added `databricks artifact-allowlists` command group. * Added `databricks model-versions` command group. * Added `databricks registered-models` command group. * Added `databricks cluster-policies get-permission-levels` command. * Added `databricks cluster-policies get-permissions` command. * Added `databricks cluster-policies set-permissions` command. * Added `databricks cluster-policies update-permissions` command. * Added `databricks clusters get-permission-levels` command. * Added `databricks clusters get-permissions` command. * Added `databricks clusters set-permissions` command. * Added `databricks clusters update-permissions` command. * Added `databricks instance-pools get-permission-levels` command. * Added `databricks instance-pools get-permissions` command. * Added `databricks instance-pools set-permissions` command. * Added `databricks instance-pools update-permissions` command. * Added `databricks files` command group. * Changed `databricks permissions set` command to start returning . * Changed `databricks permissions update` command to start returning . * Added `databricks users get-permission-levels` command. * Added `databricks users get-permissions` command. * Added `databricks users set-permissions` command. * Added `databricks users update-permissions` command. * Added `databricks jobs get-permission-levels` command. * Added `databricks jobs get-permissions` command. * Added `databricks jobs set-permissions` command. * Added `databricks jobs update-permissions` command. * Changed `databricks experiments get-by-name` command to return . * Changed `databricks experiments get-experiment` command to return . * Added `databricks experiments delete-runs` command. * Added `databricks experiments get-permission-levels` command. * Added `databricks experiments get-permissions` command. * Added `databricks experiments restore-runs` command. * Added `databricks experiments set-permissions` command. * Added `databricks experiments update-permissions` command. * Added `databricks model-registry get-permission-levels` command. * Added `databricks model-registry get-permissions` command. * Added `databricks model-registry set-permissions` command. * Added `databricks model-registry update-permissions` command. * Added `databricks pipelines get-permission-levels` command. * Added `databricks pipelines get-permissions` command. * Added `databricks pipelines set-permissions` command. * Added `databricks pipelines update-permissions` command. * Added `databricks serving-endpoints get-permission-levels` command. * Added `databricks serving-endpoints get-permissions` command. * Added `databricks serving-endpoints set-permissions` command. * Added `databricks serving-endpoints update-permissions` command. * Added `databricks token-management get-permission-levels` command. * Added `databricks token-management get-permissions` command. * Added `databricks token-management set-permissions` command. * Added `databricks token-management update-permissions` command. * Changed `databricks dashboards create` command with new required argument order. * Added `databricks warehouses get-permission-levels` command. * Added `databricks warehouses get-permissions` command. * Added `databricks warehouses set-permissions` command. * Added `databricks warehouses update-permissions` command. * Added `databricks dashboard-widgets` command group. * Added `databricks query-visualizations` command group. * Added `databricks repos get-permission-levels` command. * Added `databricks repos get-permissions` command. * Added `databricks repos set-permissions` command. * Added `databricks repos update-permissions` command. * Added `databricks secrets get-secret` command. * Added `databricks workspace get-permission-levels` command. * Added `databricks workspace get-permissions` command. * Added `databricks workspace set-permissions` command. * Added `databricks workspace update-permissions` command. OpenAPI commit 09a7fa63d9ae243e5407941f200960ca14d48b07 (2023-09-04)
arpitjasa-db
pushed a commit
to arpitjasa-db/cli
that referenced
this pull request
Sep 7, 2023
…folder (databricks#708) # Warning: breaking change ## Changes Instead of having paths in bundle config files be relative to bundle root even if the config file is nested, this PR makes such paths relative to the folder where the config is located. When bundle is initialised, these paths will be transformed to relative paths based on bundle root. For example, we have file structure like this ``` - mybundle | - bundle.yml | - subfolder | -- resource.yml | -- my.whl ``` Previously, we had to reference `my.whl` in resource.yml like this, which was confusing because resource.yml is in the same subfolder ``` sync: include: - ./subfolder/*.whl ... tasks: - task_key: name libraries: - whl: ./subfolder/my.whl ... ``` After the change we can reference it like this (which is in line with the current behaviour for notebooks) ``` sync: include: - ./*.whl ... tasks: - task_key: name libraries: - whl: ./my.whl ... ``` ## Tests Existing `translate_path_tests` successfully passed after refactoring. Added a couple of uses cases for `Libraries` paths. Added a bundle config tests with include config and sync section --------- Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com> Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>
arpitjasa-db
pushed a commit
to arpitjasa-db/cli
that referenced
this pull request
Sep 7, 2023
~(this should be changed to target `main`)~ This reveals the template from databricks#686 in CLI prompts for once databricks#686 and databricks#708 are merged. --------- Co-authored-by: Andrew Nester <andrew.nester@databricks.com> Co-authored-by: PaulCornellDB <paul.cornell@databricks.com> Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com> Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>
arpitjasa-db
pushed a commit
to arpitjasa-db/cli
that referenced
this pull request
Sep 7, 2023
This release includes permission related commands for a subset of workspace services where they apply. These complement the `permissions` command and do not require specification of the object type to work with, as that is implied by the command they are nested under. CLI: * Group permission related commands ([databricks#730](databricks#730)). Bundles: * Fixed artifact file uploading on Windows and wheel execution on DBR 13.3 ([databricks#722](databricks#722)). * Make resource and artifact paths in bundle config relative to config folder ([databricks#708](databricks#708)). * Add support for ordering of input prompts ([databricks#662](databricks#662)). * Fix IsServicePrincipal() only working for workspace admins ([databricks#732](databricks#732)). * databricks bundle init template v1 ([databricks#686](databricks#686)). * databricks bundle init template v2: optional stubs, DLT support ([databricks#700](databricks#700)). * Show 'databricks bundle init' template in CLI prompt ([databricks#725](databricks#725)). * Include $PATH in set of environment variables to pass along. ([databricks#736](databricks#736)). Internal: * Update Go SDK to v0.19.0 ([databricks#729](databricks#729)). * Replace API call to test configuration with dummy authenticate call ([databricks#728](databricks#728)). API Changes: * Changed `databricks account storage-credentials create` command to return . * Changed `databricks account storage-credentials get` command to return . * Changed `databricks account storage-credentials list` command to return . * Changed `databricks account storage-credentials update` command to return . * Changed `databricks connections create` command with new required argument order. * Changed `databricks connections update` command with new required argument order. * Changed `databricks volumes create` command with new required argument order. * Added `databricks artifact-allowlists` command group. * Added `databricks model-versions` command group. * Added `databricks registered-models` command group. * Added `databricks cluster-policies get-permission-levels` command. * Added `databricks cluster-policies get-permissions` command. * Added `databricks cluster-policies set-permissions` command. * Added `databricks cluster-policies update-permissions` command. * Added `databricks clusters get-permission-levels` command. * Added `databricks clusters get-permissions` command. * Added `databricks clusters set-permissions` command. * Added `databricks clusters update-permissions` command. * Added `databricks instance-pools get-permission-levels` command. * Added `databricks instance-pools get-permissions` command. * Added `databricks instance-pools set-permissions` command. * Added `databricks instance-pools update-permissions` command. * Added `databricks files` command group. * Changed `databricks permissions set` command to start returning . * Changed `databricks permissions update` command to start returning . * Added `databricks users get-permission-levels` command. * Added `databricks users get-permissions` command. * Added `databricks users set-permissions` command. * Added `databricks users update-permissions` command. * Added `databricks jobs get-permission-levels` command. * Added `databricks jobs get-permissions` command. * Added `databricks jobs set-permissions` command. * Added `databricks jobs update-permissions` command. * Changed `databricks experiments get-by-name` command to return . * Changed `databricks experiments get-experiment` command to return . * Added `databricks experiments delete-runs` command. * Added `databricks experiments get-permission-levels` command. * Added `databricks experiments get-permissions` command. * Added `databricks experiments restore-runs` command. * Added `databricks experiments set-permissions` command. * Added `databricks experiments update-permissions` command. * Added `databricks model-registry get-permission-levels` command. * Added `databricks model-registry get-permissions` command. * Added `databricks model-registry set-permissions` command. * Added `databricks model-registry update-permissions` command. * Added `databricks pipelines get-permission-levels` command. * Added `databricks pipelines get-permissions` command. * Added `databricks pipelines set-permissions` command. * Added `databricks pipelines update-permissions` command. * Added `databricks serving-endpoints get-permission-levels` command. * Added `databricks serving-endpoints get-permissions` command. * Added `databricks serving-endpoints set-permissions` command. * Added `databricks serving-endpoints update-permissions` command. * Added `databricks token-management get-permission-levels` command. * Added `databricks token-management get-permissions` command. * Added `databricks token-management set-permissions` command. * Added `databricks token-management update-permissions` command. * Changed `databricks dashboards create` command with new required argument order. * Added `databricks warehouses get-permission-levels` command. * Added `databricks warehouses get-permissions` command. * Added `databricks warehouses set-permissions` command. * Added `databricks warehouses update-permissions` command. * Added `databricks dashboard-widgets` command group. * Added `databricks query-visualizations` command group. * Added `databricks repos get-permission-levels` command. * Added `databricks repos get-permissions` command. * Added `databricks repos set-permissions` command. * Added `databricks repos update-permissions` command. * Added `databricks secrets get-secret` command. * Added `databricks workspace get-permission-levels` command. * Added `databricks workspace get-permissions` command. * Added `databricks workspace set-permissions` command. * Added `databricks workspace update-permissions` command. OpenAPI commit 09a7fa63d9ae243e5407941f200960ca14d48b07 (2023-09-04) Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Warning: breaking change
Changes
Instead of having paths in bundle config files be relative to bundle root even if the config file is nested, this PR makes such paths relative to the folder where the config is located.
When bundle is initialised, these paths will be transformed to relative paths based on bundle root. For example,
we have file structure like this
Previously, we had to reference
my.whl
in resource.yml like this, which was confusing because resource.yml is in the same subfolderAfter the change we can reference it like this (which is in line with the current behaviour for notebooks)
Tests
Existing
translate_path_tests
successfully passed after refactoring.Added a couple of uses cases for
Libraries
paths.Added a bundle config tests with include config and sync section