databrickslabs · nfx · Mar 7, 2024 · Mar 7, 2024
@@ -1,5 +1,21 @@
 # Version changelog
 
+## 0.15.0
+
+* Added AWS S3 support for `migrate-locations` command ([#1009](https://github.com/databrickslabs/ucx/issues/1009)). In this release, the open-source library has been enhanced with AWS S3 support for the `migrate-locations` command, enabling efficient and secure management of S3 data. The new functionality includes the identification of missing S3 prefixes and the creation of corresponding roles and policies through the addition of methods `_identify_missing_paths`, `_get_existing_credentials_dict`, and `create_external_locations`. The library now also includes new classes `AwsIamRole`, `ExternalLocationInfo`, and `StorageCredentialInfo` for better handling of AWS-related functionality. Additionally, two new tests, `test_create_external_locations` and `test_create_external_locations_skip_existing`, have been added to ensure the correct behavior of the new AWS-related functionality. The new test function `test_migrate_locations_aws` checks the AWS-specific implementation of the `migrate-locations` command, while `test_missing_aws_cli` verifies the correct error message is displayed when the AWS CLI is not found in the system path. These changes enhance the library's capabilities, improving data security, privacy, and overall performance for users working with AWS S3.
+* Added `databricks labs ucx create-uber-principal` command to create Azure Service Principal for migration ([#976](https://github.com/databrickslabs/ucx/issues/976)). The new CLI command, `databricks labs ucx create-uber-principal`, has been introduced to create an Azure Service Principal (SPN) and grant it STORAGE BLOB READER access on all the storage accounts used by the tables in the workspace. The SPN information is then stored in the UCX cluster policy. A new class, AzureApiClient, has been added to isolate Azure API calls, and unit and integration tests have been included to verify the functionality. This development enhances migration capabilities for Azure workspaces, providing a more streamlined and automated way to create and manage Service Principals, and improves the functionality and usability of the UCX tool. The changes are well-documented and follow the project's coding standards.
+* Added `migrate-locations` command ([#1016](https://github.com/databrickslabs/ucx/issues/1016)). In this release, we've added a new CLI command, `migrate_locations`, to create Unity Catalog (UC) external locations. This command extracts candidates for location creation from the `guess_external_locations` assessment task and checks if corresponding UC Storage Credentials exist before creating the locations. Currently, the command only supports Azure, with plans to add support for AWS and GCP in the future. The `migrate_locations` function is marked with the `ucx.command` decorator and is available as a command-line interface (CLI) command. The pull request also includes unit tests for this new command, which check the environment (Azure, AWS, or GCP) before executing the migration and log a message if the environment is AWS or GCP, indicating that the migration is not yet supported on those platforms. No changes have been made to existing workflows, commands, or tables.
+* Added handling for widget delete on upgrade platform bug ([#1011](https://github.com/databrickslabs/ucx/issues/1011)). In this release, the `_install_dashboard` method in `dashboards.py` has been updated to handle a platform bug that occurred during the deletion of dashboard widgets during an upgrade process (issue [#1011](https://github.com/databrickslabs/ucx/issues/1011)). Previously, the method attempted to delete each widget using the `self._ws.dashboard_widgets.delete(widget.id)` command, which resulted in a `TypeError` when attempting to delete a widget. The updated method now includes a try/except block that catches this `TypeError` and logs a warning message, while also tracking the issue under bug ES-1061370. The rest of the method remains unchanged, creating a dashboard with the given name, role, and parent folder ID if no widgets are present. This enhancement improves the robustness of the `_install_dashboard` method by adding error handling for the SDK API response when deleting dashboard widgets, ensuring a smoother upgrade process.
+* Create UC external locations in Azure based on migrated storage credentials ([#992](https://github.com/databrickslabs/ucx/issues/992)). The `locations.py` file in the `databricks.labs.ucx.azure` package has been updated to include a new class `ExternalLocationsMigration`, which creates UC external locations in Azure based on migrated storage credentials. This class takes various arguments, including `WorkspaceClient`, `HiveMetastoreLocations`, `AzureResourcePermissions`, and `AzureResources`. It has a `run()` method that lists any missing external locations in UC, extracts their location URLs, and attempts to create a UC external location with a mapped storage credential name if the missing external location is in the mapping. The class also includes helper methods for generating credential name mappings. Additionally, the `resources.py` file in the same package has been modified to include a new method `managed_identity_client_id`, which retrieves the client ID of a managed identity associated with a given access connector. Test functions for the `ExternalLocationsMigration` class and Azure external locations functionality have been added in the new file `test_locations.py`. The `test_resources.py` file has been updated to include tests for the `managed_identity_client_id` method. A new `mappings.json` file has also been added for tests related to Azure external location mappings based on migrated storage credentials.
+* Deprecate legacy installer ([#1014](https://github.com/databrickslabs/ucx/issues/1014)). In this release, we have deprecated the legacy installer for the UCX project, which was previously implemented as a bash script. A warning message has been added to inform users about the deprecation and direct them to the UCX installation instructions. The functionality of the script remains unchanged, and it still performs tasks such as installing Python dependencies and building Python bindings. The script will eventually be replaced with the `databricks labs install ucx` command. This change is part of issue [#1014](https://github.com/databrickslabs/ucx/issues/1014) and is intended to streamline the installation process and improve the overall user experience. We recommend that users update their installation process to the new recommended method as soon as possible to avoid any issues with the legacy installer in the future.
+* Prompt user if Terraform utilised for deploying infrastructure ([#1004](https://github.com/databrickslabs/ucx/issues/1004)). In this update, the `config.py` file has been modified to include a new attribute, `is_terraform_used`, in the `WorkspaceConfig` class. This boolean flag indicates whether Terraform has been used for deploying certain entities in the workspace. Issue [#393](https://github.com/databrickslabs/ucx/issues/393) has been addressed with this change. The `WorkspaceInstaller` configuration has also been updated to take advantage of this new attribute, allowing developers to determine if Terraform was used for infrastructure deployment, thereby increasing visibility into the deployment process. Additionally, a new prompt has been added to the `warehouse_type` function to ascertain if Terraform is being utilized for infrastructure deployment, setting the `is_terraform_used` variable to True if it is. This improvement is intended for software engineers adopting this open-source library.
+* Updated CONTRIBUTING.md ([#1005](https://github.com/databrickslabs/ucx/issues/1005)). In this contribution to the open-source library, the CONTRIBUTING.md file has been significantly updated with clearer instructions on how to effectively contibute to the project. The previous command to print the Python path has been removed, as the IDE is now advised to be configured to use the Python interpreter from the virtual environment. A new step has been added, recommending the use of a consistent styleguide and formatting of the code before every commit. Moreover, it is now encouraged to run tests before committing to minimize potential issues during the review process. The steps on how to make a Fork from the ucx repo and create a PR have been updated with links to official documentation. Lastly, the commit now includes information on handling dependency errors that may occur after `git pull`.
+* Updated databricks-labs-blueprint requirement from ~=0.2.4 to ~=0.3.0 ([#1001](https://github.com/databrickslabs/ucx/issues/1001)). In this pull request update, the requirements file, pyproject.toml, has been modified to upgrade the databricks-labs-blueprint package from version ~0.2.4 to ~0.3.0. This update integrates the latest features and bug fixes of the package, including an automated upgrade framework, a brute-forcing approach for handling SerdeError, and enhancements for running nightly integration tests with service principals. These improvements increase the testability and functionality of the software, ensuring its stable operation with service principals during nightly integration tests. Furthermore, the reliability of the test for detecting existing installations has been reinforced by adding a new test function that checks for the correct detection of existing installations and retries the test for up to 15 seconds if they are not.
+
+Dependency updates:
+
+ * Updated databricks-labs-blueprint requirement from ~=0.2.4 to ~=0.3.0 ([#1001](https://github.com/databrickslabs/ucx/pull/1001)).
+
 ## 0.14.0
 
 * Added `upgraded_from_workspace_id` property to migrated tables to indicated the source workspace ([#987](https://github.com/databrickslabs/ucx/issues/987)). In this release, updates have been made to the `_migrate_external_table`, `_migrate_dbfs_root_table`, and `_migrate_view` methods in the `table_migrate.py` file to include a new parameter `upgraded_from_ws` in the SQL commands used to alter tables, views, or managed tables. This parameter is used to store the source workspace ID in the migrated tables, indicating the migration origin. A new utility method `sql_alter_from` has been added to the `Table` class in `tables.py` to generate the SQL command with the new parameter. Additionally, a new class-level attribute `UPGRADED_FROM_WS_PARAM` has been added to the `Table` class in `tables.py` to indicate the source workspace. A new property `upgraded_from_workspace_id` has been added to migrated tables to store the source workspace ID. These changes resolve issue [#899](https://github.com/databrickslabs/ucx/issues/899) and are tested through manual testing, unit tests, and integration tests. No new CLI commands, workflows, or tables have been added or modified, and there are no changes to user documentation.

@@ -1,2 +1,2 @@
 # DO NOT MODIFY THIS FILE
-__version__ = "0.14.0"
+__version__ = "0.15.0"