Skip to content

Commit

Permalink
CR Fixes
Browse files Browse the repository at this point in the history
  • Loading branch information
N-o-Z committed Mar 14, 2024
1 parent f28d260 commit 85cec46
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 2 deletions.
7 changes: 6 additions & 1 deletion docs/howto/hooks/lua.md
Original file line number Diff line number Diff line change
Expand Up @@ -481,7 +481,7 @@ Parameters:

A package used to export Delta Lake tables from lakeFS to an external cloud storage.

### `lakefs/catalogexport/delta_exporter.export_delta_log(action, table_def_names, write_object, delta_client, table_descriptors_path)`
### `lakefs/catalogexport/delta_exporter.export_delta_log(action, table_def_names, write_object, delta_client, table_descriptors_path, path_transformer)`

The function used to export Delta Lake tables.
The return value is a table with mapping of table names to external table location (from which it is possible to query the data) and latest Delta table version's metadata.
Expand All @@ -495,6 +495,7 @@ Parameters:
- `write_object`: A writer function with `function(bucket, key, data)` signature, used to write the exported Delta Log (e.g. `aws/s3_client.put_object` or `azure/blob_client.put_object`)
- `delta_client`: A Delta Lake client that implements `get_table: function(repo, ref, prefix)`
- `table_descriptors_path`: The path under which the table descriptors of the provided `table_def_names` reside
- `path_transformer`: (Optional) A function(path) used for transforming the path of the saved delta logs path fields as well as the saved table physical path (used to support Azure Unity catalog use cases)

Delta export example for AWS S3:

Expand Down Expand Up @@ -744,6 +745,10 @@ The registration will use the following paths to register the table:
`<catalog>.<branch name>.<table_name>` where the branch name will be used as the schema name.
The return value is a table with mapping of table names to registration request status.

**Note: (Azure users)** Databricks catalog external locations is supported only for ADLS Gen2 storage accounts.
When exporting Delta tables using the `lakefs/catalogexport/delta_exporter.export_delta_log` function, the `path_transformer` must be
used to convert the paths scheme to `abfss`. the built-in `azure` lua library provides this functionality in `transformPathToAbfss`.

Parameters:

- `action(table)`: The global action table
Expand Down
1 change: 0 additions & 1 deletion esti/catalog_export_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -629,7 +629,6 @@ func TestDeltaCatalogExportAbfss(t *testing.T) {
AzureStorageAccount: viper.GetString("azure_storage_account"),
AzureAccessKey: viper.GetString("azure_storage_access_key"),
}
//blockstore := setupCatalogExportTestByStorageType(t, testData)

tmplDir, err := fs.Sub(exportHooksFiles, "export_hooks_files/delta")
require.NoError(t, err)
Expand Down

0 comments on commit 85cec46

Please sign in to comment.