Task: Support Azure Unity Catalog export

[N-o-Z]

Closes #7551
Closes #7553

Change Description

Add support for Unity catalog exporter in Azure
Fix delta exporter to support abfss scheme for delta log path and table physical address

Testing Details

Added esti test case

Breaking Change?

No

After fix this is the output of delta_exporter:

  delta_exporter(completed in 4.949s)

Delta Lake exported table "test-table"'s location: abfss://esti-system-testing@esti4hns.dfs.core.windows.net/cnp12lbck6tb6j1l84lg/testdeltacatalogexportabfss/_lakefs/exported/main/d39916/test_table

Delta Lake exported table "test-table"'s metadata:

	partition_columns = []


	configuration = {}


	created_time = 1707066829815

	description = This is the description of the table

	id = db5e0917-1716-4b0f-a009-c25e5b7304a1

	name = 

	schema_string = {"type":"struct","fields":[{"name":"registration_dttm","type":"timestamp","nullable":true,"metadata":{}},{"name":"id","type":"integer","nullable":true,"metadata":{}},{"name":"first_name","type":"string","nullable":true,"metadata":{}},{"name":"last_name","type":"string","nullable":true,"metadata":{}},{"name":"email","type":"string","nullable":true,"metadata":{}},{"name":"gender","type":"string","nullable":true,"metadata":{}},{"name":"ip_address","type":"string","nullable":true,"metadata":{}},{"name":"cc","type":"string","nullable":true,"metadata":{}},{"name":"country","type":"string","nullable":true,"metadata":{}},{"name":"birthdate","type":"string","nullable":true,"metadata":{}},{"name":"salary","type":"double","nullable":true,"metadata":{}},{"name":"title","type":"string","nullable":true,"metadata":{}},{"name":"comments","type":"string","nullable":true,"metadata":{}},{"name":"__index_level_0__","type":"long","nullable":true,"metadata":{}}]}

[github-actions]

♻️ PR Preview 156c05b has been successfully destroyed since this PR has been closed.

_{🤖 By surge-preview}

[github-actions]

E2E Test Results - DynamoDB Local - Local Block Adapter

[Jonathan-Rosenberg]

Can you update the export_delta_log method with the new parameter?

[Jonathan-Rosenberg]

Very cool!
Thanks for the quick fix
(just the docs thingy)

[Jonathan-Rosenberg]

delete?

[Isan-Rivkin]

Looks good, request only around docs.
I wonder about friction - not sure so it's more of a question:
Unity users will have to use this path transformer, maybe we can make it somehow default?
For example let it be the default in delta_exporter (maybe its a bad idea).
:)

[N-o-Z]

Looks good, request only around docs. I wonder about friction - not sure so it's more of a question: Unity users will have to use this path transformer, maybe we can make it somehow default? For example let it be the default in delta_exporter (maybe its a bad idea). :)

I don't think making it the default behavior for delta export is a good idea. For delta export use cases this might even be a point of friction for users who have set up a Azure Blob Storage (and not an ADLS gen2 storage account)

[Jonathan-Rosenberg]

Suggested change

	used to convert the paths scheme to `abfss`. the built-in `azure` lua library provides this functionality in `transformPathToAbfss`.
	used to convert the paths scheme to `abfss`. The built-in `azure` lua library provides this functionality in `transformPathToAbfss`.

Just had to...

[Jonathan-Rosenberg]

Awesome (one typo change request that doesn't block)

[Isan-Rivkin]

LGTM!

[N-o-Z]

Thanks - waiting to verify the fix completely before merging