Delta Lake +Unity Catalog exporter: Delta Lake table metadata #7527
Conversation
Jonathan-Rosenberg
added
the
include-changelog
|
♻️ PR Preview 5e9af10 has been successfully destroyed since this PR has been closed.
🤖 By surge-preview |
E2E Test Results - DynamoDB Local - Local Block Adapter |
pkg/actions/lua/databricks/client.go
Outdated
| luautil "github.com/treeverse/lakefs/pkg/actions/lua/util" | ||
|
|
There was a problem hiding this comment.
| luautil "github.com/treeverse/lakefs/pkg/actions/lua/util" | |
| luautil "github.com/treeverse/lakefs/pkg/actions/lua/util" |
There was a problem hiding this comment.
don't mess with goimports
There was a problem hiding this comment.
Don't mess with it, just remove the redundant newline 📏
There was a problem hiding this comment.
sure thing sure thing
pkg/actions/lua/formats/delta.go
Outdated
| "github.com/csimplestring/delta-go/action" | ||
|
|
There was a problem hiding this comment.
| "github.com/csimplestring/delta-go/action" | |
| "github.com/csimplestring/delta-go/action" |
docs/howto/hooks/lua.md
Outdated
| - the first is a table of the format `{number, {string}}` where `number` is a version in the Delta Log, and the mapped `{string}` | ||
| table (list) contains JSON strings of the different Delta Lake log operations listed in the mapped version entry. |
There was a problem hiding this comment.
Sentences are not very clear. table? list?JSON strings?? I'm not sure what I'm getting here
There was a problem hiding this comment.
I'll clarify
docs/howto/hooks/lua.md
Outdated
| The format of the response is two tables: | ||
| - the first is a table of the format `{number, {string}}` where `number` is a version in the Delta Log, and the mapped `{string}` | ||
| table (list) contains JSON strings of the different Delta Lake log operations listed in the mapped version entry. | ||
| - the second is a table of the metadata of the table, it consists of the following fields: |
There was a problem hiding this comment.
Isn't the metadata versioned as well?
I find it confusing that on the one hand we provide a list of versions for the table and on the other hand we provide only the metadata of the current snapshot. Perhaps a clarification is needed
There was a problem hiding this comment.
It's the latest version's metadata.
The intention is to make it easier to get the current state of the table to initialize it in other systems faster.
I'll clarify
Jonathan-Rosenberg
deleted the
feature/add-metadata-to-delta-export-and-unity
branch
Closes #7302
Change Description
FYI @talSofer