Protocol update for Table Features #1450
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hey @tdas and @allisonport-db, could you review this PR? |
tdas
reviewed
Nov 30, 2022
tdas
reviewed
Nov 30, 2022
tdas
reviewed
Nov 30, 2022
tdas
reviewed
Nov 30, 2022
tdas
reviewed
Nov 30, 2022
# Conflicts: # PROTOCOL.md
xupefei
commented
Dec 2, 2022
xupefei
requested review from
tdas
and removed request for
scottsand-db and
allisonport-db
|
Hey @scottsand-db and @allisonport-db, for some reason Github removed you two from the reviewer list after I re-requested a review from @tdas. Please add yourself back :) |
allisonport-db
pushed a commit
that referenced
this pull request
Dec 16, 2022
This PR implements Table Features proposed in the feature request (#1408) and the PROTOCOL doc (#1450). This PR implements the basic functionality, including - The protocol structure and necessary APIs - Protocol upgrade logic - Append-only feature ported to Table Features - Protocol upgrade path - User-facing APIs, such as allowing referencing features manually - Partial test coverage Not covered by this PR: - Adapt all features - Full test coverage - Make `DESCRIBE TABLE` show referenced features - Enable table clone and time travel paths Table Features support starts from reader protocol version `3` and writer version `7`. When supported, features can be **referenced** by a protocol by placing a `DeltaFeatureDescriptor` into the protocol's `readerFeatures` and/or `writerFeatures`. A feature can be one of two types: writer-only and reader-writer. The first type means that only writers must care about such a feature, while the latter means that in addition to writers, readers must also be aware of the feature to read the data correctly. We now have the following features released: - `appendOnly`: legacy, writer-only - `invariants`: legacy, writer-only - `checkConstriants`: legacy, writer-only - `changeDataFeed`: legacy, writer-only - `generatedColumns`: legacy, writer-only - `columnMapping`: legacy, reader-writer - `identityColumn`: legacy, writer-only - `deletionVector`: native, reader-writer Some examples of the `protocol` action: ```scala // Valid protocol. Both reader and writer versions are capable. Protocol( minReaderVersion = 3, minWriterVersion = 7, readerFeatures = {(columnMapping,enabled), (changeDataFeed,enabled)}, writerFeatures = {(appendOnly,enabled), (columnMapping,enabled), (changeDataFeed,enabled)}) // Valid protocol. Only writer version is capable. "columnMapping" is implicitly enabled by readers. Protocol( minReaderVersion = 2, minWriterVersion = 7, readerFeatures = None, writerFeatures = {(columnMapping,enabled)}) // Invalid protocol. Reader version does enable "columnMapping" implicitly. Protocol( minReaderVersion = 1, minWriterVersion = 7, readerFeatures = None, writerFeatures = {(columnMapping,enabled)}) ``` When reading or writing a table, clients MUST respect all enabled features. Upon table creation, the system assigns the table a minimum protocol that satisfies all features that are **automatically enabled** in the table's metadata. This means the table can be on a "legacy" protocol with both `readerFeatures` and `writerFeatures` set to `None` (if all active features are legacy, which is the current behavior) or be on a Table Features-capable protocol with all active features explicitly referenced in `readerFeatures` and/or `writerFeatures` (if one of the active features is Table Features-native or the user has specified a Table Features-capable protocol version). It's possible to upgrade an existing table to support table features. The update can be either for writers or for both readers and writers. During the upgrade, the system will explicitly reference all legacy features that are implicitly supported by the old protocol. Users can mark a feature to be required by a table by using the following commands: ```sql -- for an existing table ALTER TABLE table_name SET TBLPROPERTIES (delta.feature.featureName = 'enabled') -- for a new table CREATE TABLE table_name ... TBLPROPERTIES (delta.feature.featureName = 'enabled') -- for all new tables SET spark.databricks.delta.properties.defaults.feature.featureName = 'enabled' ``` When some features are set to `enabled` in table properties and some others in Spark sessions, the final table will enable all features defined in two places: ```sql SET spark.databricks.delta.properties.defaults.feature.featureA = 'enabled'; CREATE TABLE table_name ... TBLPROPERTIES (delta.feature.featureB = 'enabled') --- 'table_name' will have 'featureA' and 'featureB' enabled. ``` Closes #1520 GitOrigin-RevId: 2b05f397b24e57f1804761b3242a0f29098a209c
bart-samwel
reviewed
Dec 19, 2022
scottsand-db
requested changes
Dec 21, 2022
vkorukanti
pushed a commit
that referenced
this pull request
Dec 29, 2022
This change addresses feedback in #1450 (comment) to remove all uses of `TableFeatureDescriptor` in the codebase because that `status` field in the descriptor can only be `enabled` which can be omitted. Therefore we could simplify store feature names as `Set[String]` in the protocol and assume all listed features are `enabled`. `Protocol` action before this change: ```json { "protocol":{ "readerVersion":3, "writerVersion":7, "readerFeatures":[ {"name":"columnMapping","status":"enabled"} ], "writerFeatures":[ {"name":"columnMapping","status":"enabled"}, {"name":"identityColumns","status":"enabled"} ] } } ``` `Protocol` action after this change: ```json { "protocol":{ "readerVersion":3, "writerVersion":7, "readerFeatures":["columnMapping"], "writerFeatures":["columnMapping","identityColumns"] } } ``` GitOrigin-RevId: 445601bbaebf39f1cd41f8257ece7aeaff48e101
xupefei
requested review from
scottsand-db,
tdas and
bart-samwel
and removed request for
tdas,
scottsand-db and
bart-samwel
xupefei
requested review from
bart-samwel and
scottsand-db
and removed request for
scottsand-db and
bart-samwel
bart-samwel
reviewed
Jan 13, 2023
bart-samwel
approved these changes
Jan 16, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR proposes a change to the Delta Protocol to accommodate Table Features discussed in [Feature Request] Table Features to specify the features needed to read/write to a table #1408.
TOC is updated using
doctoc.How was this patch tested?
Not needed.