Skip to content

Commit 67dc9e5

Browse files
rdbluerdblue
authored
Spec: Add v3 types and type promotion (#10955)
1 parent 208ab20 commit 67dc9e5

File tree

1 file changed

+42
-7
lines changed

1 file changed

+42
-7
lines changed

‎format/spec.md‎

Lines changed: 42 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ In addition to row-level deletes, version 2 makes some requirements stricter for
4848

4949
Version 3 of the Iceberg spec extends data types and existing metadata structures to add new capabilities:
5050

51-
* New data types: nanosecond timestamp(tz)
51+
* New data types: nanosecond timestamp(tz), unknown
5252
* Default value support for columns
5353
* Multi-argument transforms for partitioning and sorting
5454

@@ -184,6 +184,7 @@ Supported primitive types are defined in the table below. Primitive types added
184184

185185
| Added by version | Primitive type | Description | Requirements |
186186
|------------------|--------------------|--------------------------------------------------------------------------|--------------------------------------------------|
187+
| [v3](#version-3) | **`unknown`** | Default / null column type used when a more specific type is not known | Must be optional with `null` defaults; not stored in data files |
187188
| | **`boolean`** | True or false | |
188189
| | **`int`** | 32-bit signed integers | Can promote to `long` |
189190
| | **`long`** | 64-bit signed integers | |
@@ -221,6 +222,8 @@ The `initial-default` is set only when a field is added to an existing schema. T
221222

222223
The `initial-default` and `write-default` produce SQL default value behavior, without rewriting data files. SQL default value behavior when a field is added handles all existing rows as though the rows were written with the new field's default value. Default value changes may only affect future records and all known fields are written into data files. Omitting a known field when writing a data file is never allowed. The write default for a field must be written if a field is not supplied to a write. If the write default for a required field is not set, the writer must fail.
223224

225+
All columns of `unknown` type must default to null. Non-null values for `initial-default` or `write-default` are invalid.
226+
224227
Default values are attributes of fields in schemas and serialized with fields in the JSON format. See [Appendix C](#appendix-c-json-serialization).
225228

226229

@@ -230,11 +233,32 @@ Schemas may be evolved by type promotion or adding, deleting, renaming, or reord
230233

231234
Evolution applies changes to the table's current schema to produce a new schema that is identified by a unique schema ID, is added to the table's list of schemas, and is set as the table's current schema.
232235

233-
Valid type promotions are:
234-
235-
* `int` to `long`
236-
* `float` to `double`
237-
* `decimal(P, S)` to `decimal(P', S)` if `P' > P` -- widen the precision of decimal types.
236+
Valid primitive type promotions are:
237+
238+
| Primitive type | v1, v2 valid type promotions | v3+ valid type promotions | Requirements |
239+
|------------------|------------------------------|------------------------------|--------------|
240+
| `unknown` | | _any type_ | |
241+
| `int` | `long` | `long` | |
242+
| `date` | | `timestamp`, `timestamp_ns` | Promotion to `timestamptz` or `timestamptz_ns` is **not** allowed; values outside the promoted type's range must result in a runtime failure |
243+
| `float` | `double` | `double` | |
244+
| `decimal(P, S)` | `decimal(P', S)` if `P' > P` | `decimal(P', S)` if `P' > P` | Widen precision only |
245+
246+
Iceberg's Avro manifest format does not store the type of lower and upper bounds, and type promotion does not rewrite existing bounds. For example, when a `float` is promoted to `double`, existing data file bounds are encoded as 4 little-endian bytes rather than 8 little-endian bytes for `double`. To correctly decode the value, the original type at the time the file was written must be inferred according to the following table:
247+
248+
| Current type | Length of bounds | Inferred type at write time |
249+
|------------------|------------------|-----------------------------|
250+
| `long` | 4 bytes | `int` |
251+
| `long` | 8 bytes | `long` |
252+
| `double` | 4 bytes | `float` |
253+
| `double` | 8 bytes | `double` |
254+
| `timestamp` | 4 bytes | `date` |
255+
| `timestamp` | 8 bytes | `timestamp` |
256+
| `timestamp_ns` | 4 bytes | `date` |
257+
| `timestamp_ns` | 8 bytes | `timestamp_ns` |
258+
| `decimal(P, S)` | _any_ | `decimal(P', S)`; `P' <= P` |
259+
260+
Type promotion is not allowed for a field that is referenced by `source-id` or `source-ids` of a partition field if the partition transform would produce a different value after promoting the type. For example, `bucket[N]` produces different hash values for `34` and `"34"` (2017239379 != -427558391) but the same value for `34` and `34L`; when an `int` field is the source for a bucket partition field, it may be promoted to `long` but not to `string`. This may happen for the following type promotion cases:
261+
* `date` to `timestamp` or `timestamp_ns`
238262

239263
Any struct, including a top-level schema, can evolve through deleting fields, adding new fields, renaming existing fields, reordering existing fields, or promoting a primitive using the valid type promotions. Adding a new field assigns a new ID for that field and for any nested fields. Renaming an existing field must change the name, but not the field ID. Deleting a field removes it from the current schema. Field deletion cannot be rolled back unless the field was nullable or if the current snapshot has not changed.
240264

@@ -949,6 +973,7 @@ Maps with non-string keys must use an array representation with the `map` logica
949973

950974
|Type|Avro type|Notes|
951975
|--- |--- |--- |
976+
|**`unknown`**|`null` or omitted||
952977
|**`boolean`**|`boolean`||
953978
|**`int`**|`int`||
954979
|**`long`**|`long`||
@@ -1002,6 +1027,7 @@ Lists must use the [3-level representation](https://github.com/apache/parquet-fo
10021027

10031028
| Type | Parquet physical type | Logical type | Notes |
10041029
|--------------------|--------------------------------------------------------------------|---------------------------------------------|----------------------------------------------------------------|
1030+
| **`unknown`** | None | | Omit from data files |
10051031
| **`boolean`** | `boolean` | | |
10061032
| **`int`** | `int` | | |
10071033
| **`long`** | `long` | | |
@@ -1023,12 +1049,16 @@ Lists must use the [3-level representation](https://github.com/apache/parquet-fo
10231049
| **`map`** | `3-level map` | `MAP` | See Parquet docs for 3-level representation. |
10241050

10251051

1052+
When reading an `unknown` column, any corresponding column must be ignored and replaced with `null` values.
1053+
1054+
10261055
### ORC
10271056

10281057
**Data Type Mappings**
10291058

10301059
| Type | ORC type | ORC type attributes | Notes |
10311060
|--------------------|---------------------|------------------------------------------------------|-----------------------------------------------------------------------------------------|
1061+
| **`unknown`** | None | | Omit from data files |
10321062
| **`boolean`** | `boolean` | | |
10331063
| **`int`** | `int` | | ORC `tinyint` and `smallint` would also map to **`int`**. |
10341064
| **`long`** | `long` | | |
@@ -1089,6 +1119,7 @@ The types below are not currently valid for bucketing, and so are not hashed. Ho
10891119

10901120
| Primitive type | Hash specification | Test value |
10911121
|--------------------|-------------------------------------------|--------------------------------------------|
1122+
| **`unknown`** | always `null` | |
10921123
| **`boolean`** | `false: hashInt(0)`, `true: hashInt(1)` | `true` ￫ `1392991556` |
10931124
| **`float`** | `hashLong(doubleToLongBits(double(v))` [5]| `1.0F` ￫ `-142385009`, `0.0F` ￫ `1669671676`, `-0.0F` ￫ `1669671676` |
10941125
| **`double`** | `hashLong(doubleToLongBits(v))` [5]| `1.0D` ￫ `-142385009`, `0.0D` ￫ `1669671676`, `-0.0D` ￫ `1669671676` |
@@ -1119,6 +1150,7 @@ Types are serialized according to this table:
11191150

11201151
|Type|JSON representation|Example|
11211152
|--- |--- |--- |
1153+
|**`unknown`**|`JSON string: "unknown"`|`"unknown"`|
11221154
|**`boolean`**|`JSON string: "boolean"`|`"boolean"`|
11231155
|**`int`**|`JSON string: "int"`|`"int"`|
11241156
|**`long`**|`JSON string: "long"`|`"long"`|
@@ -1267,6 +1299,7 @@ This serialization scheme is for storing single values as individual binary valu
12671299

12681300
| Type | Binary serialization |
12691301
|------------------------------|--------------------------------------------------------------------------------------------------------------|
1302+
| **`unknown`** | Not supported |
12701303
| **`boolean`** | `0x00` for false, non-zero byte for true |
12711304
| **`int`** | Stored as 4-byte little-endian |
12721305
| **`long`** | Stored as 8-byte little-endian |
@@ -1319,10 +1352,11 @@ This serialization scheme is for storing single values as individual binary valu
13191352
### Version 3
13201353

13211354
Default values are added to struct fields in v3.
1355+
13221356
* The `write-default` is a forward-compatible change because it is only used at write time. Old writers will fail because the field is missing.
13231357
* Tables with `initial-default` will be read correctly by older readers if `initial-default` is always null for optional fields. Otherwise, old readers will default optional columns with null. Old readers will fail to read required fields which are populated by `initial-default` because that default is not supported.
13241358

1325-
Types `timestamp_ns` and `timestamptz_ns` are added in v3.
1359+
Types `unknown`, `timestamp_ns`, and `timestamptz_ns` are added in v3.
13261360

13271361
All readers are required to read tables with unknown partition transforms, ignoring the unsupported partition fields when filtering.
13281362

@@ -1423,3 +1457,4 @@ Iceberg supports two types of histories for tables. A history of previous "curre
14231457
might indicate different snapshot IDs for a specific timestamp. The discrepancies can be caused by a variety of table operations (e.g. updating the `current-snapshot-id` can be used to set the snapshot of a table to any arbitrary snapshot, which might have a lineage derived from a table branch or no lineage at all).
14241458

14251459
When processing point in time queries implementations should use "snapshot-log" metadata to lookup the table state at the given point in time. This ensures time-travel queries reflect the state of the table at the provided timestamp. For example a SQL query like `SELECT * FROM prod.db.table TIMESTAMP AS OF '1986-10-26 01:21:00Z';` would find the snapshot of the Iceberg table just prior to '1986-10-26 01:21:00 UTC' in the snapshot logs and use the metadata from that snapshot to perform the scan of the table. If no snapshot exists prior to the timestamp given or "snapshot-log" is not populated (it is an optional field), then systems should raise an informative error message about the missing metadata.
1460+

0 commit comments

Comments
 (0)