Skip to content

Commit b23f67b

Browse files
rdbluerdblue
committed
Spec: Add unknown type and additional type promotion cases.
1 parent 09370dd commit b23f67b

File tree

1 file changed

+36
-6
lines changed

1 file changed

+36
-6
lines changed

‎format/spec.md‎

Lines changed: 36 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ In addition to row-level deletes, version 2 makes some requirements stricter for
4848

4949
Version 3 of the Iceberg spec extends data types and existing metadata structures to add new capabilities:
5050

51-
* New data types: nanosecond timestamp(tz)
51+
* New data types: nanosecond timestamp(tz), unknown
5252
* Default value support for columns
5353
* Multi-argument transforms for partitioning and sorting
5454

@@ -184,6 +184,7 @@ Supported primitive types are defined in the table below. Primitive types added
184184

185185
| Added by version | Primitive type | Description | Requirements |
186186
|------------------|--------------------|--------------------------------------------------------------------------|--------------------------------------------------|
187+
| [v3](#version-3) | **`unknown`** | Default / null column type used when a more specific type is not known | Must be optional with `null` defaults |
187188
| | **`boolean`** | True or false | |
188189
| | **`int`** | 32-bit signed integers | Can promote to `long` |
189190
| | **`long`** | 64-bit signed integers | |
@@ -221,6 +222,8 @@ The `initial-default` is set only when a field is added to an existing schema. T
221222

222223
The `initial-default` and `write-default` produce SQL default value behavior, without rewriting data files. SQL default value behavior when a field is added handles all existing rows as though the rows were written with the new field's default value. Default value changes may only affect future records and all known fields are written into data files. Omitting a known field when writing a data file is never allowed. The write default for a field must be written if a field is not supplied to a write. If the write default for a required field is not set, the writer must fail.
223224

225+
All columns of `unknown` type must default to null. Non-null values for `initial-default` or `write-default` are invalid.
226+
224227
Default values are attributes of fields in schemas and serialized with fields in the JSON format. See [Appendix C](#appendix-c-json-serialization).
225228

226229

@@ -230,11 +233,31 @@ Schemas may be evolved by type promotion or adding, deleting, renaming, or reord
230233

231234
Evolution applies changes to the table's current schema to produce a new schema that is identified by a unique schema ID, is added to the table's list of schemas, and is set as the table's current schema.
232235

233-
Valid type promotions are:
236+
Valid primitive type promotions are:
237+
238+
| Primitive type | v1, v2 valid type promotions | v3+ valid type promotions | Requirements |
239+
|------------------|------------------------------|------------------------------|--------------|
240+
| `unknown` | | _any type_ | |
241+
| `int` | `long` | `long` | |
242+
| `date` | | `timestamp`, `timestamp_ns` | Promotion to `timestamptz` or `timestamptz_ns` is **not** allowed |
243+
| `float` | `double` | `double` | |
244+
| `decimal(P, S)` | `decimal(P', S)` if `P' > P` | `decimal(P', S)` if `P' > P` | Widen precision only |
245+
246+
Iceberg's Avro manifest format does not store the type of lower and upper bounds, and type promotion does not rewrite existing bounds. For example, when a `float` is promoted to `double`, existing data file bounds are encoded as 4 little-endian bytes rather than 8 little-endian bytes for `double`. To correctly decode the value, the original type at the time the file was written must be inferred according to the following table:
234247

235-
* `int` to `long`
236-
* `float` to `double`
237-
* `decimal(P, S)` to `decimal(P', S)` if `P' > P` -- widen the precision of decimal types.
248+
| Current type | Length of bounds | Inferred type at write time |
249+
|------------------|------------------|-----------------------------|
250+
| `long` | 4 bytes | `int` |
251+
| `long` | 8 bytes | `long` |
252+
| `double` | 4 bytes | `float` |
253+
| `double` | 8 bytes | `double` |
254+
| `timestamp` | 4 bytes | `date` |
255+
| `timestamp` | 8 bytes | `timestamp` |
256+
| `timestamp_ns` | 4 bytes | `date` |
257+
| `timestamp_ns` | 8 bytes | `timestamp_ns` |
258+
| `decimal(P, S)` | _any_ | `decimal(P', S)`; `P' <= P` |
259+
260+
Type promotion is not allowed for a field that is referenced by `source-id` or `source-ids` of a partition field if the partition transform would produce a different value after promoting the type. For example, `bucket[N]` produces different hash values for `34` and `"34"` (2017239379 != -427558391) but the same value for `34` and `34L`; when an `int` field is the source for a bucket partition field, it may be promoted to `long` but not to `string`.
238261

239262
Any struct, including a top-level schema, can evolve through deleting fields, adding new fields, renaming existing fields, reordering existing fields, or promoting a primitive using the valid type promotions. Adding a new field assigns a new ID for that field and for any nested fields. Renaming an existing field must change the name, but not the field ID. Deleting a field removes it from the current schema. Field deletion cannot be rolled back unless the field was nullable or if the current snapshot has not changed.
240263

@@ -949,6 +972,7 @@ Maps with non-string keys must use an array representation with the `map` logica
949972

950973
|Type|Avro type|Notes|
951974
|--- |--- |--- |
975+
|**`unknown`**|`null` or omitted||
952976
|**`boolean`**|`boolean`||
953977
|**`int`**|`int`||
954978
|**`long`**|`long`||
@@ -1002,6 +1026,7 @@ Lists must use the [3-level representation](https://github.com/apache/parquet-fo
10021026

10031027
| Type | Parquet physical type | Logical type | Notes |
10041028
|--------------------|--------------------------------------------------------------------|---------------------------------------------|----------------------------------------------------------------|
1029+
| **`unknown`** | None | | Omit from data files |
10051030
| **`boolean`** | `boolean` | | |
10061031
| **`int`** | `int` | | |
10071032
| **`long`** | `long` | | |
@@ -1029,6 +1054,7 @@ Lists must use the [3-level representation](https://github.com/apache/parquet-fo
10291054

10301055
| Type | ORC type | ORC type attributes | Notes |
10311056
|--------------------|---------------------|------------------------------------------------------|-----------------------------------------------------------------------------------------|
1057+
| **`unknown`** | None | | Omit from data files |
10321058
| **`boolean`** | `boolean` | | |
10331059
| **`int`** | `int` | | ORC `tinyint` and `smallint` would also map to **`int`**. |
10341060
| **`long`** | `long` | | |
@@ -1089,6 +1115,7 @@ The types below are not currently valid for bucketing, and so are not hashed. Ho
10891115

10901116
| Primitive type | Hash specification | Test value |
10911117
|--------------------|-------------------------------------------|--------------------------------------------|
1118+
| **`unknown`** | `hashInt(0)` | |
10921119
| **`boolean`** | `false: hashInt(0)`, `true: hashInt(1)` | `true` ￫ `1392991556` |
10931120
| **`float`** | `hashLong(doubleToLongBits(double(v))` [5]| `1.0F` ￫ `-142385009`, `0.0F` ￫ `1669671676`, `-0.0F` ￫ `1669671676` |
10941121
| **`double`** | `hashLong(doubleToLongBits(v))` [5]| `1.0D` ￫ `-142385009`, `0.0D` ￫ `1669671676`, `-0.0D` ￫ `1669671676` |
@@ -1119,6 +1146,7 @@ Types are serialized according to this table:
11191146

11201147
|Type|JSON representation|Example|
11211148
|--- |--- |--- |
1149+
|**`unknown`**|`JSON string: "unknown"`|`"unknown"`|
11221150
|**`boolean`**|`JSON string: "boolean"`|`"boolean"`|
11231151
|**`int`**|`JSON string: "int"`|`"int"`|
11241152
|**`long`**|`JSON string: "long"`|`"long"`|
@@ -1267,6 +1295,7 @@ This serialization scheme is for storing single values as individual binary valu
12671295

12681296
| Type | Binary serialization |
12691297
|------------------------------|--------------------------------------------------------------------------------------------------------------|
1298+
| **`unknown`** | Not supported |
12701299
| **`boolean`** | `0x00` for false, non-zero byte for true |
12711300
| **`int`** | Stored as 4-byte little-endian |
12721301
| **`long`** | Stored as 8-byte little-endian |
@@ -1319,10 +1348,11 @@ This serialization scheme is for storing single values as individual binary valu
13191348
### Version 3
13201349

13211350
Default values are added to struct fields in v3.
1351+
13221352
* The `write-default` is a forward-compatible change because it is only used at write time. Old writers will fail because the field is missing.
13231353
* Tables with `initial-default` will be read correctly by older readers if `initial-default` is always null for optional fields. Otherwise, old readers will default optional columns with null. Old readers will fail to read required fields which are populated by `initial-default` because that default is not supported.
13241354

1325-
Types `timestamp_ns` and `timestamptz_ns` are added in v3.
1355+
Types `unknown`, `timestamp_ns`, and `timestamptz_ns` are added in v3.
13261356

13271357
All readers are required to read tables with unknown partition transforms, ignoring the unsupported partition fields when filtering.
13281358

0 commit comments

Comments
 (0)