Skip to content

Commit 37ebab2

Browse files
committed
Remove variant type promotion and fix notes in the table.
1 parent 7763bbc commit 37ebab2

File tree

1 file changed

+1
-4
lines changed

1 file changed

+1
-4
lines changed

format/spec.md

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -238,7 +238,6 @@ Valid primitive type promotions are:
238238
| `date` | | `timestamp`, `timestamp_ns` | Promotion to `timestamptz` or `timestamptz_ns` is **not** allowed |
239239
| `float` | `double` | `double` | |
240240
| `decimal(P, S)` | `decimal(P', S)` if `P' > P` | `decimal(P', S)` if `P' > P` | Widen precision only |
241-
| _all types_ | | `variant` if Variant spec allows the type | Conversion must produce a Variant containing a single value of the original type |
242241

243242
Iceberg's Avro manifest format does not store the type of lower and upper bounds, and type promotion does not rewrite existing bounds. For example, when a `float` is promoted to `double`, existing data file bounds are encoded as 4 little-endian bytes rather than 8 little-endian bytes for `double`. To correctly decode the value, the original type at the time the file was written must be inferred according to the following table:
244243

@@ -256,11 +255,9 @@ Iceberg's Avro manifest format does not store the type of lower and upper bounds
256255
| `string` | 8 bytes | `long` or `string` | Discard and ignore bounds [1] |
257256
| `string` | not 4 or 8 bytes | `string` | Use `string` if either bound is not 4 or 8 bytes |
258257
| `decimal(P, S)` | _any_ | `decimal(P', S)`; `P' <= P` | |
259-
| `variant` | _any_ | Any type | Discard and ignore bounds |
260258

261259
Notes:
262-
1. Allowed promotion from `long` to `timestamptz` makes 8-byte lower and upper bounds ambiguous becuase `long` values are in milliseconds and `timestamptz` values are in milliseconds.
263-
2. Bounds written for an `int` or `long` column promoted to `string` are ambiguous if both bounds have the same 4 or 8 byte length.
260+
1. Bounds written for an `int` or `long` column promoted to `string` are ambiguous if both bounds have the same 4 or 8 byte length.
264261

265262
Type promotion is not allowed for a field that is referenced by `source-id` or `source-ids` of a partition field if the partition transform would produce a different value after promoting the type. For example, `bucket[N]` produces different hash values for `34` and `"34"` (2017239379 != -427558391) but the same value for `34` and `34L`; when an `int` field is the source for a bucket partition field, it may be promoted to `long` but not to `string`.
266263

0 commit comments

Comments
 (0)