Cannot optimize Spark written table with timestamp (INT96) column

[rtyler]

Environment

Delta-rs version: any

Binding: rust

Environment:

• Cloud provider:
• OS:
• Other:

---

Bug

Basically the protocol says that timestamps are supposed to be microsecond precision. By default however Apache Spark stores timestamps in parquet with INT96 😱 and they're read back by the parquet crate as nanosecond precision timestamps.

What happened:

This manifests when optimizing because the schema defined in delta expects Timestamp(microsecond) but reading the parquet files their arrow schema has a Timestamp(nanosecond).

What you expected to happen:

How to reproduce it:

Painfully 😆

More details:

delta-io/delta#643

[Nugget2000]

To reproduce this in python

deltalake==0.10.1
python==3.10.6

import pandas as pd
import pyarrow.dataset as ds
from datetime import datetime
from deltalake.writer import write_deltalake, DeltaTable

data = {'RowDateTime': [datetime.now()]}

df = pd.DataFrame.from_dict(data)

parquet_format = ds.ParquetFileFormat()
write_options = parquet_format.make_write_options(
                use_deprecated_int96_timestamps=True,
            )

# write twice so there is something to optimize
write_deltalake(table_or_uri="bug_delta", data=df, file_options=write_options, mode="append")
write_deltalake(table_or_uri="bug_delta", data=df, file_options=write_options, mode="append")

dt = DeltaTable("bug_delta")
dt.optimize.compact() # <-- this is where we crash
dt.vacuum(retention_hours=0, enforce_retention_duration=False, dry_run=False)

Error:
self.table._table.compact_optimize(
_internal.DeltaError: Data does not match the schema or partitions of the table: Unexpected Arrow schema: got: Field { name: "RowDateTime", data_type: Timestamp(Nanosecond, None), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, expected: Field { name: "RowDateTime", data_type: Timestamp(Microsecond, None), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }

[IamJeffG]

I see the same error in a similar but not exactly the same scenario: when the timestamps stored in parquet are time-zone-adjusted:

pa.timestamp("s", tz="UTC"))

In other words, this can happen even when both Timestamps are Microseconds.

You will see the error:

DeltaError: Data does not match the schema or partitions of the table: Unexpected Arrow schema:
got: Field { name: "datetime", data_type: Timestamp(Microsecond, Some("UTC")), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }
expected: Field { name: "datetime", data_type: Timestamp(Microsecond, None), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }

[ion-elgreco]

@rtyler this is resolved now that we always cast the read data to the correct schema of the table in optimize