IsAdjustedToUtc flag not used while while reading DeltaTable

[ion-elgreco]

Environment

Delta-rs version: 0.10.1

Binding: Python

Environment:

• Cloud provider: Azure
• OS: Linux
• Other:

---

Bug

What happened:
When you write a PyArrow table with UTC timezone datetimes to delta table, the timezone information get's removed in the final table schema. See example below:

What you expected to happen:

Maintain UTC information by looking at the IsAdjustedToUTC flag in the parquet file and passing this back into the schema while reading to arrow. The information is there because when you read the partition directly with Polars, the timezone information is read as UTC:

print(pl.read_parquet('/home/<redacted>/polars/py-polars/test/0-f7fe1e96-4adc-425a-b75c-80d4fdb1336c-0.parquet'))

shape: (1, 1)
┌─────────────────────────┐
│ datetime                │
│ ---                     │
│ datetime[μs, UTC]       │
╞═════════════════════════╡
│ 2010-01-01 00:00:00 UTC │
└─────────────────────────┘

How to reproduce it:

import polars as pl
from deltalake import DeltaTable

df = pl.select(pl.datetime(2010, 1, 1, time_unit="us", time_zone="UTC"))

print(df.to_arrow().schema)
datetime: timestamp[us, tz=UTC]

df.write_delta('test')


dt = DeltaTable('test')

print(dt.schema().to_pyarrow())
datetime: timestamp[us]

More details:

[ldacey]

Ah, just ran into this myself. I thought UTC timestamps were supported so I did:

utc_tz = cs.datetime(time_zone="UTC")
non_utc_tz = cs.datetime() - utc_tz

df = df.with_columns(non_utc_tz.cast(pl.Datetime("us")), utc_tz.cast(pl.Datetime("us", "UTC")), cs.categorical().cast(pl.Utf8))

That adjusted my "ns" timestamp columns and I was hopefully going to leave the UTC timestamp intact. Generally I am saving data with the naive version (localized to whatever time zone the client is using with the TZ info stripped) and a UTC version.

My understanding is that at least UTC timestamps are supported, but just not with the delta-rs library currently?

[ion-elgreco]

Ah, just ran into this myself. I thought UTC timestamps were supported so I did:

utc_tz = cs.datetime(time_zone="UTC")
non_utc_tz = cs.datetime() - utc_tz

df = df.with_columns(non_utc_tz.cast(pl.Datetime("us")), utc_tz.cast(pl.Datetime("us", "UTC")), cs.categorical().cast(pl.Utf8))

That adjusted my "ns" timestamp columns and I was hopefully going to leave the UTC timestamp intact. Generally I am saving data with the naive version (localized to whatever time zone the client is using with the TZ info stripped) and a UTC version.

My understanding is that at least UTC timestamps are supported, but just not with the delta-rs library currently?

So, write support is there, it's writing the parquets properly with UTC timestamps. But the flag is not reused in the schema to read it. I can try to take a look at the issue soon, but I want to work one some other stuff first.

Also in Polars it temporarily casts always to non UTC timezone, until there is a fix in delta-rs