Unable to debug print large Timestamp(TimeUnit::Microsecond) values

[mbutrovich]

Describe the bug

While working on #7285 I noticed that some non-null values of Timestamp(TimeUnit::Microsecond) print as null.

To Reproduce

Running...

let arr: PrimitiveArray<TimestampMicrosecondType> =
            TimestampMicrosecondArray::from(vec![9089380393200000000]);

println!("{:?}", arr);

...results in:

PrimitiveArray<Timestamp(Microsecond, None)>
[
  null,
]

Expected behavior

This is a valid 64-bit microsecond timestamp that Spark says is [290000-12-30 15:00:00.0]. I have not done the math to confirm that result yet, but it's definitely a a valid int64 so we should be able to print the resulting timestamp.

Additional context

I suspect the issue is that PrimitiveArray's fmt method coerces Timestamp to the chrono crate's datetime:

arrow-rs/arrow-array/src/array/primitive_array.rs

Line 1272 in 172d79f

DataType::Timestamp(_, tz_string_opt) => {

which far enough down the call stack clips the values to 32-bit and generate a null.:

https://github.com/chronotope/chrono/blob/fa957cc8dcd4c5a2233108f94a287aab6dedf0a9/src/datetime/mod.rs#L747

I also suspect that other TimeUnits are affected by this issue.

[ndemir]

Confirmed the issue with the following PySpark code:

import pyspark
from pyspark.sql import SparkSession
from pyspark.sql.functions import lit, timestamp_seconds, from_unixtime, expr

# Create a Spark session
spark = SparkSession.builder \
    .appName("TimestampTest") \
    .getOrCreate()

# The timestamp value in microseconds
timestamp_value = 9089380393200000000

# Create a DataFrame with the timestamp value
df = spark.createDataFrame([(timestamp_value,)], ["timestamp_micros"])

# Convert microseconds to timestamp using different methods
df = df.withColumn("timestamp_from_micros", expr(f"TIMESTAMP_MICROS({timestamp_value})"))

# Convert to a human-readable string
df = df.withColumn("timestamp_string", df["timestamp_from_micros"].cast("string"))

# Show the results
df.show(truncate=False)

# Alternative approach - create the timestamp directly
df2 = spark.sql(f"SELECT TIMESTAMP_MICROS({timestamp_value}) as timestamp_value")
df2.show(truncate=False)

# Try with milliseconds by dividing the value
timestamp_value_millis = timestamp_value // 1000
df3 = spark.createDataFrame([(timestamp_value_millis,)], ["timestamp_millis"])
df3 = df3.withColumn("timestamp_from_millis", expr(f"TIMESTAMP_MILLIS({timestamp_value_millis})"))
df3 = df3.withColumn("timestamp_string", df3["timestamp_from_millis"].cast("string"))
print("Testing with milliseconds:")
df3.show(truncate=False)

# Clean up
spark.stop()

+-------------------+----------------------+----------------------+
|timestamp_micros   |timestamp_from_micros |timestamp_string      |
+-------------------+----------------------+----------------------+
|9089380393200000000|+290000-12-30 23:00:00|+290000-12-30 23:00:00|
+-------------------+----------------------+----------------------+

+----------------------+
|timestamp_value       |
+----------------------+
|+290000-12-30 23:00:00|
+----------------------+

Testing with milliseconds:
+----------------+----------------------+----------------------+
|timestamp_millis|timestamp_from_millis |timestamp_string      |
+----------------+----------------------+----------------------+
|9089380393200000|+290000-12-30 23:00:00|+290000-12-30 23:00:00|
+----------------+----------------------+----------------------+

[ndemir]

Continued Investigation.

In chrono library, date types are limited to about +/- 262,000 years. Source: Limitations Section of Chrono That seems to be the reason.

Just to be clear, the underlying data is still there, print does not work if the number is not in the supported range.

use arrow_array::{types::TimestampMicrosecondType, PrimitiveArray, TimestampMicrosecondArray};

fn main() {
    let arr: PrimitiveArray<TimestampMicrosecondType> =
    TimestampMicrosecondArray::from(vec![9089380393200000000]);

    println!("{:?}", arr);
    println!("{:?}", arr.values());
}

PrimitiveArray<Timestamp(Microsecond, None)>
[
  null,
]
ScalarBuffer([9089380393200000000])

[tustvold]

IMO we should just print an error, supporting such extreme dates is probably out of scope for arrow-rs