Logical optimizer pushdown_filters rule fails with relatively simple query #17512
Description
Describe the bug
COPY (
SELECT arrow_cast('2025-01-01T00:00:00Z'::timestamptz, 'Timestamp(Microsecond, Some("UTC"))') AS start_timestamp
)
TO 'records.parquet'
STORED AS PARQUET;
CREATE EXTERNAL TABLE records STORED AS PARQUET LOCATION 'records.parquet';
SELECT 1
FROM (
SELECT start_timestamp
FROM records
WHERE start_timestamp <= '2025-01-01T00:00:00Z'::timestamptz
) AS t
WHERE t.start_timestamp::time < '00:00:01'::time;Optimizer rule 'push_down_filter' failed
caused by
Internal error: Uncomparable values: TimestampMicrosecond(1735689600000000, Some("UTC")), Time64Nanosecond("1000000000").
This has been failing since #16624 although it seems this PR just exposed the bug, it didn't create it . cc @findepi for confirmation.
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| initial_logical_plan | Projection: Int64(1) |
| | Filter: CAST(t.start_timestamp AS Time64(Nanosecond)) < CAST(Utf8("00:00:01") AS Time64(Nanosecond)) |
| | SubqueryAlias: t |
| | Projection: records.start_timestamp |
| | Filter: records.start_timestamp <= CAST(Utf8("2025-01-01T00:00:00Z") AS Timestamp(Nanosecond, Some("+00:00"))) |
| | TableScan: records |
| logical_plan after resolve_grouping_function | SAME TEXT AS ABOVE |
| logical_plan after type_coercion | Projection: Int64(1) |
| | Filter: CAST(t.start_timestamp AS Time64(Nanosecond)) < CAST(Utf8("00:00:01") AS Time64(Nanosecond)) |
| | SubqueryAlias: t |
| | Projection: records.start_timestamp |
| | Filter: records.start_timestamp <= CAST(CAST(Utf8("2025-01-01T00:00:00Z") AS Timestamp(Nanosecond, Some("+00:00"))) AS Timestamp(Microsecond, Some("UTC"))) |
| | TableScan: records |
| analyzed_logical_plan | SAME TEXT AS ABOVE |
| logical_plan after eliminate_nested_union | SAME TEXT AS ABOVE |
| logical_plan after simplify_expressions | Projection: Int64(1) |
| | Filter: CAST(t.start_timestamp AS Time64(Nanosecond)) < Time64Nanosecond("1000000000") |
| | SubqueryAlias: t |
| | Projection: records.start_timestamp |
| | Filter: records.start_timestamp <= TimestampMicrosecond(1735689600000000, Some("UTC")) |
| | TableScan: records |
| logical_plan after replace_distinct_aggregate | SAME TEXT AS ABOVE |
| logical_plan after eliminate_join | SAME TEXT AS ABOVE |
| logical_plan after decorrelate_predicate_subquery | SAME TEXT AS ABOVE |
| logical_plan after scalar_subquery_to_join | SAME TEXT AS ABOVE |
| logical_plan after decorrelate_lateral_join | SAME TEXT AS ABOVE |
| logical_plan after extract_equijoin_predicate | SAME TEXT AS ABOVE |
| logical_plan after eliminate_duplicated_expr | SAME TEXT AS ABOVE |
| logical_plan after eliminate_filter | SAME TEXT AS ABOVE |
| logical_plan after eliminate_cross_join | SAME TEXT AS ABOVE |
| logical_plan after eliminate_limit | SAME TEXT AS ABOVE |
| logical_plan after propagate_empty_relation | SAME TEXT AS ABOVE |
| logical_plan after eliminate_one_union | SAME TEXT AS ABOVE |
| logical_plan after filter_null_join_keys | SAME TEXT AS ABOVE |
| logical_plan after eliminate_outer_join | SAME TEXT AS ABOVE |
| logical_plan after push_down_limit | SAME TEXT AS ABOVE |
| logical_plan after Optimizer rule 'push_down_filter' failed | Internal error: Uncomparable values: TimestampMicrosecond(1735689600000000, Some("UTC")), Time64Nanosecond("1000000000"). |
| | This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+