Adding some support for PyArrow Date and Datetimes to Rust

[MonoMoggy]

Which issue does this PR close?

Attempts to close #949.

Rationale for this change

This adds some (probably not ideal) support for converting pyarrow date/timestamp fields to rust. Since the datatypes are not uniquely identified by id field.

Requires #967 - otherwise tests will fail.
Maybe this approach is redundant due to: #873

Also I understand that this might not be the proper way to do this - keen to understand more. Also I'm not well versed in rust, so I'll understand if the PR requires a lot more work.

Any pointers/directions are appreciated.

What changes are included in this PR?

Are there any user-facing changes?

[mmuru]

@charliec443:You should check str_ob is date or timestamp and call data_type_date or data_type_timestamp method instead of calling them based on the id since it could be any pyarrow datatypes e.g Date64. Also, you should add tase case for it. If needed, I could provide a test case for it.

[MonoMoggy]

Thanks @mmuru for your feedback. I'll shall add some tests first and then get back to you on the latter bit. With respect to the id side of things.

[MonoMoggy]

Actually could @mmuru provide test cases? I don't think I 100% understand the issue you're pointing out.
Sorry about that.

[mmuru]

@charliec443: Now, you fixed other date and time related issues. Added test case looks good. Here is the my test case before your fix.

import pyarrow as pa
import pytest
from datafusion import ExecutionContext
from datafusion import functions as f
import datetime
from . import generic as helpers

@pytest.fixture
def df():
    ctx = ExecutionContext()

    # create a RecordBatch and a new DataFrame from it
    batch = pa.RecordBatch.from_arrays(
        [helpers.data_datetime("s"),helpers.data_date32(), helpers.data_date64()],
        names=["ts", "dt1", "dt2"],
    )

    return ctx.create_dataframe([[batch]])


def test_select_ts_date(df):
    df = df.select(f.col('ts'),f.col('dt1'),f.col('dt2'))

    # execute and collect the first (and only) batch
    result = df.collect()[0]

    assert result.column(0) == helpers.data_datetime("s")
    assert result.column(1) == helpers.data_date32()
    assert result.column(2) == helpers.data_date64()

[kszucs]

@charliec443 indeed, this change is redundant since apache/arrow-rs#691 already handles the DataType exchange between pyarrow and arrow-rs (hence arrow-datafusion as well). The actual conversion happens here using the C data interface.

#873 should expose the functionality to datafusion.

[mmuru]

@houqp: Currently, Arrow date and timestamp datatypes not supported on the Python binding. It is a blocker for the parquet and other data sources have these columns and not able to use datafusion. Do we know an ETA for #873? Otherwise, we need this fix for the short term till #873 gets merged. Thanks.

[houqp]

I think #873 might need to wait for the arrow 6.x release, which might take some time.

[alamb]

Arrow 6 is planned for mid-October, so a few weeks away

[alamb]

Marking PRs that haven't had activity in over a month as 'stale-pr' to help me filter the list. Please remove the label or let me know if "stale" is not the correct designation

[kszucs]

The new C data interface based approach in #873 should supersede the python string representation based one.

[houqp]

Thank you for your work on this @charliec443. Closing it for now since #873 has already been merged into master.