[data] Add better support for udf returns from list of datetime objects #46762
Conversation
Signed-off-by: Matthew Owen <mowen@anyscale.com>
omatthew98
requested review from
ericl,
scv119,
c21,
amogkam,
scottjlee,
bveeramani,
raulchen and
stephanie-wang
as code owners
omatthew98
changed the title
| if dt.microsecond != 0: | ||
| highest_precision = 'datetime64[ns]' | ||
| break | ||
| elif dt.second != 0: | ||
| highest_precision = 'datetime64[s]' | ||
| elif dt.minute != 0: | ||
| highest_precision = 'datetime64[m]' | ||
| elif dt.hour != 0: | ||
| highest_precision = 'datetime64[h]' |
There was a problem hiding this comment.
For my own understanding, is this list of precisions exhaustive?
There was a problem hiding this comment.
The full list is here: https://numpy.org/doc/stable/reference/arrays.datetime.html#datetime-units.
Not including lower precision units (W, M, Y) is not a problem unless people need to represent times so distance that the extra range is needed (for context day precision allows for linux epoch +/- 2.5e16 years).
For higher precision units (ns, ps, fs, as), the precision of python datetime is microseconds so I don't think we need to support those in this function.
| return highest_precision | ||
|
|
||
| def _convert_datetime_list_to_array(datetime_list: List[datetime]) -> np.ndarray: | ||
| # Detect highest precision |
There was a problem hiding this comment.
Nit: This comment seems superfluous since the function is already name detect_highest_dataetime_precision
Signed-off-by: Matthew Owen <mowen@anyscale.com>
omatthew98
force-pushed
the
mowen/transform-date-list
branch
from
6092528
to
f9e9b44
Compare
omatthew98
force-pushed
the
mowen/transform-date-list
branch
from
c682e64
to
a7c6b4d
Compare
Why are these changes needed?
This PR adds better support for the translation of lists of datetime objects. It ensures that if pyarrow blocks are used then the datetime objects will be loaded as timestamps.
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.