Skip to content
This repository has been archived by the owner on Sep 18, 2023. It is now read-only.

[NSE-609] Complete to_date expression support #622

Merged
merged 7 commits into from
Dec 14, 2021
Merged

[NSE-609] Complete to_date expression support #622

merged 7 commits into from
Dec 14, 2021

Conversation

PHILO-HE
Copy link
Collaborator

In spark, to_date(string [, fmt]) is used to parse a string to DateType according to given format or default format.

There is no dedicated expression for to_date in vanilla spark.

  1. If fmt is not given, spark will cast string to DateType by corresponding cast expression, which is already supported in gazelle.
  2. If fmt is given, spark will firstly use GetTimestamp expression to get timestamp (TimestampType) from the string, then cast the TimestampType to DateType. For this case, we proposed this patch to support it. Currently, only "yyyy-MM-dd" format is supported, otherwise, it will fallback to vanilla spark's execution path.

@github-actions
Copy link

#609

@PHILO-HE PHILO-HE mentioned this pull request Dec 10, 2021
Closed
@PHILO-HE
Copy link
Collaborator Author

A unit test failure needs to be fixed.

zhouyuan
zhouyuan reviewed Dec 14, 2021
View reviewed changes
arrow-data-source/script/build_arrow.sh Outdated
@@ -62,7 +62,7 @@ echo "ARROW_SOURCE_DIR=${ARROW_SOURCE_DIR}"
echo "ARROW_INSTALL_DIR=${ARROW_INSTALL_DIR}"
mkdir -p $ARROW_SOURCE_DIR
mkdir -p $ARROW_INSTALL_DIR
git clone https://github.com/oap-project/arrow.git --branch arrow-4.0.0-oap $ARROW_SOURCE_DIR
git clone https://github.com/philo-he/arrow.git --branch check_input $ARROW_SOURCE_DIR
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this change?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just removed.

@zhouyuan zhouyuan merged commit dfd86a8 into oap-project:master Dec 14, 2021
@PHILO-HE PHILO-HE mentioned this pull request Dec 16, 2021
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@zhouyuan zhouyuan zhouyuan left review comments

Assignees
No one assigned
Labels
feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@PHILO-HE @zhouyuan @weiting-chen