Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement to_unixtime function #5568

Closed
Tracked by #8282 ...
waynexia opened this issue Mar 13, 2023 · 7 comments · Fixed by #9077
Closed
Tracked by #8282 ...

Implement to_unixtime function #5568

waynexia opened this issue Mar 13, 2023 · 7 comments · Fixed by #9077
Assignees
Labels
enhancement New feature or request

Comments

@waynexia
Copy link
Member

waynexia commented Mar 13, 2023

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Like from_unixtime in #2871, to_unixtime is sometimes also useful when compared with other data that is in UNIX timestamp format or don't want to care about the timezone.

Describe the solution you'd like

Take to_unix_timestamp in spark as example:

select to_unixtime(ts) from t;

-> 1460041200

Describe alternatives you've considered

Additional context

@comphead
Copy link
Contributor

comphead commented Mar 13, 2023

@waynexia As a workaround please use #5555

select extract(epoch from ts)

@alamb here we comes again to spark functions, should we support all of them like listed here https://spark.apache.org/docs/3.2.0/api/sql/ even if not all of them compatible with PG
For some people spark syntax is more important than PGs, but for others it can be opposite

@alamb
Copy link
Contributor

alamb commented Mar 13, 2023

should we support all of them like listed here https://spark.apache.org/docs/3.2.0/api/sql/ even if not all of them compatible with PG

I don't think we have a strong or consistent story here -- my personal opinion is to focus on postgres compatibility, and only add spark functions for thing that have no postgres equivalent.

Given DataFusion has all the extension points already, maybe we could add a separate crate for "spark_extensions" or something 🤔 that way anyone who wanted the spark library functions could have them if they wanted but it could be opt in.

@waynexia
Copy link
Member Author

@waynexia As a workaround please use #5555

Thanks for this 👍

Given DataFusion has all the extension points already, maybe we could add a separate crate for "spark_extensions" or something

Also makes sense to me. We can implement and provide it in a separate crate (something like datafusion-spark-ext?) so that others can easily integrate wanted functions (like to_unixtime in this ticket) while keeping the core clean. I'd like to help if we decide to do it.

@alamb
Copy link
Contributor

alamb commented Mar 15, 2023

Cross posting link to discussion on spark functions: #5600

@Tangruilin
Copy link
Contributor

@alamb I can help with this.

Please assign to me

@alamb
Copy link
Contributor

alamb commented Jan 29, 2024

Thanks @Tangruilin -- you already have several items assigned to you:

https://github.com/apache/arrow-datafusion/issues/assigned/Tangruilin

If there are some that you are not working on, or we found blockers, perhaps you can unassign yourself so it is clear what the status is.

Thank you so much for your help

@Tangruilin
Copy link
Contributor

Thanks @Tangruilin -- you already have several items assigned to you:

https://github.com/apache/arrow-datafusion/issues/assigned/Tangruilin

If there are some that you are not working on, or we found blockers, perhaps you can unassign yourself so it is clear what the status is.

Thank you so much for your help

I unassign #8824, other issues i am all working on now.
I will finish them in two weeks.

Tangruilin added a commit to Tangruilin/arrow-datafusion that referenced this issue Jan 31, 2024
Signed-off-by: tangruilin <tang.ruilin@foxmail.com>
Tangruilin added a commit to Tangruilin/arrow-datafusion that referenced this issue Mar 1, 2024
Signed-off-by: tangruilin <tang.ruilin@foxmail.com>
Tangruilin added a commit to Tangruilin/arrow-datafusion that referenced this issue Mar 1, 2024
Signed-off-by: tangruilin <tang.ruilin@foxmail.com>
Tangruilin added a commit to Tangruilin/arrow-datafusion that referenced this issue Mar 2, 2024
Signed-off-by: tangruilin <tang.ruilin@foxmail.com>
Tangruilin added a commit to Tangruilin/arrow-datafusion that referenced this issue Mar 2, 2024
Signed-off-by: tangruilin <tang.ruilin@foxmail.com>
Tangruilin added a commit to Tangruilin/arrow-datafusion that referenced this issue Mar 3, 2024
Signed-off-by: tangruilin <tang.ruilin@foxmail.com>
Tangruilin added a commit to Tangruilin/arrow-datafusion that referenced this issue Mar 3, 2024
Signed-off-by: tangruilin <tang.ruilin@foxmail.com>
Tangruilin added a commit to Tangruilin/arrow-datafusion that referenced this issue Mar 3, 2024
Signed-off-by: tangruilin <tang.ruilin@foxmail.com>
Tangruilin added a commit to Tangruilin/arrow-datafusion that referenced this issue Mar 7, 2024
Signed-off-by: tangruilin <tang.ruilin@foxmail.com>
Tangruilin added a commit to Tangruilin/arrow-datafusion that referenced this issue Mar 7, 2024
Signed-off-by: tangruilin <tang.ruilin@foxmail.com>
Tangruilin added a commit to Tangruilin/arrow-datafusion that referenced this issue Mar 7, 2024
Signed-off-by: tangruilin <tang.ruilin@foxmail.com>
Tangruilin added a commit to Tangruilin/arrow-datafusion that referenced this issue Mar 7, 2024
Signed-off-by: tangruilin <tang.ruilin@foxmail.com>
alamb added a commit that referenced this issue Mar 8, 2024
* [task #5568] add_to_unixtime_function

Signed-off-by: tangruilin <tang.ruilin@foxmail.com>

* Fix clippy

---------

Signed-off-by: tangruilin <tang.ruilin@foxmail.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants