Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spark insert CH,Date will shift -1 day #343

Closed
Guanpx opened this issue May 17, 2021 · 3 comments
Closed

Spark insert CH,Date will shift -1 day #343

Guanpx opened this issue May 17, 2021 · 3 comments

Comments

@Guanpx
Copy link

Guanpx commented May 17, 2021

Environment

  • OS version: centOS 7
  • JDK version: 1.8
  • ClickHouse Server version:
  • ClickHouse Native JDBC version: v2.5.4
  • (Optional) Spark version: 2.4.0
  • (Optional) Other components' version: N/A

Steps to reproduce

insert data is :
|2021-01| 2021-01-13|2021-05-19 08:00:00|2021-05-13| 1| |2021-05-13 19:21:07|2021-05-13 19:21:46

when use spark insert to CH, that 2021-01-13 change to 2021-01-12 . . . look like this
|2021-01| 2021-01-12|2021-05-19 08:00:00|2021-05-12| 1| |2021-05-13 19:21:07|2021-05-13 19:21:46

the local tz is Asia/Shanghai
the CH tz is Asia/Shanghai

Other descriptions

@pan3793
Copy link
Member

pan3793 commented May 17, 2021

Sounds like you are using Spark to read data from CSV and write into ClickHouse.
The data type Date itself should be timezone no-related.
Thus, I guess there are something wrong when you reading data into Spark, can you provide the catalyst schema of the data? And print the data use Spark?

@Guanpx
Copy link
Author

Guanpx commented May 17, 2021

the 2021-01-13 type is hive Date

spark df.show
|2021-01| 2021-01-13|2021-05-19 08:00:00|2021-05-13| 1| |2021-05-13 19:21:07|2021-05-13 19:21:46

insert to CH
|2021-01| 2021-01-12|2021-05-19 08:00:00|2021-05-12| 1| |2021-05-13 19:21:07|2021-05-13 19:21:46

@pan3793
Copy link
Member

pan3793 commented May 19, 2021

I cannot reproduce the reported issue. If you are still struggling with it, please provide reproducible code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants